Instructions

The Impact of Uncertainty Representation

on Perceived Trustworthiness

in LLM Responses

Chair: Prof. Dr. Ulrich Gnewuch

Supervisor: Ana-Maria Sîrbu

Bachelor Thesis Survey

Christopher von Kurowski

Welcome

Thank you for supporting this research study!

Large Language Models (LLMs) such as ChatGPT can produce answers that sound confident even when the presented information is uncertain. With my Bachelor Thesis at the University of Passau, I am investigating how users perceive different ways of uncertainty representation in an LLM assistant interface.

In this study, you will interact with three versions of an LLM assistant. Each version represents the model’s uncertainty in a different format: Verbal, numerical and visual. Each interaction follows precise instructions and is followed by a short questionnaire.

This website (that you are presently looking at) in combination with the questionnaire (that will open in a separate tab) will guide you through the experiment.

Before you start

  • Please complete the study on a laptop or desktop computer.
  • The study is anonymous, no directly identifying data is collected. Demographics are collected in aggregated form.
  • Please communicate in English with the assistant.
  • Please do not close the tab of the website and the tab of the questionnaire.
  • The assistant will assess uncertainty live, please be patient while waiting for an answer, this may take up to a minute or two.
  • Do not enter a question a second time while the assistant is thinking.
  • Please evaluate each format based on your own judgment. There are no right or wrong answers.

Access to the questionnaire

Please click the button „Questionnaire“ below to open the tab for the evaluation questionnaire. It is suggested to place the instructions tab and the questionnaire tab next to each other. Please directly fill out the general questions 1-9, then return to this webpage tab. The questionnaire will ask you to return to this webpage, when finished. Do not close the questionnaire tab for the entire study.

Questionnaire

1. LLM interface interaction

You will now experience the first LLM assistant interface. To get access, please return to the questionnaire tab and proceed to the next page. There the link for the first LLM assistant interface will be provided. Please copy the link into a new browser tab and return to this webpage.

The interaction with the assistant will follow detailed instructions in the form of a task. The task consists of different steps. Please make sure to follow the instructions closely. After checking the box below, the task for the first interface will appear.

Task 1

Please interact with the assistant as described in the steps.

Step 1

  1. Read through the information on the right.
  2. Copy the question only.
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 1

You are playing a geography quiz game with your family. One question starts a debate: some people think Australia is the largest island, others say Greenland. To settle it, you want a clear answer.

Question: What is the largest island in the world?

Step 2

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 2

You are going on a trip to Italy with your friends. As you all don’t have a lot of money, you have to rely on public transport. Your friend group decided to go to Brisighella, a small town near Bologna. Your task is to find a way to get there.

Question: How do I get from Passau to Brisighella by public transport? (60 words)

Step 3

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 3

You would like to invest in the stock market. Your time is valuable, so instead of self-analyzing, you ask a LLM for help. You are currently deciding whether to buy Intel or Apple.

Question: Should I buy Intel or Apple stocks for profitability? (80 words)

Step 4

Now please close the tab with the first LLM assistant interface. Return to the questionnaire tab and click next to proceed to the evaluation. Fill out all provided questions, regarding the interaction with the first assistant interface. The questionnaire will ask you to return to this webpage, when finished.

2. LLM interface interaction

You will now experience the second LLM assistant interface. Again, to get access please return to the questionnaire tab and proceed to the next page. There the link for the second LLM assistant interface will be provided. Please copy the link into a new browser tab and return to this webpage.

After checking the box below, the task for the second interface will appear. Again, please make sure to follow the instructions closely.

Task 2

Please interact with the assistant as described in the steps. After each step take a look at the uncertainty representation.

Step 1

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 1

You want to invest for the long-term but feel unsure which ETF best fits your goals. Since you don’t have time to deeply analyze global markets, you ask an LLM for guidance. You are currently deciding between two ETFs.

Question: Should I buy MSCI World or Europe ETF for long-term? (80 words)

Step 2

  1. Read through the information on the right.
  2. Copy the question only.
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 2

You are helping your younger sibling study for a chemistry test. While reviewing the periodic table, you find a worksheet about common metals used in jewelry. One question asks for the chemical symbol of gold, and you want to answer it correctly.

Question: What is the chemical symbol for gold?

Step 3

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 3

You are planning a low-budget trip to Spain to visit your family. As you don’t have a car, you have to rely on public transport. Your relatives live in Platja d’Aro, a small coastal town near Barcelona.

  • Question: How do I get from Passau to Platja d’Aro by public transport? (60 words)

Step 4

Now please close the tab with the second LLM assistant interface. Return to the questionnaire tab and click next to proceed to the evaluation. Fill out all provided questions, regarding the interaction with the second assistant interface. The questionnaire will ask you to return to this webpage, when finished.

3. LLM interface interaction

You will now experience the third LLM assistant interface. Again, to get access please return to the questionnaire tab and proceed to the next page. There the link for the third LLM assistant interface will be provided. Please copy the link into a new browser tab and return to this webpage.

After checking the box below, the task for the third interface will appear. Again, please make sure to closely follow the instructions.

Task 3

Please interact with the assistant as described in the steps. After each step take a look at the uncertainty representation.

Step 1

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 1

You are planning a summer vacation with your friends along the Croatian coast. Because you don’t have a car, you must rely on public transport. Your destination is Rovinj, a small and charming coastal town with beaches and old streets.

Question: How do I get from Munich to Rovinj by public transport? (60 words)

Step 2

  1. Read through the information on the right.
  2. Copy the question only (including word count).
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 2

You want to protect part of your savings against inflation and economic uncertainty but are unsure which precious metal suits you better. You consult an LLM for help. You are deciding whether to invest in gold or silver for the long-term.

Question: Should I buy gold or silver for long-term? (80 words)

Step 3

  1. Read through the information on the right.
  2. Copy the question only.
  3. Go back to the first LLM assistant interface tab.
  4. Paste the question into the text input field without making any changes and click Send/Enter.
  5. Get an overall impression of the answer.

Information for Step 3

You’re on a trip through Australia and planning your next stop. You don’t know whether the capital is Canberra or Sydney.

Question: What is the capital of Australia?

Step 4

Now please close the tab with the third LLM assistant interface. Return to the questionnaire tab and click next to proceed to the evaluation. Fill out all provided questions, regarding the interaction with the third assistant interface. The questionnaire will ask you to return to this webpage, when finished.

Final questions

In the last step you will answer four questions about all formats combined. Return to the questionnaire tab and click Next. Please fill out the final questions and submit the questionnaire. After submitting, return to this webpage and check the box below.

Thank you for taking part in this study. Your responses are valuable for my Bachelor Thesis research.

Your Prolific completion code: C19AKDIQ

You may now close this window.

Prolific Redirect

Christopher von Kurowski

University of Passau