Writing effective unmoderated usability test questions

In an unmoderated usability test, there is no moderator who can answer clarifying questions or set the discussion tone or agenda. Therefore, writing questions that provide them clear guidance is crucial to getting the insights you want. Keep these tips and best practices in mind when you're authoring the unmoderated usability activity.

Tip:

Need to get started with unmoderated usability testing fast? Check out our quick start guides:

• Authoring and distribution
• Reporting
• Participant experience

Tip:

Participant Training - Digital Experience Testing Video

Need to show participants what to expect when they complete an unmoderated usability test? Download our how-to video and add it to the beginning of the survey using a Media Player element. Participants can view the video and learn how to complete an unmoderated usability test step by step.

Usability Task questions

Usability Task questions ask participants to complete actions and activities. Based on how the participants fare, you can better understand user experience and how to optimize it.

In an unmoderated test, it's important that tasks are as clear as possible to participants. Participants do not have someone to turn to if they are confused and have clarifying questions, or if they misunderstand the assignment and need someone to get them back on track.

Tip: Sometimes participants hesitate to provide a video response if they don't feel good about their appearance or their environment. We recommend reminding participants that during the usability test, they can fix themselves up, tidy up, or move to a different room if they want and come back another time when they feel camera-ready.

Tip: Be clear in your question text about when participants should return to the survey tab.

Since participants are completing the task in a new browser tab, you need to give them a clear end point to indicate when they should switch back to the survey tab (for example, Once you're done exploring or you've been able to sign up to join, come back to this tab and click Next).

The question text should also cover a scenario in which participants are unsuccessful (for example, If you haven't been able to sign up to join within five minutes, please return to the survey and click Next). You want participants to return to the survey and let you know they were unsuccessful instead of just closing their browser tabs and being marked as incomplete.

Tip: Add these reminders in the instruction text:

• Include language that reminds participants to think out loud.

  This is also shown to participants when they see the Recording action's tips and reminders, but reiterating this message helps.

• Include language that reminds participants not to stop the recording manually.

  Participants may not understand that the Recording action has to run until the end of the survey, and may be tempted to stop the recording manually.

Usability Task questions can fall into two broad categories:

1. Open-ended tasks
2. Specific tasks

The following table outlines the differences in both.

	Open-ended tasks	Specific tasks
Description	Use this task type to see how contributors explore or think about a question, having been given some direction and context or a scenario to imagine. Answers are varied, but offer an unbiased view of participants' positive and negative sentiments.	Use this task type to investigate exact issues you're interested in. Behaviors tend to be more focused because you're providing direct guidance on what you want participants to look at and the tasks they have to complete.
Use cases	Understand what users are thinking. Open-ended questions/tasks allow contributors to fully explore and describe a genuine experience. Conduct exploratory research. Open-ended tasks can help you figure out how people actually use your product or service. Exploratory research often uncovers areas of interest that can be studied further in a more targeted follow-up test. Identify usability issues. A test featuring open-ended tasks can reveal pain points or areas of friction with the product or prototype that you may not be aware of.	Feature testing Specific tasks are ideal for testing the usability of a certain feature or area of your product. For example, "Please use the search bar to find a pair of size 28 women's jeans and add them to the cart." Complex products or scenarios If you have a very specific product concern or are investigating a complex scenario, specific tasks can provide context and instruct participants on how to use the product. Optimizing a specific workflow and validating specific design decisions (for example, online shopping checkout)
Potential pitfalls	Not having a clearly defined test objective. Have a clear test objective and make sure the tasks support that objective. Not having a well-defined goal will likely make analyzing your test results more difficult and time-consuming. Contributors stop talking. You don't want them to forget to think out loud as they explore, so remind them to explain why they're doing what they're doing.	Giving overly specific step-by-step instructions. Keep a balance between guiding participants and having them experience the product or prototype on their own. Telling participants every single thing to do means you won't learn much, so be judicious when doing any hand-holding.
Examples	Please explore the app as you normally would and provide your first impressions. Spend no more than three minutes to complete this task. Imagine you are settling in for the night and trying to decide on a TV show to watch on your streaming service. Show us what that process looks like. Remember to think out loud and describe what you are thinking as you complete the task. Imagine you want to learn German. Show us how you would go about that in this language learning app.	Open up the workout tracking feature and try to start tracking your gym session. Go to the home page of the streaming app and find the new releases section. Go to the company website and find information about their products' security compliance and features.
Question writing tips	You don't have to give lots of elaborate details in your backstory. Just offer enough details to sketch the scenario.	Name the task, not the thing participants need to click. For example, if you're testing whether participants can find the Join Now link on your website, don't say "Find the Join Now link." Instead, say something like "Go to our website and sign up to join."

Tip: Using both open-ended and specific questions will result in more well-rounded and actionable feedback and a more engaging testing experience.

If your usability test features multiple tasks and you want to have a mix of both question types, we recommend structuring your Usability Task questions like a funnel, working from general to more precise. Start with open-ended questions and drill down to more specific tasks.

Tip:

Use piping to include participants' prior responses in the text of a Usability Task question.

For example, "You indicated in the screener questions that you subscribe to 2 or more streaming TV services. Please describe how you decide on which streaming service to watch on any given evening, how you look for content that interests you, which movies or shows you're most interested in, etc."

Follow-up questions

In an unmoderated test, you don't have a moderator to dig deeper in the moment and follow up on interesting tangents that participants' comments may present. Having good follow-up questions that probe a little deeper, and prompt participants to reflect more on the tasks they just completed, could be a viable substitute. You can use follow-up questions to collect reflections or any further data points you need in your analysis later. It's a good idea to follow up with both closed- and open-ended questions for the best understanding and ease of analysis.

Here are a few examples to get you started.

Example: Close-ended questions

How easy was it to find what you were looking for?

• Very easy
• Somewhat easy
• Somewhat difficult
• Very difficult

How likely are you to recommend this site to a friend or colleague?

Tip: You can use a Net Promoter Score℠(NPS®)¹ question for this. For more information, see Create a Net Promoter Score℠ question.

Did you notice if there was another way to [complete a specific step/task]?

Was there anything about [task] that surprised you?

Example: Open-ended questions

What are your thoughts on the layout of [section]?

What frustrated you most about this task?

If you had a magic wand, how would you improve this site?

What did you like about the site?

What came to mind when you had to [task]?

What would you expect to happen once you've [task]?

What would enable you to accomplish [task] more effectively?

What are your overall impressions of the product or the session?

What worked well/poorly during [task]?

What difficulties did you have with [task]?

What comments did you want to add during the test but didn't?

Do you have any final thoughts on the task or on today's session?

Tip: Use survey logic to show different sets of follow-up questions to participants depending on whether they answered yes or no to the task success question.

Related information

• The unmoderated usability testing workflow
• Create a Usability Task question
• The participants' experience during an unmoderated usability test

¹ Net Promoter, NPS, and the NPS-related emoticons are registered U.S. trademarks, and Net Promoter Score and Net Promoter System are service marks, of Bain & Company, Inc., NICE Systems, Inc. and Fred Reichheld.