Writing Tasks for Quantitative and Qualitative Usability Studies

A carefully crafted set of tasks is necessary for success in any type of usability testing. There are some characteristics your tasks should always have, regardless of the specific methodology of your research. However, what precisely counts as a good task depends on whether you’re running a quantitative (quant) or a qualitative (qual) usability study.

Background: Quantitative vs. Qualitative Studies

There are many differences between quantitative and qualitative usability testing. Each one is appropriate for different research goals.

Quant studies closely resemble scientific experiments, with rigorous well-controlled conditions intended to enable the capture of metrics (such as success or time on task).

In contrast, qual studies attempt to discover problems in the UI. Qual research aims to understand the thinking and the difficulties experienced by individual users. It often relies on presenting users with open-ended activities that have the potential to expose issues in the UI.

As a rough summary, the goal of quant user testing is numbers, while the goal of qual testing is insights.

Since the goals and characteristics of these two methodologies are quite different, you’ll need to write tasks differently depending on the type of study. In many cases, a task that would be perfect for a qualitative study wouldn’t work at all for a quantitative study, and vice versa.

Let’s first look at the basic principles of writing tasks for any user testing — quant or qual.

Writing Tasks for Any Kind of Study

• Look at what your users need to do with your product for inspiration for your tasks. Remember that tasks must be as realistic as possible. You don’t want to force participants to do something they wouldn’t do in real life, or you’ll measure something that’s irrelevant to your project. User interviews, search-log data, customer-service calls, and analytics data are good places to find out what people want to use your product for.
• Avoid accidentally providing any clues in your task. Don’t describe the exact steps users need to take — let them figure it out on their own. Avoid the exact language used as labels in your UI. You don’t want to prime users to look for a specific word.

Bad: Sign up for the Alertbox.

Better: Sign up for this site’s newsletter.

• Try to keep the task emotionally neutral. Avoid making assumptions about your user’s personal life or relationships when you’re writing tasks. When possible, it’s better to write a task referencing a “friend,” rather than “your father,” “your spouse,” or some other family member.

Bad: Find a gift for your mom’s birthday.

Better: Find a gift for your friend’s birthday.

• Always pilot test your tasks. This is a critical step that far too many researchers skip. You may think your tasks will work well, but you don’t know until you try them out with one or two representative users. Always plan for a pilot in your study budget and schedule. It’ll save you from wasting your resources by accidentally using a bad task and from getting bad data.

Writing Tasks for Qualitative Studies

• Open-ended tasks are OK. For qualitative studies, it’s OK to leave the task open to interpretation. While sometimes you’ll want  to give users specific criteria, you can also leave tasks open to see what they care about, and what factors into their decision-making.

Specific qual task: Open a Cash Rewards Credit Card.

Open-ended qual task: Find a credit card that fits your needs.

• Provide enough detail to set the scene, but don’t go overboard. It’s important to make sure you’re establishing the participant’s motivation for doing this task, but you don’t have to write a novel. If you’ve made sure this participant is representative of your actual users, and you’re sure this task is realistic for them, then you shouldn’t need to provide too much context.
• If you aren’t getting the insights you need, change the task. In a qualitative study, it’s OK to change your task wording mid study, or even throw out or add a new task altogether. Qualitative research is all about getting useful observations, and it’s fine if you don’t have every participant do the exact same tasks. (In fact, better than wasting precious participant time on repeating a useless task.)

Writing Tasks for Quantitative Studies

• Make sure there’s only one way to do the task. In contrast to qualitative tasks, quantitative tasks must have only one possible interpretation and solution. They can’t be ambiguous.
  If the instructions are too broad or open-ended, each participant may do essentially different tasks and take different paths through the interface. The metrics you collect won’t describe the same thing, and you’ll have a lot of variability in your data. And how will you know what counts as a “success” if there are many different possible solutions?

Bad: Find a credit card that fits your needs.

Better: Open a Cash Rewards Credit Card.

• Provide as many details as necessary to keep the task narrow and focused. In some cases, this means providing a level of detail that would feel excessive in a quantitative study.
  Often, you might think you have enough detail, but then in a pilot test you’ll realize you forgot something. For example, it isn’t enough to tell users which hotel to book at which time period — you also need to tell them what kind of room to book. This level of direction would not be required in a qualitative study, but it’s necessary in a quantitative test to make sure everyone will perform exactly the same task.

Bad: Book a room for 2 people at the Hyatt Regency in Chicago from January 17 to January 19.

Better: Book a Queen Room for 2 people at the Hyatt Regency in Chicago from January 17 to January 19.

• Provide fake credentials (dummy data) for login, checkout, or any other task that requires entering personal information. The fact that everybody will enter the same information minimizes variability. All participants will type the same strings. Additionally, some people may be more hesitant than others to share their data. It may take up extra time to come up with an acceptable solution. (For many qual studies, you may want to have participants use their real data instead.)
• Each task should stand alone. Ideally, you want to be able to randomize the order of tasks in quantitative studies. If you have two tasks that build on each other, those tasks must always be presented in the same order. Additionally, someone who fails the first task will automatically fail the second.

Bad:

Task 1: Find the personnel file on Joe Smith.

Task 2: On Joe Smith’s personnel file page, find his direct manager.

Better:

Task 1: Find the personnel file on Joe Smith.

Task 2: On the page provided at this link, find who the direct manager of Jim Grant is.

• Each task should have one single success criterion. Avoid combining two tasks into one. In quantitative studies, you need a single, clear end point to help you determine time on task and whether the user was successful. If you have multiple success criteria in a single task, what happens if a user finds one piece of information but not the other? Fix stacked tasks by splitting them up.

Bad: Find the museum’s address and holiday hours.

Better:

Task 1: Find the museum’s address.

Task 2: Find the museum’s holiday hours.

• Once you’ve begun the study, don’t change the tasks. It’s important that each participant is doing the exact same task in quantitative studies. These studies should hold all conditions constant except your independent variable (for example, your app vs. your competitor’s app). That means you don’t want to get halfway through your study and realize you want to change the wording of a task. Doing so may contaminate your final result. Pilot testing is critical for any research, but particularly important for quantitative testing, when you don’t have the flexibility to change things mid study.
• Focus on top tasks. Quant studies are expensive, and it does not pay off to test tasks that aren’t high priority for your users and your organization. Whereas qual tests may have more flexibility to try tasks that are realistic edge cases, quant studies must focus on the most important, core tasks.

Recap: Qualitative vs. Quantitative Tasks

Regardless of the type of study you’re planning, writing good tasks takes practice.

For hands-on practice writing good qualitative tasks and tips for facilitating a qualitative usability study, check out our full-day seminar Usability Testing.