This is the second in a series of posts looking at the advice provided in the excellent book Think like a UX Researcher, by David Travis and Philip Hodgson, from the perspective of involving people with disabilities in UX research. Part 1 of this series focused on defining the research question(s) an activity should address. In this post, we look at what the book has to say about the question of how many participants to include in a research activity.
How many participants?
Some organizations might be skeptical about involving people with disabilities at all in user research, treating them as edge cases. Other organizations may stop in their tracks before they get started on planning user research with people with disabilities as they agonize over concerns of representativeness—how many people to include, and what profiles to cover?
The book includes a section “Arguments against a representative sample” that provides some helpful guidance on how you can approach recruitment of people with disabilities without getting tied up in concerns about sample size or data validity. In this section, Travis and Hodgson present four key reasons challenging the assumption that a representative sample of participants is important for design research. These are listed below with some comments on how their principles might apply to working with people with disabilities.
- Representative samples require lots of participants. Representativeness might cover common demographic characteristics like gender, age, education, and geographical location. For disability, you might also think about types of disability (vision, hearing, mobility, and cognitive; in isolation or in combination), plus severity, nature of acquisition, type of assistive technology used, experience using assistive technology. So an attempt to recruit a representative sample may take a lot of time, effort and money, and it may soon become impractical.
- Representative samples are hard to achieve in an agile development environment. Iterative, agile development approaches don’t align too well with large-scale, complex research studies. In an agile development process, a research activity’s focus is on quickly gathering specific insights to help to move the design forward. Recruitment should focus on participants who will help you understand that problem and give you feedback on whether the design addresses it. If an iteration has focused on improvements intended to address issues experienced by a particular disability group, focus on an evaluation with members of that disability group.
- Representative samples stifle innovation. If you’re developing a new product, you may have a rough idea about your target audience, but you might not yet appreciate the product’s potential usage scenarios. Design research for innovation focuses on identifying these new usage opportunities, and one way to do this is to look at the margins of a population bell curve. People with disabilities are an often-overlooked sector of the population, yet can be the catalyst for innovation (think predictive text entry, text-to-speech, and voice input). So focusing on people from this sector of population might help you discover something significant.
- Representative samples reduce your chances of finding usability problems that affect a small number of users. UX research is driven by finding and understanding problems that affect users so that you can design to avoid those problems. When you focus on design research with people with disabilities, it follows that you increase the chances of finding accommodation-related usability problems. But it turns out that many of these are also problems that other users may also experience in different scenarios. So by involving even a small group of participants with disabilities, you increase the chance that you find significant problems with a design that are worth solving.
Focus on iterative research over representativeness
If you do need a recommendation for a number, and you’re starting off with a usability study or user research activity, we find that eight participants is a good ballpark figure. That gives you potential for a decent cross-section of disabilities and other characteristics, and a study that can be run in a couple of days.
But in an increasingly agile world, you need to think more about UX research as a series of rapid and focused activities with a small group of people. This can work for research with people with disabilities too. Run a small usability study to identify problems that participants with disabilities encounter; analyze the source of the problems, and identify potential design solutions. Implement them while thinking about who you want to recruit to evaluate these solutions; then run another usability study with another set of participants to validate your efforts. Repeat, and over time product usability should improve.
Here are some tips that might help guide you in planning the involvement of participants with disabilities:
- Be strategic with each sample of participants—rather than trying to cover all disabilities and assistive-technology uses in one sample, aim to cover as many as you can in multiple research activities over time.
- The observed experience of an experienced assistive technology user is valuable, but treat it as you would treat the experience of any expert. There’s a difference between expert users providing you with expert feedback on the accessibility of your product and watching non-experts, where limited experience with an assistive technology can magnify the effect of a usability issue. So try to ensure that user research includes people with less experience using their assistive technology.
- You could include participants from previous studies to provide feedback on the improvements in a new design, but this should be in addition to, rather than instead of recruiting new participants.
- Invite observers from the product development team to observe studies. The more observers, the more chance there is of someone noticing a usability issue that others miss.
- And even if you start by talking to just one person with a disability in your UX work, that’s much better than none at all.
The next post in this series looks at what Think like a UX Researcher says about moderating usability studies and how you can avoid making some common mistakes when running studies with people with disabilities.
