🤔 Q&A Research and Discovery Part 1: Respondents
Some time ago, me (Moa) and Jens Wedin had a product discovery talk with a company and thereafter, we opened up for questions. And of course, the questions were brilliant and relevant, but the answers might be a bit less brilliant. But I thought I would share the questions and the answers. And these are questions I do get frequently.
Remember, these answers are based upon my experience working with operative and strategic research, as well as planning and coaching. Things I have found to work well in the places I’ve worked in, and that might differ hugely from workplace to workplace. And adapting frameworks, guidelines, ways of working and routines is up to you to test and experiment with. Dare to test, and don’t forget to reflect — as we say in Sweden “ingen minns en fegis“ (no one remembers a coward).
I realised I had a bit too much to say around each topic, so I have divided this into four pieces.
The upcoming topics will be:
Recruiting respondents, cost, and incentives
Methods, processes, and mindset
Qualitative and Quantitative methods
Other important things to keep in mind
First one out is probably the one I get most questions about, and that is regarding recruiting respondents for usability tests, interviews, and observations.
Questions and Answers in this piece:
How many respondents do I need?
How do you recruit users?
Fixed test slots, how did you recruit users?
When doing weekly tests, do you offer any gift or similar to attract users?
Roughly how much does it cost to recruit a responder through a firm?
🧮 How many respondents do I require?
⏱️ Short answer: 5–6 respondents, but less is fine, but more is even better.
🦥 Long answer: Well, this has been a hot potato lately. The discussion has been around if there even is a golden number? And as it turns out, there is no easy answer to this question.
The number of respondents you need for a qualitative study depends on several aspects, such as:
What kind of study you’re conducting — exploratory interviews often require more respondents than usability tests.
Your budget and time. Would you rather skip doing research because you don’t have the time or budget to interview enough respondents? My opinion here is to just do it with the people you have available and can afford. If you can show results that have effect upon business and product outcome, that is the best way to increase your research budget long term.
Method — what kind of method you’re using
What kind of criteria do you need for a representative sample
Consider the risk and think about how big your project is, if the risk is high, you might need a much bigger sample size. This could be an expensive new feature or a big project, or something that might have a big effect on the users/customers or organisation.
And many more factors, which you will probably find out for yourself.
This is a great quote from Nielsen Norman Group — “Interviews are a qualitative research method. Qualitative research aims to understand the human experience in detail, not to determine how many people have had a given experience or express a particular need.”
So what you aim for in qualitative research is identifying unique themes. The above article refers to another article from N/N Group which states that after 5 interviews you have reached 85% of the unique themes. That statement was based upon a literature study made upon many papers covering sample sizes in qualitative studies. For every respondent over 5, you find fewer and fewer unique themes, which means you require so many more respondents to cover the last 15%. And that might not be worth the effort or financial cost.
Researchers working in academia have not come to a consensus around the right number of respondents, sample sizes in these studies have ranged from 1 to 95. So there isn't a right or wrong answer to this question. If you don’t work in academia, my recommendation is — work with what you have and can afford, and take your own decision based upon budget, time, risk, and recourses.
So my guidance here is to land around 5–6 respondents, or try to aim for that. And if you have not found what you’re looking for after those interviews, add 1 respondent at a time, so you don't add up unnecessary cost and unnecessary time. If you can’t recruit or find 5–6 respondents, do it anyway, just consider the small sample size during analysis. So consider the aspects above and make your own decision. But remember, one is better than none.
Read more: Link to study, Link, Link,
📫 How do you recruit users?
⏱️ Short answer: No budget = friends, family, and colleagues. With budget = recruitment agencies.
🦥 Long answer: It depends (like everything else). I think many UX-designers and researchers agree that they probably started by recruiting friends and family because that was the only way to do it. And here we go again, isn’t that a problem with biases? Yes, of course, in an ideal world you would not recruit friends, not family and not people in the company who work close to the product you work with. But it is a way to get the job done when the budget isn’t there, and to prove how valuable it is to conduct customer research. And that could lead you to get a budget to recruit outside of friends, family, and company.
But let’s be a bit more useful, here are some pointers depending on where you are in your journey and what budget you have.
Friends and Family
If you’re recruiting friends and family, if you’re two people or more who conduct the test, let someone else interview them. Treat them like you would have if they were not your family, so don’t do the interview over a glass of wine at a party (if that’s not what you’re exploring). But at the same time take advantage of knowing that person. The respondent feels comfortable around you, you know how to make them talk and how to read them. Make sure to consider potential biases when you analyse the data.
Colleagues
If you need to recruit from the company you work at, and you’re not evaluating any internal system. Try to find people working far away from the product and who are experts on your product. Maybe people who have just started at the company or work with different things. I have often taken advantage of colleagues, mostly for usability tests because. These tests don't have a lot of risk, like finding usability problems compared to exploring mind-sets, behaviours, and values.
Occasionally, I have even sent out scenarios and tasks to my team, and let them record themselves trying to solve a task and then send it to me. A good way of testing quick and dirty.
Recruitment agencies
So if you have played your cards right, you might have the luxury to have someone else do the recruitment for you. I typically say this is the one thing you should spend your money on, you can work yourself around expensive software, but this is so extremely time-consuming and can have a big impact on the quality of your test. I have worked with several partners for recruitment.
Try to document some requirements 1) what respondent profile you need and how often 2) length of a regular session 3) what kind of sessions 4) methods and so on. And when you talk to different agencies, see if you can find a good deal. Are there any packages you can get, including various target groups and various sessions? Do you want to pay extra for services you won't use? Are they offering to host software you require and can that be included? Do they have a database with your target group, or do they need to collect them and so on? But it has been one of the things I think has been the most valuable to spend research budget on.
Prepared customer panel
Check if your company already has a customer panel, or customer database, which you can contact. Here you have to check with the legal or privacy departments what is OK to approach them with. You can send out mail asking if anyone is interested in having a more in-depth involvement and to contributing to your product, of course, offering incentives to do so. The problem with customer panels is that they are often quite attached to your product, and you can’t ask them questions regarding mindsets toward your product. But might work fine if you want feedback on an up-and-coming service or usability tests.
💻 How did you recruit users for fixed test slots?
⏱️ Short answer: through recruitment agencies, so worth it.
🦥 Long answer: If you work with an agency, there are several ways to go about it. One way is to ask the agency to recruit x number of respondents and then give them dates that works for you. You have to consider how many interviews you have the energy to do in one day. Are you doing 6 interviews? Maybe spread them over two days instead of trying to do them during one day. Because it takes a lot of energy, and you want to do them with great quality. The agencies are good at finding respondents who are available when you need them to be. So you can be a bit more precise with time in this scenario.
Are you handling the scheduling yourself? You might need numerous days open in your schedule to just handle interviews and let the respondents suggest when it suits them. You have to make yourself more available if you’re recruiting yourself because you have much fewer respondents to choose from.
Fixed slots, you might also have a concept where you have a set of fixed slots. You have scheduled tests on Tuesdays every second week. For this one, you probably require someone to help you recruit to make sure you have respondents for every slot every second week. And you most likely have to be a team doing research because doing tests yourself every second week is quite a lot. But it is a great way to make customer interaction a natural part of your product development process.
🎁 When doing weekly tests, do you offer any gift or similar to attract users?
⏱️ Short answer: Yes, always pay with something. Gift cards, fika, candy, movie tickets or just a coffee.
🦥 Long answer: In my experience, there is no difference to paying respondents whether you have weekly tests or not. Agencies have a set price list depending on how many screening criteria you have, which means if you have criteria which fewer people in their database fulfils, and they need to recruit for you specifically, it’s going to be pricier. And what kind of test or interview, and how long the session is. The agencies I have worked with have offered their respondents gift cards. If you work with an agency, they handle all of this, and you don’t have to mind that except opening up your wallet.
In other cases where I have been recruiting myself, I have offered film tickets. But make sure what is OK in your organisation. When I worked for companies who could offer credit in their service, I offered credits. Which only works if the respondent likes using your product, otherwise why would they accept that kind of payment.
And there is an important balance of giving too little and too much. The main motivation should be that the respondent wants to contribute to your product or product development in general. And getting paid is just something extra. Then you get the most genuine respondents who really would like to contribute and are not only there to get paid, which can result in them not answering truthfully.
💵 Roughly how much does it cost to recruit a responder through a firm?
⏱️ Short answer: OK, this differs a lot, but around 12k for 6 respondents, 45-60 min, with no super hard criteria. Which means you can ask for a spread in gender, age, maybe how they live and income. Demographic values and perhaps 1 to 3 more specific criteria such as — have savings account at a bank, have a mortgage and so on. What is also included is that you don’t need to handle any contact with the respondent before or after the test. And the recruiter also handles no-shows, respondents that do not show up for your schedule time. But this is just a price I have come across, make sure you compare for yourself. And there is often a deal to be made.
The next part will be regarding methods, processes, and mindset around research and discovery. Released next week.
