Embracing the Distance: Strategies for Successful Unmoderated User Research

Over the last several years, remote user interviews have become a staple for many product teams. It has truly never been easier to talk to users. While we celebrate this forward momentum, a key variable that causes product teams considerable pain is — humans and our competing priorities.

Online user interviews have allowed us to get feedback from consumers thousands of miles away, but it doesn’t remove the steep time commitment that you have to invest to connect with them. You are required to align your calendars and block out time to meet with your participants; organize your time and prepare for a conversation that will help drive product discovery and get to the bottom of your riskiest assumptions. These activities will often be followed by frantic adjustments for the inevitable. A conflict arises just before your scheduled interview session. Things happen to keep well-intentioned participants from joining you or even worse a 5-alarm fire breaks out and diverts your attention away from your study entirely.

These operational pitfalls of live person-to-person research can be limited but never fully eliminated. That’s where unmoderated user testing has reemerged as the shiny new thing for many.

This method offers a unique blend of capturing insight paired nicely with logistical convenience that is undeniably alluring. Researchers can create workflows and tasks for consumers to complete on their own, all while using tools to capture verbal responses and even monitor how they engage with your product. Think of unmoderated user testing as a no-hassle buffet for a researcher. You set the platter and your customers serve themselves, all the while you sit back and watch their choices.

However, like any tool, its power is only found when you wield it correctly. The value of unmoderated testing shines through when you consider the breadth of potential consumer feedback it can provide but temper it with intentional practice. The lack of a real-time moderator means you need to be attentive, prepared, and strategic with its use to ensure you are capturing feedback you can use.

To make the most of your testing efforts, we have found it crucial to focus on five key areas: recruitment, instruction, redundancy, stimuli, and synthesis.

📣 Recruitment

To ensure you get the most accurate feedback from your target audience, it’s essential to focus on recruitment. Invite a diverse group of participants you are confident have relevant experience with your subject matter. However as you source and select individuals, ensure that your test will lead them to speak to relevant experience rather than speculate as to what others may do. Keep the testing window short and make sure participants can complete their tasks immediately after being invited. Work to capture their session when they are most engaged to limit the potential for them to simply forget about you! Stay available and present to troubleshoot or answer any questions that may arise. Unmoderated research is certainly more asynchronous. However, if a participant gets lost or frustrated without having somewhere to turn, you will conclude your study without the data you set out for.

🚏 Instruction

Instructions play a critical role in the success of unmoderated research. Start by building participants’ confidence by guiding them to a few quick wins while offering positive feedback. Consider doing this by including a sample task for the participant to complete at the beginning of their session. This task should be designed to teach them how to participate in the simplest way possible. If they will be using a prototype, encourage them to practice scrolling or walk them through how you would like them to structure their responses with an unrelated prompt. Doing this before they engage with the meaty tasks will help ensure your participants are primed and ready. At the end of the test, set clear expectations for participants including an acknowledgment that they have finished their tasks and when they may be incentivized for their participation.

♻️ Redundancy

Getting to the root of reasoning in a live interview often comes from asking follow-up questions to identify why a consumer thinks or behaves as they do. Without this ability in an unmoderated study, we rely on just the right amount of redundancy. Identify the areas you may need to know the most about. Dig deeper into these specific areas by asking the questions in varying levels of detail and by phrasing things differently along the way. Start broad by asking them to describe an experience or simply explore a feature. Then slowly narrow in by asking more pointed questions or by asking them to revisit a particular element and discuss it.

For example, if you’re studying the user’s perception of a new navigation bar within your app, this may look like asking the participant, “Describe what you see on this screen,” then follow with, “How would you use the navigation bar?” and finally, “If you were to search for an item, which part of the navigation bar would you use and why?” This helps encourage participants to provide more detailed and deeper responses. While doing this, know that an acknowledgment that things may be slightly redundant could be necessary to keep the participant engaged in some cases.

📱 Stimuli

The stimuli used during the test can greatly impact its success. When including product prototypes, ensure that you optimize the experience for the platform it will be viewed on, tailoring your design or prototype to align with platform-specific expectations. If you’re testing an Android prototype, make sure it resembles what the users would typically see on their Samsung or Pixel phones. If they are suddenly confronted with an interface that screams “iPhone”, they might spend more time figuring out the new layout than giving you the data you need. This will help eliminate distractions and allow you to accurately evaluate the experience without introducing unexpected variables. Keep the stimuli concise and limit the number of dynamic paths and questions your participant could travel. This will help ensure that you avoid overwhelming participants while increasing the chances that you will collect data; the kind that develops patterns when set beside the others completing your test.

🧫 Synthesis

The synthesis of the data collected is crucial to the success of all research, not just those that are unmoderated. In this form specifically, though, you can quickly become overwhelmed with data. Work to standardize and structure responses in your test to help aggregate and identify initial trends quickly. Favor oral responses when possible, as consumers tend to make decisions and evaluate products quickly. Avoid asking participants to write their feedback, as it can create a less authentic environment and shield you from more genuine responses. Set clear expectations with stakeholders and ensure they understand the focus and limitations of the research.

Conducting successful unmoderated user research requires a focus on five key areas: recruitment, instruction, redundancy, stimuli, and synthesis. By paying attention to these items, product teams can more effectively pull insights and feedback from their target audience without even having to show up. 😉

• Jack Cunningham

  Director, Product Management

  More from Jack