One caution—When considering doing unmoderated user research, it’s important to keep in mind that unmoderated user research is never as good as moderated user research. You should always avoid attempting to replace necessary moderated user research with unmoderated user research.

One huge fallacy we sometimes encounter is the belief that some user research is always better than none. Unfortunately, this is completely untrue. Improperly conducted user research can lead to bad decisions about product direction that can result in your inaccurately defining a product’s target market, defining the wrong key functionality for a product, or designing poor user interfaces. Each of these issues is enough to doom a product to failure when you release it to the market.

To compound this problem, often decisions that are based on the findings of user research—regardless of its soundness—receive more trust than they deserve, so they are less likely to be challenged and corrected than if you’d conducted no user research. There are many ways in which user research can go wrong, but we’ll save that for another column. For now, we’ll focus on ways of making good use of unmoderated user research tools.

Beyond Usability Testing

Unmoderated user research tools tend to focus on usability testing, but there’s no reason why you can’t use some of these tools for performing unmoderated concept testing or even miniature ethnography studies. For example, you could construct tasks along the lines of Please demonstrate how you would make a purchase from your favorite online store. 

While unmoderated user research does not replace moderated user research, it can be very effective in augmenting moderated user research. For example, generative user research such as ethnography can be extremely costly, but a company can hold down costs by performing ethnographic research with fewer participants, then supplementing their data through unmoderated research sessions. 

When performing user research, we look for trends. It’s very important to distinguish between behavioral trends and idiosyncratic behaviors when determining design recommendations. Distinguishing between trends and idiosyncrasies requires many participants—a major factor affecting schedule and budget. Unmoderated user research can be an effective and low-cost method of obtaining the data that lets you make this distinction. You can use moderated sessions to identify and thoroughly understand the behaviors that are of interest. Then, to verify the trends you’ve observed, look for those same behaviors in unmoderated sessions. It’s best to follow this rule: Do not use the unmoderated sessions to identify additional behavioral trends, because the understanding you can glean from an unmoderated session tends to be superficial. 

Combining limited ethnographic studies with unmoderated user research isn’t as effective as doing a full ethnographic study, but it is a way for cash-strapped startups to get some invaluable consumer insights. This approach of augmenting your moderated user research by involving larger numbers of participants through unmoderated sessions works with nearly any form of user research. 

Longitudinal Usability Studies

Unmoderated user research enables some innovative approaches to usability testing. One of these is longitudinal testing. Using a longitudinal approach to usability studies, we can learn how a person forms a long-term relationship with a product by getting data from people over an extended period of time. 

This approach lets us explore different levels of usability. Most usability studies that evaluate iterative designs focus heavily on discoverability, but a longitudinal study can also acquire data about the learnability and ultimate usability of a product once a user has become fully accustomed to its user interface. 

To get a more complete picture of a product’s user experience, it’s useful to pair your data from longitudinal testing with another longitudinal data source such as diary data. The unmoderated testing sessions provide data that is similar to diary entries describing interactions, while the diary data indicates participants’ goals in using a product, their perceptions of the product, and the nature of their relationship with the product. For example, diary data might indicate that usage is extraordinarily high at first, but then dies out as the product loses its novelty, indicating a need for continuously updated content. Conversely, diary data may indicate low early usage, followed by an explosion in long-term usage, indicating usability issues affecting adoption. 

For longitudinal unmoderated usability testing to be effective, it has some special requirements. First, the automated testing software must allow you to select your own participants, so you can test the same people multiple times. Second, you need to determine a testing schedule that would allow you to observe the factors affecting change. For example, to obtain data about learnability, you must schedule a test session during the critical period when a participant might be having difficulty learning a new user interface. Too early and you would be testing discoverability; too late and the participant would already have learned the user interface. Determining the proper frequency of testing depends on a participant’s frequency of use of the product. Thus, a higher-use product requires more frequent testing. 

It might be helpful to give participants some basic structure for how they’ll use the product during testing—for example, engaging with the product for at least 30 minutes each night. However, the trade-off for this kind of structure is your ability to examine participants’ organic usage of the product. This kind of study can provide some amazing data—of a kind that rarely gets captured at present—but you’d need to tailor your testing method to each product. 

Conclusion

This column has described just a couple of creative, innovative ways in which you can use unmoderated user research tools. We’ve described how by using unmoderated user research tools 

• you can augment the data you’ve acquired through moderated research sessions, reducing the number of participants you need for your moderated studies when identifying trends
• you can perform research that requires repeated measures—for example, longitudinal studies that explore the long-term relationship between a user and a product

However, there are some constraints on the use of unmoderated user research tools that you should be aware of. For one, these tools currently work only for computer-based software. Therefore, they may not be useful to anyone developing software for a mobile platform or for hardware. At some point in the future, these tools may become available for mobile platforms, but we are unaware of any such solutions at present. 

Unmoderated user research sessions are always inferior to moderated sessions—all other factors being equal—so use them wisely. And since it’s untrue that some user research is always better than none, it’s extremely important that you should become a conscientious and discriminating consumer of research. As we mentioned in our column last month, a little education can be extremely valuable in this regard. A thorough education in research methods also enables you to innovate new research methods that can open up new ways of understanding users. 

The two applications of unmoderated user research tools we’ve described in this column illustrate the general concepts behind combining unmoderated sessions with other forms of data to get a more complete picture, but there are likely other effective methods, too. We invite you to share your ideas for other useful methods of using unmoderated user research tools in the comments

Mobile Upload by: Bryan McClain

Thoughts?

Mobile post by Bryan McClain

I was out for walk during lunch in downtown SJ and I was happy to see families enjoying the water fountain area setup for their enjoyment on these hot afternoons. It makes me wonder when more cities will design for the people and not just for aesthetics. Thoughts?

Mobile post by: Bryan McClain

Challenged with the task of creating a bag company that can differentiate itself from the pack while maintaining competitive costs, Mark arrived at a solution that involved low oversees labor costs, the customization of in-house assembly, and innovative bag design. The result is a rugged bag with a waterproof interior lining, plenty of pouches and slots, stands up on its own, and has a completely customizable outer skin, all for the prices comparable to anything you’d expect from Timbuk2 or Chrome. For the same price that you would spend on a pre-built messenger bag from one of these manufacturers, you can have a bag from Rickshaw that has your company logo, your child’s picture, or any pattern or even fabric that you can think of (Mark recommends upholstery-grade fabrics), as if you had it custom made just for yourself. During our visit to Rickshaw, we were able to see bags with conference logos being assembled and other bags with hand-woven Southeast-Asian fabrics. Rickshaw’s ability to cost-effectively integrate amazing flexibility in customization is breathtaking.

Mark accomplished this feat through innovative design, not just of his bags, but also of his manufacturing process. He recognized a fundamental challenge with the messenger bag industry: producing custom bags in the U.S. was expensive and limited production numbers and producing bags overseas limited customization and required large shipping and storage costs. His solution to this challenge was an innovative fusion of the two manufacturing options that allowed him to leverage the strengths of each approach, while minimizing there liabilities. By borrowing a page from the automotive industry, Mark was able to design a bag “chassis” that could be built in China for a low labor rate, these chassis could then be shipped, in two pieces, to San Francisco, where his in-house factory labor assembles the chassis and then applies a graphic skin to finish the product. The fact that the chassis pieces were shipped unassembled, allowed them to be packed flat, reducing shipping volume as well as storage requirements. The ingenuity of such a process is quite impressive, as is Mark’s enthusiasm in describing it.

We realized that it took a unique kind of person to create such a product. We also noticed that Mark’s situation was unique. After spending several years running a successful messenger bag company, he chose to start a new messenger bag company, using his own money! Surely, the fact that he put his own cash on the line helped to grease the wheels of creativity, but it was also his well-rounded background that allowed him to design and engineer not just a bag, but a manufacturing process to maximize his business returns. Indeed, some of the hot topics in the industry have been empathy, cooperation, and teamwork. In Mark, we found a one man team, and it really illustrated the value of understanding the business aspects of design. It hasn’t been unusual, in our experience, for us to come across truly inspired design that didn’t make it to market because of the costs associated with actually building the product. It takes a designer that understands manufacturing processes and associated costs as well as business logistics to overcome these challenges. It’s up to each of us in the design field to do our best to understand the tasks faced by all of those around us in order for us to help produce the best products in the market.

Bryan McClain and Demetrius Madrigal