This month, we’ll take a look at the role that research plays in innovation—from generating ideas to refining them throughout the development process to smoothing the transition to mainstream success.
Inspiration and the Birth of Ideas

Sometimes we feel like we have the best job in the world. It’s true that we spend much of our time determining the best research approach to solve unique problems or pouring over data to identify trends, but we also spend a lot of time out in the field with people, learning about their lives and challenges in hopes of discovering ways to help them through the introduction of new and innovative products.

It’s through our research process that we gain a full understanding of users and, thus, are able to see opportunities for introducing new products into their lives. Whenever we see elaborate workarounds, simple processes that just take way too long, or users’ shoulders slumped in exhaustion or stress, we know that we’ve identified an opportunity.

Leonardo da Vinci also drew inspiration from the world around him. Da Vinci studied animals to understand how fish swam and birds flew. The insights he gained through research helped him to create early concepts for flying machines, parachutes, floatation belts, swim fins, and many other inventions. The Wright brothers would later duplicate Da Vinci’s early attempts at biomimicry in creating their own flying machine.

As much as many of us might hope that we could just sit at our desk and wrack our brain until we come up with the next great idea, the truth is that inspiration tends to come from the world around us. The camera grew from ideas garnered from watching light pass through a pinhole or the leaves on a tree. Thorns and living fences inspired barbed wire. The earliest form of the bicycle was called the dandy horse, recognizing the animal that inspired it.

Beyond biomimicry, there are numerous examples of technology drawing inspiration from the world. Every popular computer operating system uses a desktop metaphor to support user interaction. Craigslist mimics the newspaper classifieds that it’s currently overtaking. The shopping cart, which has become an ecommerce mainstay, takes its name from the physical carts that inspired their virtual counterparts. And social networks are modeled after—well, actual social networks. It’s easy to see how, over and over again, experience of the real world has inspired the technology that has reshaped our lives. But inspiration is only one part of invention. It’s not enough to have an idea, you have to execute it.
Growing an Idea

Over the years, we’ve seen a variety of concepts evolve from ideas to products to mainstream market conventions. In each case, a product matured thanks to the concerted effort of a team of people with different capabilities. A good product team almost always includes a UX designer who is responsible for designing the product, a developer to build it, a business person to create strategies for monetization, and a user researcher to support all other efforts. Research plays an important role in generating and developing concepts into products because of the information it provides. Whether market data or user data, information nourishes inspiration and helps us to evolve our ideas into fully realized products.

Research provides guidance and validation throughout the entire design and development process—all the way through to the release of a product and beyond. During the definition phase, research can help to determine a product’s feature set and differentiate between core features and added value. During the design phase, research results enable you to tell a story about who your users are and how they will tend to use your product.

Data from research helps to inform usage considerations such as the environment or conditions under which people will use a product. For example, if research indicates that the main users for a product would be working mothers, it should also indicate the design considerations for that group of users. That could include the need for quick engagement and multitasking, because a working mother is often juggling multiple demands on her time and attention.

Research can also help to validate a design as it develops through early mockups and concept testing. Acquiring user feedback on early designs before you begin spending time and money on development can save a tremendous amount of both.

Once you reach the development phase, user data on likely product usage can inform key platform and development decisions. Knowing where and how people might use your product can make the difference between, for example, your deciding to use Flash or PHP for the front end of a Web application. Making the wrong choice and starting down the wrong path, then having to reverse course could result in a very costly change in direction. Knowing a great deal about likely usage also allows you to set your development priorities. For example, knowing that working mothers tend to like using tablets, because they enable quick engagement, can help you to prioritize mobile over Web or desktop applications.

As you get close to completing product development, research can guide your marketing, distribution, and sales strategy. Knowing where your users consume content can help you to determine what your marketing content and ad placement should be. For example, you might find that working mothers like to visit celebrity news sites. Therefore, celebrity endorsements and ad placements on People magazine’s Web site might be a key element of your strategy to build awareness of your product among working mothers.

Like other aspects of product strategy, missteps with your marketing and advertising approach can be extremely detrimental to the health of your product and company. Throwing away valuable advertising money by pursuing avenues of marketing that don’t effectively build awareness of your product could leave you stranded, with empty coffers and lagging product adoption. For a startup, this could mean challenging attempts to recapitalize without demonstrating traction in the marketplace.
Conclusion

Research is an extremely important aspect of innovation—both in generating ideas for new and innovative products and in guiding the development of products from their inception to their release. Relying on research to generate ideas, inform decision making, and verify design and development directions is absolutely key to maximizing your chance of releasing a successful product. For startups, spending scarce funds on research to support critical design and development phases may seem like an expensive luxury. But we’ve encountered companies that were able to achieve success by partnering in equity relationships with freelance researchers or small research firms like ours. Establishing such partnerships can be an effective way of getting the guidance and validation that a successful product requires, without devoting a great deal of seed capital in research during the early stages of a startup.

Take, for example, the case of Netflix. Late last year, Netflix management announced that they would split DVD lending and video streaming into two separate services—coupled with a 60% price increase. The company quickly realized that they had made a mistake when customers cancelled their subscriptions in droves and Netflix stock lost 60% of its value. Netflix CEO, Reed Hastings, admitted that they had moved too quickly in announcing the service split and had fumbled the announcement badly because they had misunderstood their customers. Ironically, Hastings talked about performing research, but they had conducted only focus groups to help identify names for their spin-off service. If they had done more concept testing to ascertain customers’ readiness for the splitting of these services, they might have discovered that both content availability and the quality of streaming video are not yet sufficient to fully supplant DVD lending—or at the very least, that there would be massive customer backlash on the split they were planning.

Therefore, this month, we’re going to examine ways in which your UX research efforts could help your organization recognize whether you’re going down the wrong path.

You Don’t Know Who Your Users Are

If you’ve been developing a product or technology and are getting ready to put something in front of users, but you don’t know who your users are, you might be on the wrong track. We’ve seen this happen quite a bit with startups that have an engineering focus. A company may be pursuing an idea for a technology without thinking about a specific application or user need for it. We often refer to this as a solution in search of a problem.

This doesn’t mean you can’t find a problem that your technology can solve, but doing so is often quite costly. In addition to your doing the research that is necessary to properly segment the market, identify users, then understand those users, it is likely that you will have to make modifications to your technology for it to be viable in the marketplace. Such modifications could require significant backtracking or so much additional spending that making them would be cost prohibitive.

Your investment in product development is safer if you first conduct market research to ensure that you’re developing a technology that can return your investment. It’s true that there are companies that have been able to develop successful products without knowing their users or doing market research, but these companies seem to be exceptions to the rule. We highly recommend having a user clearly in mind before you commit financial resources to a development project.

You Can’t Find Your Users

If you have spent three weeks trying to recruit users and you can’t seem to find them, you might be on the wrong track. Even if you have particular users in mind, your inability to find them may indicate that they exist in very small numbers or may be difficult for sales, marketing, and advertising to reach. We’ve had clients who were attempting to target a specific user group that had a definite need, but a market that was just too small to address through a consumer product. For example, an application targeting fighter pilots and providing improved training might meet a strong need, but not have a large enough user base. Despite this significant need, a small and declining market of just 3,600 users meant that it would be extremely difficult to create a profitable consumer product for these users. Additionally, it meant that the population would be extremely difficult to recruit for user research.

Aside from there being a small market comprising few users, there are other reasons why a user population that is difficult to recruit might mean you are on the wrong track. Low-tech seniors, for example, are in need of services, but are difficult to reach through the most common recruiting channels such as social media, craigslist, email, or other digital communications. When recruiting low-tech seniors, you must often rely on friends and family and word of mouth. This difficulty in recruiting participants portends a similar problem when it comes to marketing and advertising the finished product to that market segment. While modern advertising has moved primarily to the Web, a company would need to rely on traditional media to reach this type of user. Purchasing advertising such as television airtime would be very costly for a startup or a company exploring a new vertical. These are all factors to keep in mind when identifying a market.

Your Users Don’t Agree on Anything

If you are working with a market segment and users are completely divided about what they want in a product, you might be on the wrong track. You might find that some users love a certain feature set or design direction, while other users hate that same feature set or design direction. In this situation, it becomes very difficult to determine the appropriate feature set or an effective design direction for your product. There’s a very good chance that you’ll alienate half of your intended users, but there’s a much more significant danger of losing all users by trying to please everyone.

If this happens, chances are that you are dealing with two or more unique groups within a market segment. Take the camera market, for example. One market segment comprises professional photographers, but within that segment, you might find significant differences in peoples’ needs and preferences. When investigating further, you might find that landscape photographers have very different needs from portrait photographers. In this case, you should further segment the market. Once you’ve done so, you’ll have a choice to make: either to try to design for both segments or to pick a single segment on which to focus. If you focus on one segment, obviously you are giving up the entire other segment and targeting a smaller market than you had previously intended. You’ll have to go back to your market research to determine whether this smaller segment is worth your investment. However, if you try to target both segments, there’s a danger of creating a product that doesn’t work well for anyone by attempting to incorporate wide-ranging features and design elements that begin to interfere with each other.

Your Users Don’t See Value in Your Product

If you put a product in front of users and they don’t tell you that they want it, you might be on the wrong track. Steve Jobs famously said, “People don’t know what they want until they see it.” This is typically true. People are often so accustomed to their own unmet needs and pain points that they can’t see them anymore. It’s for this reason that self-reporting through surveys and static interviews are unreliable sources of data.

If you really want to know whether users would see value in your product, you have to show it to them. Concept tests are a very effective means of doing this. A concept test involves showing users concept drawings, mockups, and/or prototypes of your product to determine whether they understand the value proposition and agree that the product has value. We recommend your doing concept testing early, incorporating the testing of concept drawings into contextual interviews and ethnography sessions. As a project proceeds, you can progress to testing early mockups, then functional prototypes.

You’ll get an early warning if users don’t understand the concept drawings for your product; or worse yet, if they understand the concept perfectly, but just don’t want such a product. If this occurs, it’s important to determine what you could do to clarify your product’s value proposition by asking participants what would increase or establish its value. Subsequent tests should determine whether any adjustments you’ve made have helped to instill value in your product and clarify its value proposition.

There is also the possibility that users just can’t grasp the utility of your product by seeing a concept drawing, but would by experiencing an actual finished product. While this might be true, the longer you go without users’ seeing value in your product, the greater the risk that you might be investing a great deal of money and resources in a product that is doomed to fail in the marketplace. Managing risk is ultimately a judgment call, but one of the primary purposes of UX research is to mitigate risk and keep companies from losing money. One successful application of research is identifying what projects to cut before making significant investment in them. That said, with the right information at the right stages of product development, most projects can pivot to become something that would be useful to people and, thus, be successful in the marketplace.

The Right Track

None of this means that, if you run across one or more of these phenomena, you are definitely on the wrong track. These are simply indicators that you might be in for some trouble. The more of these difficulties you encounter, the more likely the chances that you are sniffing up the wrong tree and need to make some serious adjustments to your product or even cancel a project entirely. Pivots can result in enormously successful products, because you’ve built them upon knowledge you’ve gained from an impending failure.

On the other hand, if you have a clear understanding of your users and are able to reach large numbers of them easily, and they agree on what they want from a product and can’t wait to get their hands on the finished product, you are almost certainly on the right track. Seek to achieve such a result through a close collaboration between research and design, and doing research will serve you well.

Technology As Fashion

For quite some time now, people have been decorating their notebook computers with stickers and using amazing photographs or graphics for their desktop wallpaper. Apple products have achieved their level of popularity as much for their visual impact as for their technological capabilities, attaining the status of fashion icons. More recently, HP has followed suit, developing its ENVY line of notebook computers, shown in Figure 1, which take some fairly obvious styling inspirations from Apple’s aluminum counterparts.

Figure 1—HP ENVY

Tablets and smartphones have also become fashion statements as much as they are technological tools. The fashion case market has exploded in the past few years, with thousands of artistic designs now available. People are paying hundreds of dollars for fashionable aluminum phone cases like the iPhone case from Element Case that is shown in Figure 2.

Figure 2—Element Case for iPhone

Because fashion is ultimately a matter of personal expression, people value new and innovative products that let them express their own sense of the aesthetic. Custom smartphone and tablet cases and skins let people cover their devices with just about anything they might want. Custom messenger and computer bags like those available through Rickshaw Bags—Figure 3 shows an example—let people choose any fabric or design for the construction of their bag. People can even avail themselves of custom etching services for personal electronics. Figure 4 shows an etching on an iPad. With so many companies offering products and services that provide fashion outlets for technology, the next step may be customizing technology itself for people.

Figure 3—A custom bag from Rickshaw Bags

Figure 4—Etching of a cherry tree on an iPad

Recently, we’ve seen the release of a wide variety of new over-ear headphones, all sporting the same basic technology. The popularity of headphones from companies like Skullcandy, shown in Figure 5, and Urbanears, shown in Figure 6, is as much about their visual style as their sound quality. With so many similar headphones on the market, these companies needed some means of differentiation to avoid being lost in the noise.

Figure 5—Skullcandy headphones

Figure 6—Urbanears headphones

Many manufacturers have aligned with popular artists for help developing and marketing their products—like Dr Dre and Beats headphones or Rza and WESC. Skullcandy has differentiated its products by providing advanced features like the ability to daisy-chain headphones together using detachable connectors. In contrast, Monster gained significant advantage by developing and releasing Beats headphones years ahead of the competition—successfully predicting that high-quality, fashionable headphones would become a significant market trend over time.

With quality over-ear headphones becoming a new fashion trend, other important factors are emerging—such as using fashion as well as features to differentiate products that are fundamentally similar to one another, building customization into the product design and purchasing process, and predicting what future technology might become a fashion trend over the next few years.

Technology Is Social

A major technology trend that we are seeing emerge is that of being social. While social technology has been a developing trend for many years, the trend is re-emerging in new ways, with new devices and connectivity. For example, over the past year or so, music has become social in entirely new ways through Web applications like Turntable.fm. By providing a social aspect to music rather than just access to a library of songs like the popular applications Spotify and Pandora, Turntable.fm has provided a uniquely sticky music experience. The social aspect of Turntable.fm helps people to leverage the social experience of sharing, discovering, and discussing music in a compelling, new way.

For the past few years, Zynga has been leading the charge in social gaming. Recently, one of their most viral games has been the wildly popular Words With Friends. Rather than just allowing people to play a game online in which involving friends is something of an afterthought, Words With Friends is an inherently social, person-to-person game that requires people to play with their friends—and even includes a chat function that enables trash-talking. Through Words With Friends, Zynga has found a way of connecting people directly—sometimes across vast distances—through a competitive and fun activity.

Breathing new life into an old activity by using new technology and leveraging a social experience, Hasbro has updated the game of Lazer Tag for smartphones, releasing Lazer Tag guns that operate with mobile phones. People can shoot their real-life friends with electronic signals that register on their friends’ smartphones. Rather than connecting people across distances through the Web or mobile networks, these new smartphone Lazer Tag guns connect people through technology much closer to home. It’s the kind of game that helps people make new friends.

Each of these technologies establishes new ways of connecting people, whether across great distances or right next door. They do so through established recreational activities that companies have repackaged for electronic delivery. Expect to see new social technologies over the next few years, leveraging the inherently social aspects of newer devices like tablets and connected televisions. Also, think about social activities in which we currently engage that are ripe for an electronic makeover.

Increasing Device Connectivity

Just a few short years ago, the only device users were likely to own was a computer of some sort, likely a desktop computer. However, as the years progressed, desktop computers became notebook computers; then smartphones gained widespread adoption. Right away, transferring data between smartphones and computers became a pain point for users, and easy ways of syncing contacts, calendars, and email were in high demand.

Now, with the introduction of tablets, ultrabooks, connected televisions, game consoles, and cars, we are dealing with more and more data-housing devices and risk our data becoming increasingly fragmented. In response, services like mint.com have sought to consolidate information from multiple sources into a single location and cloud solutions like Dropbox have begun to blur the file-location barrier and provide access from any location or device. However, cloud storage limitations, as well as reliability and connectivity questions are preventing a wholesale migration to the cloud.

Amazon seems to be setting itself up to tackle this problem through their advanced cloud solutions, growing out of Amazon Web Services. Through Whispersync, Amazon provides an excellent model of how to tear down barriers between different devices, having already accomplished it beautifully for ebooks. With Amazon’s forays into music, movies, and now tablets with the Kindle Fire, they seem to be positioning themselves to own content delivery across platforms.

Apple, Microsoft, and Google may have something to say about that, but they are behind when it comes to device connectivity. Apple just recently released iCloud, but their history with connectivity between devices has been mixed—with iTunes providing clunky interactions between smartphones and computers. Microsoft has provided significant device connectivity between computers and TVs using Windows 7 and the Xbox 360, and Windows 8 promises to do even more. Google has made significant moves with their Google Apps line, providing useful quick calendar syncing across platforms, as well as shared access to documents, spreadsheets, and calendar events.

Over the next few years, look for each of these companies to jockey for position in the market by creating a seamless flow between devices and users. Content delivery will be key, especially with the storage landscape changing rapidly, becoming cheaper and cheaper, and connectivity becoming faster and more omnipresent. The ultimate solution will likely be a cross-device platform that is similar to the latest versions of Android—for smartphones and tablets, Android’s Ice Cream Sandwich—and Windows 8 for PCs and tablets. Both are optimized for multiple devices. These large companies will likely be looking to acquire companies that excel at solving data fragmentation issues and establishing the seamless flow of data that they all hope to create.

Conclusion

User experience is about more than just developing new technology and perfecting it. It’s also about paying attention to what users do and want to do with existing technology. Steve Jobs was not a visionary because he created new technology; he excelled at seeing how new technology fit into the lives of consumers.

By understanding users, trends that are emerging in user behavior, and future technology directions, we can set ourselves up to be at the forefront of innovation. By understanding patterns in people’s current and past behavior, we can do more than recognize their current needs; we can anticipate their future needs. Identifying recent trends has led to fashionable products that bring people together and operate seamlessly across devices.

If you are familiar with various research methods and their applications, your organization’s business goals, and product development constraints such as schedule, budget, and the development process, you can create an end-to-end user research plan to cover an entire product development cycle. Having a user research plan helps you to establish and communicate research goals and expectations regarding logistics and set research milestones for a product development effort.

Research Phases

For a User Experience team, the key phases of a product development cycle are Discovery and Design. Discovery consists of defining the market for a product, its potential users, and product requirements. Common user research approaches during the Discovery Phase include survey research, focus groups, expert interviews, and ethnography.

The Design Phase involves bringing a product to life through product or user interface design and refinement. User research during the Design Phase focuses on gaining a deeper understanding of users, confirming research findings from the Discovery Phase, identifying the relative value of various features, and refining a design as it evolves. Common user research methods during the Design Phase include concept testing, usability testing, and competitive benchmarking.

Understanding the different challenges you’ll need to address during these different phases allows you to plan your user research effort throughout a development cycle.

Planning ahead of time lets you prepare for user research studies in advance, reducing the time and effort you must devote to a given research activity. For example, once you’ve identified your target users, you can start building participant lists and networks to facilitate recruiting. When preparing for your first study, you must establish communication with many potential participants and select study participants from that pool of participants for your study. Once you’ve established that participant pool, it becomes easier to find participants for subsequent studies, simplifying your recruiting efforts, reducing the man-hours you must devote to contacting potential participants, and shortening your research schedule. Over the course of multiple studies, this preparation pays enormous dividends.

Discovery Phase

The main goal of the Discovery Phase is to define the product your team is developing. To do that, you must understand the market and the needs of users within that market. Therefore, the first step in planning user research is understanding your team’s information needs. Your organization’s business goals always drive its need for user research. Understanding these goals and needs is a little more difficult during the early part of the Discovery Phase, before you’ve even defined a product concept.

At this early stage, your business goals tend to focus on serving a given market. So, through your user research, you should endeavor to understand that market, define its segments, model the users belonging to them, and learn about the needs of those users. Surveys are an effective way of getting an overview of a market and understanding its characteristics. Through the results of a survey, you can define your market size, identify market segments, and define the demographics of the different segments. However, surveys can tell you very little about user needs and behavior.

Following your survey research, you’ll need to learn more about users and their behavior. Your best option for identifying users’ needs is some form of contextual user research. Performing any kind of needs-finding research that uses self-reported data—including interviews, surveys, and focus groups—is notoriously unreliable. People have a great deal of difficulty being objective about their own lives, and they tend to be blind to the workarounds they employ in meeting their needs.

If you ask people what they need, they’ll tell you a better, faster, and cheaper version of what they already have. Steve Jobs was famous for saying, “People don’t know what they want until you show it to them.” Contextual research approaches such as ethnography or interviews that you conduct within the context of product use, whether at home or in the office, circumvent this difficulty by letting you directly observe users. This allows you to identify needs from their actual behavior. This kind of research can also provide quite a bit of information about a user’s environment and behavior, which can help you to further define a product.

Once you’ve identified a need, the next step is determining the best way to meet that need. At this point, you define the product. The product’s primary features usually target meeting the identified user needs in some way. The nature of the needs often determines the form the product takes. For example, if a need tends to arise when a user is in a coffee shop or walking around outside, the product would likely be a mobile app or a portable device. When defining a product, user research focuses on learning more about current user behavior and verifying identified needs through additional interviews, focus groups, and other forms of generative research. Finally, you must gain as deep an understanding of your users as possible to inform design. This includes identifying additional desirable features and defining the scope of a product’s complete feature set. It is during this phase of research that you might decide to develop personas.

Design Phase

The goal of the Design Phase is to create the design for a product. User research during the Design Phase focuses on gaining an understanding of users and their needs to inform design. As designers develop concepts, you can do concept testing to evaluate whether a design adequately meets users’ needs and creates an effective vehicle for delivering a solution. Activities such as collaborative design sessions and card sorting let you gather ideas about how to design a product to best meet users’ needs. This research also helps you to prioritize a product’s potential features—again to inform design.

As a product’s design evolves, usability testing can help to ensure that its user interface enables users to realize the benefits of the product. Usability testing should be iterative. Include some directed tasks during concept testing to ensure that you discover usability problems early, before the development of a user interface element begins. Doing iterative testing helps you to ensure that any design changes your team has made in response to prior tests have successfully resolved the issues.

Competitive benchmark testing allows you to see how your product stacks up against its competition or against your alternative designs for a user interface. During this period, you should be able to identify competitive differentiators for your product. These are the attributes that make your product different and better from the competition. This research informs marketing as well as design.

Planning Ahead

Generating an end-to-end user research plan can help you to save money by leveraging the findings of and resources from previous studies. As you envision the product development cycle, create a rough user research plan at the very beginning, then refine your plan as additional findings come to light. Start with a general plan that segregates Discovery and Design into separate phases. Then identify what you already know about the market, users, and their needs. If you are familiar with a market through your prior research, you might already have sufficient data about that market, so you can probably skip some of the survey research. Beyond that, build a framework that is flexible, but well structured. Figure 1 shows an example of user research framework for a Web product.

Figure 1—User research during Discovery and Design

Conclusion

When employing an end-to-end user research plan, make sure that you respond to data as it emerges. For example, if important data is not coming to light through interviews, you may want to try another approach to acquire the information you need—for example, journaling or ethnography. It is important to utilize the proper research approaches to satisfy needs as they arise. So fill out your tool chest by making sure that you are familiar with as many research approaches as possible.

Remember, concept tests and usability tests should always be iterative. By planning your user research in advance, you’ll be able to perform this testing relatively quickly by reusing test plans from previous studies and drawing from the same pool of participants when recruiting. With proper planning, you can integrate user research smoothly into the development cycle, including agile development approaches, without disrupting schedules. Good user research helps to mitigate risk and ensure that your product is successful upon its release. Consider doing post-release research that your team can apply as it continues improving a product or plans a follow-up release

In much the same way governments use intelligence in making military or diplomatic decisions, user research provides the information that product designers and stakeholders need to make decisions. If a government’s intelligence is accurate, leaders can make good, well-informed decisions. However, if the intelligence is inaccurate, leaders might make costly decisions that accomplish little or have negative consequences. Think of user research as design intelligence that helps product designers and stakeholders make informed decisions regarding product direction. Inaccurate design intelligence tends to lead to bad decisions, even when the best designers and savviest stakeholders make them. So, what are some of the ways in which user research can go wrong?

The Wrong Users

Perhaps the most serious problem that you might encounter is user research with the wrong participants. When you do user research with participants who are not representative of your target users for a product, any and all of your findings are likely to be incorrect. Information from the wrong participants can lead to designing a product for the wrong population of users. These could be people who are unlikely to adopt the product, do not experience the need that the product attempts to address, or are not qualified to provide accurate feedback on the product.

For example, early in our research careers, we assisted in a research study for which others defined the protocol and performed the recruiting prior to our involvement. We helped only to moderate the sessions. The study focused on how people drive and technologies that could help them drive more effectively and safely. The problem was that several of the participants did not drive on a regular basis, but instead took buses or carpooled. A couple of them didn’t even have their driver’s licenses! Any data that you gather from the wrong participants, as in this example, would not be representative of your actual, intended users, so making decisions based on that data could take you in the wrong direction.

The most obvious cause of doing research with the wrong users is making mistakes in recruiting research participants. To prevent this, make sure that your recruitment screener includes both descriptive and behavioral questions. For example, if you are recruiting Web developers, ask them to describe their work and what programming languages they use. An appropriate participant who is a Web developer should be able to tell you, for example, that he primarily does back-end development and uses JavaScript as his primary programming language. If you’re recruiting users who develop mobile games, be sure to ask what type of phone they own, whether they play games, and what their favorite games are. At the beginning of each research session, start off with similar questions to double check that you’re doing research with the right participant. Even if you find out too late that someone isn’t right for your study and have to scratch a participant, it’s better than getting inaccurate data.

A less obvious cause of doing research with the wrong participants is inaccurate user profiles or personas. If the research that identified users was flawed, it can result in personas that don’t accurately describe users. Factors such as small sample size, idiosyncratic users, too much focus on extremely salient findings, and investigator bias can lead to inaccurate conclusions. If you do not correct such problems through a review process or follow-up research, you can develop personas that don’t accurately reflect your users, and these inaccurate personas can start a domino effect, resulting in inaccurate research throughout the complete development cycle for a product.

The Wrong Questions

Understanding a problem you’re trying to solve is fundamental to meeting user needs through product design. Conversely, failing to sufficiently understand the problem tends to lead to an inability to solve it. The biggest obstacles to gaining this understanding are your own preconceived assumptions regarding users, their needs, and the product. Researchers tend to form their research questions around their assumptions, so our assumptions are the foundation for all the knowledge we gain through user research. If your assumptions are incorrect and you never realize it, everything else that you discover will be flawed.

For example, according to Guy Kawasaki, the prevailing assumption in the early 80s was a “better, faster, and cheaper MS-DOS machine.” If Steve Jobs had never challenged this assumption, the Macintosh would never have revolutionized personal computing with its GUI and desktop metaphor. But he did, and Apple competitors that were committed to MS-DOS soon realized that a market dominated by the Mac, then Windows had left them behind. More than that, they missed out on the chance to take the lead by thinking outside the box and creating their own innovations.

Unchallenged assumptions are dangerous, so it’s always important to challenge your assumptions in your research. When we’ve worked with the military and the FAA—organizations where lives can depend on the accuracy of our research—they’ve required that we document all of our assumptions in our test plans and final reports. Doing this helps with interpreting the findings and judging their reliability. We advise our clients to engage in a similar exercise. We recommend that you organize a collaborative session in which designers, engineers, researchers, and stakeholders try to get all of their collective assumptions on a whiteboard, discuss each one to determine the safety of each assumption, as well as the need to test them and an effective method of testing them. Incorporating assumptions into existing research efforts is relatively easy, only marginally increases cost—if at all—and pays off in big ways.

The Wrong Conclusions

Once you have recruited good research participants and obtained good data by asking good research questions, the next step is interpreting the data to arrive at accurate conclusions that drive product and design decisions. However, interpreting data sometimes leads to inappropriate conclusions, which tends to occur for two primary reasons: investigator bias and being overly aggressive in interpreting data.

Investigator bias can be very difficult to overcome. Biases tend to surface in the form of findings that seem like they should be right or that you expected to see. They become a problem when the data doesn’t support your conclusions. At times, research data may refute your conclusions, but in a way that might be too subtle to easily recognize. As the investigator, you may not be aware of your own biases, which can make it extremely difficult to guard against this problem. The more deeply you’ve worked within a subject area, the more deeply ingrained your biases become.

One effective method of overcoming your biases is to work with a partner who can act as a check against your biases. A good partner is someone who is knowledgeable about user research methods, but isn’t deeply involved in the project or subject area of your research—and thus can maintain objectivity. Everyone has their biases, so it’s important that you serve as a check against your partner’s biases as well.

Biases are closely related to the other source of inappropriate conclusions: interpreting data too aggressively. This can happen when a participant reports a particularly interesting and salient point of data—the kind of data that really gets your attention, usually because it’s something that you haven’t thought about before. However, for this very reason, it’s likely that you haven’t systematically tested the finding across multiple users.

You have to be careful not to rush into turning such a finding into a recommendation without fully vetting it. Our rule of thumb is that once is an outlier, twice is a coincidence, and three times is a trend. You want to look for trends to report rather than outliers.

You might find that someone has a really interesting idea, but upon investigation, most other users hate it. This doesn’t necessarily mean that you have to discard the idea, but it does indicate that it is a much more complex topic than you might originally have perceived. This is something you should be aware of before recommending the idea to stakeholders—or the product team may implement the idea to the detriment of the product and its users.

During your research, if you uncover something interesting with a given participant, try to assess your discovery with subsequent participants by adding it to your research protocol. If this isn’t possible, in your final report, document it as a new research question to explore further and recommend including it in follow-up research.

Conclusions

This month we’ve taken a look at some common reasons for faulty and inaccurate research findings that could end up threatening a product’s success. There are other threats—ways in which research can go wrong—but these are the most common ones and tend to be the most damaging. It is extremely important that the consumers of user research are able to recognize the difference between good and faulty research to maximize their chances of success in product development. The best protection against faulty research is to have a good research team and a good research plan. Next month, we’ll get into planning effective research for an entire product lifecycle and how doing the right research at the right time can help to ensure that you are getting good product intelligence.

If you find yourself in such a position, there are some key principles you should keep in mind that you aren’t likely to discover in any textbook or research manual. Learning any skill involves both knowledge from books and what you learn through some form of personal mentorship in which you can receive feedback on your work. Research is a complex skill, and people who are just getting started need both knowledge and guidance to gain mastery. We’ve tried to distill what we think are the most important concepts that we emphasize when we’re mentoring new researchers. They are principles that we have discovered through our years of experience, and we’ve found that they apply in all instances.

Make Friends

Research is a social activity. Many people have a vision of researchers as stoic intellectuals in white lab coats, sitting and observing people from behind one-way mirrors. The reality is quite different. User research is a process in which you communicate with people so you can learn about their lives and their needs. In the long run, forming an understanding of the people who will be using your product is much more important than just knowing their individual thoughts on a design concept or user interface.

The only way that you’ll be able to gain this understanding is if you can put participants at ease and talk to them as real people rather then just using them as a means of improving your product. The best way to connect with participants is to take your time and get to know them as people before you dive into your research protocol. Ask them questions about their lives, their jobs, and how their day is going. You need to get to a level deeper than just superficial chitchat. You want your communication with research participants to feel like you’re talking to a good friend.

Keep in mind that you have two goals: first, to help a person feel relaxed and comfortable, so he or she will open up and communicate freely; second, to learn as much as you can about their lives. This understanding can provide invaluable insights that can help you to better form your questions, as well as interpret their answers. For example, if a person reports that he doesn’t see any value in a product and you know about his life, you can ask him about potential use cases that he may have overlooked. If someone is being vague in his answers to your questions, you’ll know how to direct him toward making a definitive statement by applying the concept to a specific example in his life.

More specifically, if you were testing a product that lets users find their parked car, but a participant mostly rides his bike and parks his car for days at a time, you might ask whether he sometimes has difficulty remembering where he’s parked his car three or four days ago. The participant might not have immediately thought about applications for this product because he doesn’t drive often, but by suggesting an important use case, you may help the participant look at the product from a different perspective. This data can inform both design and marketing strategies.

Don’t Answer Questions

The purpose of research is to gather information, not to provide it. When people ask you questions, your instinct is to try to answer them, but you must resist this urge, because it will interfere with your ability to get accurate and actionable data.

I’ve seen people doing user research explain a product or user interface to a participant, including all of its features and how it operates. This prevents your having the opportunity to get a participant’s immediate reaction to the product. Instead of explaining a product’s value proposition to a participant, ask the participant What do you think this is? What do you think you would use it for? This lets you get an idea of how clearly a product conveys its concepts. If they are unclear, you can then explain the product and ask participants how you could make the ideas clearer.

When testing user interfaces, present participants with scenarios for tasks that would motivate them to try to figure out how to use the user interface properly on their own. Here’s an example: You’re moving into a new apartment, and you need to sell a couch that you aren’t going to take with you. How would you go about doing that? If a participant asks, Do I go to auctions? Don’t answer the question! Just note his response and tell him to feel free to try things. If a participant notices something in a user interface and asks you, What’s this over here? you should respond, What do you think it is? What would you expect it to be? Try to be like a therapist and always answer a question with a question.

It is more important for you to get a sense of participants’ impressions and reactions to user interface elements than for them to understand every aspect of a user interface. Try to keep in mind that customers won’t have you sitting next to them in the real world. It’s important to try to replicate that reality during research.

Don’t Try to Sell a Product

When you’re doing research, you want to make sure you get an idea of the value that a product provides to users. In fact, conscientious researchers would recommend radically altering or even cancelling a project if they learned that users would not want it or use it. If you point out all of the great things about a product, people will tend to agree with your opinions rather than form their own and tell you about them.

As you are going through your research protocol, make sure you document participants’ natural reactions in the course of their first exposure to a product. As we mentioned earlier, try to see whether they can understand the product on their own, then examine their reaction to the product’s value proposition. You’ll tend to see a variety of reactions, ranging from This is amazing, it’s going to change my life! to I don’t think this would do anything for me at all. Be sure to obtain this information—or you could be in for a significant beating when you release your product to the market. If you identify problems early, you’ll have the opportunity to pivot before devoting any more resources to something that is unlikely to take off.

To do this successfully, you must establish your impartiality. We actually inform people that we didn’t design a product and that we are performing the research to provide an objective opinion. We also inform participants that the goal of a research study is to improve the product rather than to make sure that we are on the right track. This helps to put them into a mindset of providing constructive feedback. As a researcher, you should foster this mindset and try to avoid influencing participants.

Conclusion

In this month’s column, we’ve covered three important guidelines that you’re most likely to learn on the job rather than in a textbook or manual. The higher-order guideline is to avoid influencing participants and keep your opinions to yourself.

Remember, the most important thing to gain through user research is a deep and meaningful understanding of your potential users. This can guide you through all aspects of development, including coming up with marketing and advertising strategies.

Of course, there is much more for researchers to learn, including how to generate actionable recommendations, fit into an agile development cycle, build compelling presentations, and communicate effectively with stakeholders, but those things are often better experienced firsthand. If there are any other essential guidelines that you think should be covered, we invite you to share them with everyone in the comments.

In this month’s column, we’ll discuss both the value of working with a professional recruiter when doing user research and best practices for making this a smooth process.

Why a Reputable Recruiter Is Essential to the Research Process

When conducting research, it is very important that the population you’re examining ties directly to the product or service that’s under development. If you sampled the wrong user population, the results would guide the design and engineering teams down the wrong path, leading to the development of a product that very few people would use.

When you hire a reputable recruiter to assist with participant sourcing, they use a highly structured, well thought out, and detailed screening process to ensure that they pull the right types of people into the study. The recruiting process is a partnership between the researcher and the recruiter. For example, while a researcher might request participants who have limited smartphone experience, the recruiter might refine that request, focusing on people who don’t own or operate a smartphone on a regular basis and, thus, helping to more specifically define the intended participant. As a result, the participants who are in the study will understand the concept of a smartphone, but won’t own one. It’s such tiny details as these that play a role in acquiring the right people and ensuring your study’s success.

Building the Screener

Typically, a recruiter requires you to develop a participant screener that outlines the types of people you want to recruit. The screener includes information such as age, education, income level, and experience with certain technologies. It is also very important to add behavioral questions such as How many hours per day do you watch television? or How many times a week do you go shopping? A good recruiter examines your screener and asks clarifying questions to further refine it before starting the recruitment process. This includes examining your questions to determine whether some are too strict, making it hard to gather together enough participants, or too loose, resulting in too wide an array of participants.

The screener also includes the incentives participants are to receive for helping with your study, and the recruiter will help you to determine the right amount for the incentive to ensure that you’ll get the people you need. If you provide too small an incentive, people might feel it’s not worth their time to participate. However, if you provide too large an incentive, you’ll find some people simply want to collect the cash and don’t care about helping you refine the design of a new product you’re building.

Locating the Participants

Using Craigslist is definitely an option for recruitment, but most reputable recruiters have an extensive database of participants they can contact at a moment’s notice. Their databases indicate what participants respond well during sessions, deceive or lie to researchers, have been no-shows for sessions, or have been involved in too many studies, so are becoming professional participants. Recruiters’ databases also help identify which participants are very communicative and which tend to provide only superficial data. Some of these databases are extremely impressive and powerful, allowing recruiters to gather high-quality participants with surprising speed.

When necessary, a recruiter might use Craigslist to find participants, but they’ll use a rigorous screening process to ensure the participants truly meet the needs of the researcher. Unfortunately, it’s not uncommon for participants to deceive recruiters and researchers just to be part of a study and collect the incentive. However, recruiter databases can help you to avoid this by tracking these kinds of participants and eliminating them from consideration during future recruitment efforts.

Without recruiters and their participant databases, you’ll often be gambling with your recruiting—significantly increasing the likelihood that you may inadvertently recruit a participant who is interested only in collecting the incentive. When research sessions go to waste because of bad participants, you end up wasting a great many resources, including the researcher’s time, the cost of the incentive, and the opportunity cost of not obtaining good data.

Participant Management

A reputable recruitment firm ensures that participant management is part of the recruitment process. Most firms include this as part of their core service offering. Participant management includes calling the participants two or three times in the days leading up to your study to ensure they actually show up, handling cancellations, and dealing with any last-minute issues such as shifting session times. Usually, for a small additional charge, a recruitment firm can also mail out participant incentives, so the researcher does not have to deal with money on the day of the study.

Getting participants to show up for a study is much more difficult than it sounds, which makes participant management a great deal more important than you might expect. People who agree to participate in research usually have good intentions, but when they get stuck at work or have sick kids or car problems, you’ll need a recruiter who quickly notices such issues and can find replacements at the last minute. We’ve found that, with the right recruiter and good participant management, the percentage of participants who actually show up for studies tends to be around 90 percent; without participant management, its closer to 60 percent. This part of the process can make or break your study.

Best Practices When Working with a Recruiter

These guidelines can help you to successfully engage a recruiter and keep the recruitment process moving smoothly.

1. Discuss the types of participants you need with your team before you contact a recruiter.
2. Be ready to quickly build a participant screener when you engage a recruiter.
3. Provide the recruiter with any personas or other documents that describe the product’s intended users.
4. Discuss the screener and your research goals with the recruiter to give him or her a thorough understanding of your needs.
5. Respect the opinions of the recruiter, because they work with researchers and study coordinators all the time.
6. Address issues with individual participants with your recruiter, so they can record them for future reference.
7. Make sure your recruiter provides up-to-date participant schedules, including participant contact information.

Conclusion

Professional recruiters are an extremely valuable part of the research process, so if your budget allows you to use one, we highly recommend that you do. As we said earlier, getting the right participants is just as important as conducting the research. Getting feedback from the wrong participants can have huge negative consequences for your product and may even lead to its failure upon its release. Even if your team discovers problems with a design that is based on faulty research prior to a product’s release, solving them is often very costly and time consuming.

If your budget does not allow you to hire a recruiter, we recommend that you create a very detailed screener for your screening process, and ask people who have a lot of research experience to review it for you. Doing so, helps to ensure that your screener is solid, so you’ll tap the right people to help out with your research.

When you find recruiters who do a good job of finding quality participants for your studies, make sure you hold on to them. They will become your best friend.

We are always looking for ideas and comments on how to improve the recruitment process for user research, so if you have something to share, please let us know.

The Path of Innovation

Everett Rogers introduced his theory of the diffusion of innovations in 1962. [1] Since that time, his theory has provided guidance to businesses and individual innovators when bringing new technologies to market. The normal path to mainstream success begins with innovators and early adopters, as Figure 1 shows. Innovators are typically people who actually develop new technologies. They plug into the newest technologies as a way of feeding their need to innovate. Early adopters are the first consumers in the market and the test bed for other users. Early adopters are typically itching to try new technology, regardless of the quality of its execution. So they usually acquire new products immediately, just to try them out and determine whether something will be a revolutionary success or an innovative misfire. In this respect, they provide a valuable service.

Figure 1—Rogers’s adoption curve

Right behind the early adopters is a group of users called early mainstream. These users typically pay attention to new technologies, but wait for the market to test them. Following early mainstream users are late mainstream users—that is, people who don’t usually pay a great deal of attention to technology innovations, but eventually adopt new technologies and products once they have become firmly established or even have an overwhelming presence in the marketplace. These users usually come to new products passively.

Finally, there are late adopters, who are people who tend to be suspicious of new technology, don’t see the value of its new capabilities, or don’t feel capable of engaging with new products because of their lack of experience with technology. Older users are a classic example of late adopters, but this group also tends to include people who live in rural or isolated areas and adult users who earn less than $55K annually.

Throughout each of these steps in the adoption curve, social influences tend to play a major role. Early mainstream users hear about new products and technologies from early adopters. Late mainstream users get introduced to new technologies by their early mainstream friends. When late adopters do take up new technology, it’s typically pushed on them by a more tech-savvy friend or family member—someone they trust who influences them directly.

The Problem

Trust and direct social influence are powerful forces that can be quite difficult to replicate. In directly targeting late adopters with new products, companies would be skipping ahead in the adoption process, which presents a series of difficult challenges.

One of these challenges is reaching these users in the first place. Many late adopters do not regularly use computers to access the Internet, so common startup marketing strategies such as social media and online advertising do little to build awareness in the target audience. Late adopters typically still have stronger ties to traditional media such as television and print, with their much higher advertising costs, reduced exposure, and fewer measures of advertising effectiveness. Large corporations can tackle this challenge, because they have greater resources at their command, but startups often have difficulty generating awareness of products that target late adopters.

The attitudes of late adopters toward technology present other challenges these companies must overcome, because it’s not uncommon for late adopters to feel suspicious of new technology or think that most new technology is unnecessary.

Substantial experience with technology that has enriched their lives motivates early adopters to try new technology products. However, late adopters tend to lack the behavioral precedent of successfully adopting innovations, so the idea can be somewhat alien and uncomfortable to them. Establishing demand for a new technology among late adopters requires their embracing new behavior, so involves much more than just introducing the technology to the market, making its adoption much more difficult.

The process of designing products for late adopters also presents challenges, because of the perspective gap that exists between the innovators who create such products and late adopters. It can be very difficult for product designers and developers to relate to users who are on the opposite end of the adoption curve. As a result, they could develop something they think is relatively simple, but is still too complex for inexperienced users. They might also employ user-interface design conventions they think are fairly standard, but might be completely alien to a late adopter who lacks similar experience interacting with the Web or consumer electronics. Many of the interaction design standards that we use today are not necessarily truly intuitive; they have just been around for so long that most people accept them. Interaction design concepts as fundamental as the desktop and windows are metaphors for physical items that may not be immediately clear to inexperienced users.

Strategies for Targeting Late Adopters

By understanding the challenges of targeting those users who are late adopters, we can start exploring strategies that can help us to be successful with this demographic. Once we understand the difficulty that late adopters encounter when using complicated technologies and software user interfaces, it becomes fairly obvious that any products targeting this market segment must be very simple to learn and easy to use. However, the challenge is determining whether a product actually is sufficiently simple to use. To ascertain this, you’ll need to conduct usability studies, recruiting participants from this market segment.

Begin with early concept testing of mockups. Don’t rely on demographics when recruiting, or you may accidentally recruit participants who are not truly late adopters. For example, if you merely recruit people who are over the age of 55, you could end up with a 65-year-old retired software engineer who plays World of Warcraft constantly and practically lives on Facebook. Instead, your recruiting criteria should assess behaviors that are consistent with a late adopter such as the following:

• Does not own a computer.
• Does not know how to access email.

Do not rely on friends and family for concept testing, because late adopters can be particularly susceptible to telling you what they think you want to hear rather than the brutal truth.

During testing, show the mockups to participants and ask them to describe what they think they depict and the value such a product could provide to them. Make sure not to provide participants with any additional explanation of a product until they’ve first given you their description, or you will corrupt their answers. Although their answers are very likely to be incorrect, if they are reasonably close, you are probably on the right track. In this case, you might get away with just making a few tweaks or adding some minimal explanatory text to the product before its introduction. However, if they are way off in their descriptions of your product or cannot see its value, you might need to completely rework your product description, value proposition, or even the entire product. It’s not entirely unlikely that you may find the product is just a bad idea, which is unfortunate, but can save you a great deal of money and effort in the long run.

As you develop your product further, you’ll need to evaluate the actual user interface through usability testing. Again, it is essential to recruit the right participants and ask them to complete tasks without any assistance. Keep a close eye on where participants encounter difficulty and document their actual success rates. Late adopters tend to be very forgiving when using a user interface, and they often blame their own inadequacies in using technology rather than providing deserved criticism. Keep this in mind, and don’t rely too heavily on self-reported data. Instead, focus on objective behavioral data.

For innovative new products, it can be helpful to find a way to piggyback on the normal technology adoption process rather than trying to circumvent it. One way of doing this is to keep early adopters in mind during development. This could mean including some advanced features or creating a separate version of a product that provides value specifically to these users.

Ultimately, your goal is to create enough of a following to create a social influence on late adopters. If you intend to employ this approach, include a couple of early adopters in your usability testing and ask them who they think would get the most out of using your product. If they respond by saying something like This would be great for someone like my grandfather, you’ll know that you are on the right track. Apple’s iPad, with its simple interaction model and variety of features, is a great of example of an innovative product that successfully appeals to both early and late adopters. In such a case, trusted brand perception is a major advantage.

However, it’s important to keep in mind that late adopters might interact with these products in a very different way than you would. For example, users who are 55 and older—many of whom would historically be considered late adopters—are currently the fastest growing market for smartphones. [2] But this doesn’t necessarily mean they are using smartphones to play Foursquare and Angry Birds like early adopters would. In our user research, we’ve found that users from this age group tend to use the most basic capabilities like making calls and a few very popular apps such as the eBay app. Keep this in mind if you are developing a mobile application and, when conducting user research, ask about users’ existing behaviors, so you can discover the reasons why late adopters might use some apps, but not others.

Conclusion

Late-adoption populations offer tremendous potential for growth, but they remain untapped for a reason. Breaking new ground with late adopters presents a significant challenge. While, as always, the key to success is understanding your users, this can be especially difficult in this case because of the experience gap between late adopters and the innovators who develop technology products. To bridge this gap, make sure you do adequate user research. Remember, what you think is simple and intuitive might not be for a late adopter. You’ll have to challenge all of your assumptions to create a successful product.

It’s also important to think about how a late adopter might become aware of your product. Common marketing strategies such as social media and online ad campaigns would probably fail to reach this audience. Finally, try to find ways to leverage the adoption curve by creating interest in your product among early adopters and mainstream users. By employing these methods, you are much more likely to produce a product strategy that will find success in the marketplace. We invite our readers to share their thoughts and experiences with developing products for late adopters.

Do Use Focus Groups Early to Identify Important Research Questions

Focus groups can be an effective way to get started on a project. When you are just starting to explore a product idea, getting a group of potential users together to discuss ideas can be extremely helpful. Focus groups can provide you with a cost-effective means of testing your team’s initial assumptions, beginning to identify relevant market segments, exploring product ideas, and acquiring the data you need to create a UX research plan to guide product development.

Don’t Try to Do Too Much with Your Findings from Focus Groups

The biggest mistake people make with focus groups is an overreliance on the findings in informing design decisions. Focus-group findings are subject to a variety of confounds, including the relatively low numbers of participants, participants’ speculation during the session, and the significant influence the opinions of others can have on participants in a focus group. For these reasons, researchers do not typically use focus groups in rigorous scientific research. Focus groups provide high-level feedback that helps you to make strategic decisions. When you are getting to the point where you have to make very specific, tactical design decisions, it is time to move on to more reliable research methods.

Do Recruit Participants from a Variety of Backgrounds

One of your goals in doing early, discovery research is to identify relevant market segments. One way of  doing this is to gather representatives of different market segments into a single focus group to determine how people in different segments respond to particular ideas. You might find that certain ideas resonate strongly with some participants, but don’t interest other participants at all. In this case, exploring the reasons for their different responses can help you to identify different products that might fit into a single product line.

For example, when examining eReaders, you might find that some participants want a very compact product, while others want a larger screen. By exploring further, you might find that participants who are interested in a compact product like to read novels and tend to travel a lot, so the small form factor supports their reading habits. On the other hand, the participants who want a larger screen might like to read newspapers and magazines at home or in their office. These different use cases could lead to different products, each of which is designed to support the reading habits of one of these two very different market segments.

Don’t Recruit Participants Who Are Too Similar or Too Different

It’s important that focus-group participants have something in common for you to get useful information about a topic you’re exploring. For example, if you are considering developing a photography product, you’ll likely want all participants to be interested in photography in some way. On the other hand, it’s important to ensure that participants aren’t too similar for you to acquire a variety of perspectives and foster discussion.

Continuing with the photography example, if you have photojournalists, wedding photographers, and amateur photographers in a session together, you might find that they have very different needs regarding a photography product, but you also might find that there are a couple of themes that are consistent across all groups.

When you uncover such differences, it’s important to explore the reasons why people’s needs are different. For example, while amateur photographers might be extremely interested in advanced photo-sharing capabilities, wedding photographers might not. By exploring further, you might determine that wedding photographers need to keep very tight control of their photos, or work product, so they need more security than amateurs do. You might not discover such needs if you didn’t have these different types of users in the room together, talking with each other about their commonalities and differences.

Do Include More Than Two People in a Focus-Group Session

Good focus groups should include a variety of perspectives and foster communication among people with slightly different backgrounds. In a good focus group, participants comment on what other participants say, either expanding on a topic or expressing a difference of opinion. If you do a focus group with just two people, you’ll tend to end up with something more like two separate interviews that you’ve conducted concurrently, because you’ll have a harder time establishing a discussion with a lively exchange of ideas.

Don’t Include More Than Six People in a Focus-Group Session

Acquiring thoughtful comments on several topics from multiple people can take quite a bit of time. The more participants you add, the more time a focus group takes. We usually find that extending a session beyond two hours causes participants to tire and pull back from the discussion. For this reason, it’s a good idea to limit sessions to only six participants. This also helps you to maintain control of a session as the discussion gathers energy.

Do Prepare a UX Research Plan

A good focus-group session flirts at the edge of chaos. When comments are flying around the room, it can be difficult to maintain a session’s focus and ensure that it covers all of your planned topics. When you’re putting together your UX research plan, begin with the business goals for your research, then identify the research goals that would support those business goals. You should be able to extract some specific research questions from your research goals, and these will be the main topics for your focus group to discuss. Make sure you work all of these topics into each session without disrupting the flow of the discussion.

Don’t Use a Research Plan As a Script

During a focus group, striking the right tone is essential to facilitating open discussion and the flow of ideas. It’s extremely difficult to exercise any control over a discussion’s tone if you are reading from a script rather than responding to comments and ideas as they emerge. Also, focus groups are unpredictable, so you don’t know what new ideas or concepts may surface. It’s important that you be able to respond to new discoveries fully as they arise, and this can be very difficult if you are following a script.

Do Work with a Partner

In our experience, focus groups are the most challenging UX research sessions to moderate, because of the dynamic nature of the sessions, the number of participants you must interact with simultaneously, and the need to strike the right balance between a free-form discussion and a structured research session. Having a partner in the room is incredibly helpful in this kind of setting. Your partner can help out by taking notes, keeping track of time, and helping to enforce structure on the session when it starts to go off the rails. Your partner can also jump in and ask questions when he or she notices that there is a key point that the group isn’t addressing.

Don’t Rely on Your Partner to Help Control a Focus-Group Session

You should introduce your partner as someone who is helping you by taking notes, and you should think of that person as a coach. Only one person can control a focus-group session. If control begins to diffuse across multiple moderators, a session can quickly disintegrate, especially if the two moderators don’t completely agree on where to direct a session.

Do Invite People to Observe Focus-Group Sessions

It’s difficult to distill a complex interchange between multiple people into a report, even when using audio or video clips. For focus groups, it’s common for the dynamic exchange of ideas to be an essential takeaway. The best way to ensure stakeholders get this information is to allow them to observe the sessions. We like to do this by setting up video cameras and piping a feed over a secure network connection to any stakeholders who want to see it. We’ve also found that Microsoft RoundTable is especially well suited to streaming and recording focus groups.

Don’t Allow Observers to Interfere with a Session

As we mentioned earlier, the moderator should be the only person in control of a session. If participants witness others stepping in and directing a session, they may feel that they are also free to influence the direction of the session. As we said, if this happens, a session can quickly disintegrate.

Do Encourage Conversation Between Participants During a Session

Sometimes people find talking in a group to be intimidating, so it can take a little encouragement to get participants to open up. To facilitate everyone’s participation, set the tone for a session by taking the time to chat with each participant before launching a discussion of your planned topics. Introduce yourself and volunteer some information about  your personal experiences to let participants get a feel for who you are.

Do a round of introductions, and really pay attention to what each person says. Follow up each introduction with thoughtful questions that show you really care about who a person is and what he or she has to contribute. If one person seems more comfortable with open communication, begin the session by chatting with that person to set an example of the kind of communication you want, then transition to another participant who might not be quite as talkative. By making this transition, you’ll show more reserved participants that their thoughts and opinions are equally important to you and that they can communicate with you and the group in the same way as more gregarious participants.

Don’t Lose Control of a Focus-Group Session

If a discussion gains too much momentum, it can take on a life of its own and get away from you, so you might spend 20 minutes talking about something that is completely unrelated to your research goals. To avoid this happening, exercise strict control over a session, and don’t be afraid to stop people if they wander too far off topic or take too much time with their responses. But do this by saying something along the lines of I’m really interested in what you’re saying, but I need to get us back to the main topic of our discussion because our time is limited. This way, the participant won’t feel invalidated or shut down. As you get good at this, you’ll find more subtle ways of directing a conversation back on topic by asking questions like So, how does that relate to XYZ?

Also, take the time to identify any alphas in a group. Alpha is our term for people who like to control a conversation and direct its topics of discussion, whether intentionally or unintentionally. Such people tend to have forceful personalities and promote their ideas to a point where they influence the opinions of others. When your focus group includes one or more alphas, try to get their responses toward the end of each topic of discussion, so you can avoid their having undue influence. Be ready to redirect the conversation if they start to pick up too much steam and want to keep talking. If you have two or more alphas in the same group, try to avoid letting them talk one after the other. Insert a buffer between them. If they get going speaking to each other, they can dominate an entire session.

Do Discuss Each of Your Topics and Respond to Ideas As They Arise

As you work through each of your topics and new ideas arise, explore them to see what they mean and whether you can get multiple perspectives about them. Often, when people first introduce an idea during brainstorming, it isn’t yet fully formed, or they may not communicate it effectively. By asking further questions or getting responses from other participants, you can get a much better understanding of an idea. In such cases, a natural sense of curiosity serves you well.

Don’t Dig Too Deeply into Any Given Topic

Remember, your goal in a focus group is to properly frame your questions rather than trying to arrive at definitive answers. You’ll likely find that some of your questions turn out to be less important than you anticipated, while others that you hadn’t considered to be pivotal turn out to be more important than you thought. Don’t feel pressured to leave a focus-group session with all of the data you need to guide your team’s design decisions. Instead, try to leave each session with a firm understanding of your information needs. This can be extremely valuable because it can help you to avoid spending significant resources investigating topics that are unimportant. It can also help you to identify important issues that could seriously hamper your product’s success if you do not address them properly through additional research.

Conclusion

Although UX research and design have begun to migrate away from using focus groups, they do provide significant value when you use them properly. The most important guideline for the effective use of focus groups is avoiding an overreliance on the resulting findings. The findings that result from this method of UX research are not as reliable as those you can gain through more rigorous research methods. However, focus groups can be a cost-effective way of building the understanding that forms an excellent basis for the UX design research you ultimately need to do. The guidelines we’ve presented here can help you get the most out of your focus groups. Of course, it falls to you to exercise your best judgment in determining what research approaches to implement and how to proceed with them. But keep in mind that focus groups are always a tool at your disposal.

Figure 1—Focus group lab with a one-way mirror

Why a One-Way Mirror Is Not Always the Best Option

In the fields of physics, psychology, and biology, there is a phenomenon called the observer effect, which refers to how observation changes the object or person being observed. For example, taking pictures or videos of a tribal culture that has never experienced such technology introduces new concepts into the society and fundamentally changes their behavior, because this is not a natural experience they would normally encounter during their daily lives.

This is a major concern for user research and focus groups that are conducted in labs with one-way mirrors. When you place people in an environment where they sense they are being observed, their behavior changes. These changes in behavior can drastically affect your ability to accurately understand how your target market feels about your product or service. In our experience, people tend to focus on positive responses and have a very difficult time being completely open and honest about their thoughts and feelings.

One of our primary goals when conducting research is to make participants feel extremely comfortable. One of the ways we do this is by taking a little time to get to know participants and enable them feel that they know us a little, too. To minimize the impacts of observation on research, we emphasize the need for honest feedback and also reassure participants that we are not actively watching or evaluating them; rather, we are evaluating the product and taking notes and listening to their comments to enable us to improve the product. What we have found creates the most anxiety for participants is the idea that hidden strangers are staring at them through a window.

The perfect example is when participants try to complete a series of challenging tasks as part of a usability test. It is easy to see their level of frustration rise as they work through the tasks—while repeatedly glancing at the mirror.

User Feedback on the One-Way Mirror

We’ve had the opportunity to work with thousands of participants, and the feedback we tend to get about one-way mirrors is that they are scary, intimidating, nerve racking, and uncomfortable. In one case, when we asked a participant about the one-way mirror, she responded: “I could just feel everyone watching me, and it made me feel dumb for having such a hard time.” This type of feedback is worrisome because users may not be fully expressing their thoughts about the product. They become distracted by the feeling of being watched when they make mistakes rather than remainingve focused on improving the product by providing thoughtful feedback.

In almost every case, participants were in a very nice room with comfortable couches and chairs, and we had provided them with food and beverages. You might think they would walk away from the experience with a feeling of satisfaction and excitement, considering they had just had the opportunity to get a look at a new product or service. Unfortunately, the mere fact that these people feel like they are being watched changes their overall experience.

We took a deeper look into these anecdotal findings and asked a few research participants more detailed questions about their experience with one-way mirrors, and this is what we found out. Once participants walk into a room and see a one-way mirror, they believe there is a certain way they need to act, a certain way they need to respond to questions; and they feel that they need to do things correctly.

The only way we’ve been able to mitigate these feelings is to talk through them and reassure participants that we’re using the room only so we can tape the session, enabling further review. Since we know, in general, what feelings people tend to have when sitting in a lab with a one-way mirror, we try to address them immediately, as best as we can, and are able to put some of them to rest.

Best Practices When Using a One-Way Mirror

By combining our experience with what we’ve learned from discussions with other UX professionals who use labs with one-way mirrors for user research, we can offer some best practices:

• Discuss the elephant in the room and point out the two-way mirror, explain why you are using it, and address any concerns a participant may have about it.
• Talk with participants for a few minutes before a session and help them relax. Ask about their day or their work. It’s best to quickly build friendly feelings with participants to help them relax and feel more comfortable with the situation.
• Wear clothes that are similar to what participants are likely to wear to a session—perhaps jeans and a casual top. If you look like a psychologist or evaluator, that can make participants nervous and less likely to open up. We find that, when you blend in, you build rapport faster and get more honest feedback.
• Don’t let participants sit facing the mirror—have them sit to the side, facing away from the mirror, or if possible, with their back to the mirror. There is nothing more nerve racking than staring at a mirror for an hour when trying out a new product.
• If possible, moderate sessions from within the room with participants. Providing participants with instructions and asking them questions face to face helps them to forget about the mirror and focus on the person in front of them rather than on a disembodied voice coming over the speakers.
• Finally, don’t act like you are formally examining participants. Act casually and treat participants as if you are working with them, not just watching them. For example, when you ask participants questions about their feedback on a product, have someone else take notes, so you can have a normal conversation with them, and they can focus on you the whole time. We find this method very helpful, because it frees up a researcher to ask lots of follow-up questions and focus deeply on what participants are saying in the moment.

Alternatives to the One-Way Mirror

There are a few alternatives to using a lab with a one-way mirror that have proven successful in alleviating the uncomfortable feelings participants endure in such a situation.

• Rather than using a one-way mirror, install an overhead camera and put observers in a room down the hall. Participants tend to forget about cameras, but have a hard time becoming habituated to a large, mirrored wall.
• Bring participants into an office setting and conduct sessions in front of team members working at their desks. This may sound like a stressful environment for participants, but when they see everyone working and just going on with their business, they tend to relax and feel part of the team. This is an excellent opportunity for a team member to take notes, because participants won’t be paying attention to what everyone else around them is doing.
• Visit participants in their own environment—whether their home, business, or a local coffee shop. By doing this, you alleviate the stress of participants’ being outside their usual environment. In this case, you can use Web video services to stream sessions to observers and also tape sessions for future review. More realistic behavior tends to be prevalent when participants are in their own environment.
• Remote sessions are becoming more and more common and involve participants’ using their own computer to connect with a moderator and working through a test scenario or responding to questions like those typically asked in focus groups. This option is less personal, but allows participants to be in a setting that is comfortable for them. Plus, it creates an interpersonal disconnect that typically leads to increased confidence and honesty.

Conclusion

There are many ways in which we can conduct user research—using a one-way mirror is just one option. Although we do not think doing research in a lab with a two-way mirror is always the best option in all situations, we do think it can provide value in certain situations—especially if you follow best practices and give the proper attention to the needs of participants. But, in this column, we wanted to point out that there are alternatives to using a lab with a one-way mirror that have worked well for us over the years. Ultimately, our goal is to ensure that participants are comfortable, understand their role in our design projects, and provide the best product feedback possible.

We would love to hear your thoughts about this. Have participants given you any interesting feedback on how they feel about their experience of being in a lab with a one-way mirror? Let us know!