Prototype Testing

Readers of these articles know that I view the high-fidelity prototype as the primary means of describing the product to be built. I have written elsewhere why a prototype is significantly more useful to the product team than the typical paper-based specification. However, that’s really the secondary benefit. The primary reasons to create a high-fidelity prototype are to help you gain a much deeper understanding of your product, and ulimately so that you can actually test your ideas with real users before you have your engineering teams take months to go build something that you have no real evidence will serve its purpose.

In this article I’d like to talk about how to actually do this prototype testing. I’ll warn you up front that this article is relatively long, but I will also say that testing your ideas with real users is probably the SINGLE MOST IMPORTANT ACTIVITY in your job as product manager.

If your company is large enough to have its own usability testing team, by all means secure as much of their time for your project as you absolutely can. Even if you can’t get much of their time, these people are terrific resources and if you can make a friend in usability research or usability testiing it’ll be a huge help to you.

If your organization has funds earmarked for outside services, you may be able to use one of many excellent firms to conduct testing for you. But at $10,000 – $20,000 per round of testing (typically around 10 users) that most firms charge, chances are that you won’t be able to afford as much of this testing as your product will need.

But if you’re like most companies, you have few resources available, and even less money. But you can’t let that stop you. It is absolutely essential that you test your ideas out with real users. It is arguably the single most important part of your job.

So I’ll show you how to do this testing yourself. Don’t get me wrong, you won’t be as proficient as a trained usability engineer, and it’ll take you a few sessions to get the hang of it, but in most cases you’ll find that you can still identify the serious issues with your product, which is what’s important.

One thing to note is that while most people think of “usability testing” (seeing if people can figure out how to actually use your product) I consider that just one type of testing. You also need to test the desirability or usefulness of your product (do people actually want to use it?), and we’ll discuss both forms of testing here.

Finding Test Subjects

Before you do the prototype testing, you’ll need to round up some test subjects. If you’re using a lab they’ll recruit and schedule the users for you, which is a big help, but if you’re on your own, you’ve got several options:

– If you’ve established a “charter customer program” as I’ve described earlier, you should have quite a few readily available. If you haven’t you should.

– If you’re doing a product for business, then trade shows are a great source of target customers.

– It’s increasingly common to advertise for test subjects on Craigslist. If you do this, try to keep your participant description a notch more general than specific, and then screen on the general when you call them to talk about this.

– For consumer products you can use your “friends and family” network – just try to avoid people too close to you, and those in the tech industry, unless that’s specifically your target, and be sure to use subjects from outside this network too.

– If you have a list of e-mail addresses of your users you can do a selection from there – often your marketing team can help you narrow down the list.

– You can solicit volunteers on your web site – lots of major sites do this now – remember you’ll still call and screen the people to make sure you don’t get a bunch of early adopter types.

– One technique I especially like is to set up regular prototype test sessions, say every other Friday, where you arrange for 10 – 20 or so users to come into your offices for a couple hours each, and then your product managers sign up for the time slots, so a given user might test a couple prototypes each. I like this a lot because one person can do the logistics of invites and screening, and many PM’s can count on a ready set of test users on a regular basis.

– You can always go to where your users congregate. If you’re doing an e-commerce product, you may want to go to a mall. If you’re doing a sports product, go to a sports bar. If your product is addressing a real need, you won’t have trouble getting people to give you an hour. Just bring gifts, and try not to look like you’re trying to convert their religion.

– If you’re asking users to come to your location – especially for business use – you will likely need to compensate the people for their time. If you’re doing a consumer service sometimes a big sincere thanks along with a hat with your company logo on it will suffice as people genuinely want to help in the creation of products, especially for companies they like. However, if you do compensate consider providing product like $50 of credit on your site.

– Realize that there’s a very high no-show rate when you schedule people to come in. It’s just a fact. Sometimes as much as 30%. But you can drop it to 5- 10% if you give people a personal phone call the day before — even if you leave a voicemail. Note that email does not work equally well.

Preparing The Test

You’ll need to define the usability tasks you’ll want to test, and the interview questions concerning desirability:

– You will need to define in advance the set of tasks that you want to test. Usually these are fairly obvious. If you’re building an e-mail client, your users will need to do things like compose a message, read new mail, and file away messages. There will also be more obscure tasks, but concentrate on the primary tasks, the ones that users will do most of the time. If you have time, you can get to less common tasks but it’s essential the key tasks are tested well.

– There is a one-time only opportunity you have with each user you test. That is the opportunity to learn how they think about this problem today.

If you’re testing a new online restaurant rating service, rather than start them out at your prototype’s home page, maybe you want to just start them out with an empty browser and see what they do. What review sites do they use today? Do they use Google or Yahoo’s search to find the specific restaurant, or do they go somewhere like OpenTable or Zagat? Do they search by neighborhood, by cuisine type, or price range?

This type of incredibly valuable information is missed if you jump right into your prototype, which will necessarily have quite a few assumptions built in. Once they play with your prototype they can tell you what they like better, but they won’t be thinking about the problem anymore the way a first-time visitor would.

– Of course, you’ll then want to get them to your prototype, but there’s one more thing before you jump into your tasks. See if they can tell from the home page or landing page of your prototype what it is that you actually do, and especially what might be valuable or appealing to them. Again, once they jump into tasks they’ll lose that first time visitor context, so don’t waste the opportunity. You’ll find that landing pages are incredibly important to bridging the gap between expectations and what the site actually does.

– After you’ve seen if your user can figure out how to do the tasks you’re testing, it is now the right time to have a conversation with this user. Think of it as a one-person focus group. Does the user use a different product or site today with the same thing? Or is this something they do manually or off-line today? How much better is this than what they use today? And don’t forget to ask my favorite question: How likely would you be to recommend this product to your friends? Now that the user has interacted with your prototype they understand the topic and you can have an extremely useful dialogue with them about this problem. Most importantly, you’re trying to gauge how much this person values this product.

– One technique I like for gauging value is to ask how much the user would be willing to pay for it, even if you have no intention of actually charging for use this way. It’s a way to assess value and especially to track how the average value goes up or down over time as you change the prototype. Also, it’s useful if you structure your questions on a scale, like 1- 10. This is so that you can track the averages as they improve.

– Note that you don’t have to wait until you have a complete prototype in order to begin testing. You can start with the main tasks, and it’s ok if you have dead ends in the rest of the prototype. If the user wanders over to one of those dead ends, just ask “And what would you expect to happen if you did that?” This is a great question whether you have that path laid out or not. If you do not have it laid out, you can see if they match. And if you don’t, you’ll get important info about that direction anyway.

The Test Environment

Here’s how to prepare your test environment:

– Formal testing labs will typically have set ups like two-way mirrors, or closed circuit video monitors, as well as cameras often capturing both the screen and the user from the front. Just know that while that’s great if you have it, you do not need these things to have an extremely useful and valuable test. I can’t count how many prototypes I’ve tested at a tiny table at Starbucks just big enough for the laptop, with three chairs around the table. In fact, in some ways this is preferable to the testing lab because the user feels a lot less like a lab rat.

– The other environment that works terrific is your customers office. It is time consuming to go there, but often even 30 minutes in their office will tell you a lot, and they are “master of their domain” and often very talkative. Also, all the cues are there to remind them of how they might use the product. You can also learn from seeing what their office looks like. How big is their monitor? How fast is their computer and network connectivity? How do they communicate with their colleagues on their work tasks?

– There are tools for doing this type of testing remotely, but while you can see where their mouse is, and what the user is clicking on, it’s not the same as looking at the person’s eyes and body language, so again, while the more testing the better, to me this is not a substitute for face-to-face testing.

– As product manager, you need to make sure you are at every single test. Do not delegate this. Real value comes from experiencing as many users as possible, first hand, interacting with and responding to your ideas. Even if you use an outside firm to arrange and administer the tests, you need to be there with them during the testing. No one knows your product as well as you do, and you will have insights from watching the slightest hesitation or confused look, or the nuance of a question that reveals that they don’t understand the product model or particular feature. What gets summarized for you by a proctor will probably miss 4 or 5 key insights.

– Some people believe that the product manager (and the interaction designer) are too close to the product to do this type of testing objectively, and they may either get their feelings hurt, or only hear what they want to hear. My view is that the good product managers and interaction designers get past this very quickly. They know they will get the product wrong initially, and that almost nobody gets it right the first time, and they know that learning from these tests is the fastest path to a successful product.

– Ideally, you should have one person administer the tests and another person taking notes. It’s very useful to have someone to debrief with afterwards to make sure you both saw the same things and came to the same conclusions. That said, if it’s just you and your laptop, and you’ve got a ready and willing target user in front of you, do it. It is all good.

– If you as product manager have a user researcher or usability engineer along with you, let him or her administer the test and you take notes. Otherwise, you’ll probably be the one that administers. It’s great to invite others on the product team to be your note taker. Most often it will probably be the interaction designer, but the visual designer, developers, and especially mangers are all useful and they’ll get a lot out of it.

Testing Your Prototype

Now that you’ve got your prototype ready, you’ve lined up your test subjects, and you’ve prepared the tasks and questions, here are a set of tips and techniques for administering the actual testing:

– When you first sit down with the test subject, make sure to tell him or her that this is just a prototype, it’s a very early product idea, and it’s not real, they won’t be hurting your feelings, and you’re testing the ideas in the prototype and most importantly you’re not testing her. She can’t pass or fail – only the prototype can pass or fail.

– Also, you should greet the person warmly and offer a coffee or something, but the sooner you get to the prototype the better. Tell them you’ll chat with them about this after they test the prototype, but you want to get their untainted impressions first. Realize that the more you chat beforehand, the more clues you are giving away and the less of a true first impression your test subject can provide. If more than 5 minutes goes by without the user starting in on the prototype you are talking too much.

– When testing, you’ll want to do everything you can to keep your users in “use mode” and out of “critique mode.” What matters is whether users can easily do the tasks they need to do, and whether they value the product. It really doesn’t matter if the user thinks something on the page is ugly or should be moved or changed. Sometimes misguided testers will ask users questions like “what three things on the page would you change?” To me, unless that user happens to be an interaction designer, I’m not really interested. If users knew what they really wanted, software would be a lot easier to create. So watch what they do more than what they say.

– During the testing, the main skill you have to learn is to keep quiet. Normally when we see someone struggle most of us have an urge to help the person out. You need to suppress that urge. You have to turn into a horrible conversationalist. Get comfortable with silence.

– There are three important cases you’re looking for: the user got through the task with no problem at all and no help; the user struggled and moaned a bit but he eventually got through it; he got so frustrated he gave up. Sometimes people will give up pretty quick, so you may need to encourage them to keep trying a bit longer, but if he gets to the point that you believe he would truly leave the site and go to a competitor, then that’s when you note he truly gave up.

– In general, you’ll want to avoid giving any help or “leading the witness” in any way. If you see the user scrolling the page up and down and clearly looking for something, it is ok to ask the user what specifically he’s looking for, as that’s very valuable to you. Some people ask users to keep a running narration of what they’re thinking, but I find this tends to put people in critique mode, as it’s not a natural behavior.

– Act like a parrot. This helps in many ways. First, it helps avoid leading. If they’re quiet and you really can’t stand it because you’re uncomfortable, tell them what they’re doing: “I see that you’re looking at the list on the right.” This will prompt them to tell you what they’re trying to do, looking for, whatever. If you ask a question, rather than giving a leading answer you can play back the question to them, “Will clicking on this make a new entry?” “You’re wondering if clicking on this will make a new entry?” and they will take it from there because they’ll want to answer your question: “Yeah, I think it will.” Parroting also helps avoid leading value judgments. If you have the urge to say “Great!”, instead you can say “you created a new entry.” Finally, parroting key points also helps you note-taker because they have more time to write down important things.

– Fundamentally, you’re trying to get an understanding of how your target users think about this problem, and to identify places in your prototype where the model the software presents is inconsistent or incompatible with how the user is thinking about the problem. That’s what it means to be counter-intuitive. Fortunately, when you spot this it is not usually hard to fix, and can be a big win for your product.

– You will find that you can tell a great deal from body language and tone. It is painfully obvious when they don’t like your ideas, and it is also clear when they genuinely do. They’ll almost always ask for an email when the product is out if they like what they see, and if they really like it, they’ll try to get it early from you. Recently I attended some prototype testing with one of my clients in Berlin, and even though I don’t speak German, it was obvious what the issues were, and which ideas worked well and which ones didn’t.

Updating the Prototype

The whole idea of this testing is to identify what you need to fix in the prototype to make it more useful and usable. So, as fast as possible, you’ll want to work to correct the problems:

– Some people believe you have to freeze the prototype, the tasks and the questions for a complete round of testing (generally 6-8 users) before drawing any conclusions. I don’t think that’s true, and have found that you can significantly accelerate this process of getting to a good product faster by responding more quickly to the feedback.

– You don’t have to be hit on the head by 8 users in a row to know you need to fix something big. So go ahead and fix it when you feel you’ve identified a problem, even if that’s after only 2 or 3 users. The harder question is when you’re done. Generally, if you can get through 6 consecutive users where they understand and appreciate the value, and then can get through the key tasks, you’re in good shape and you’ve done your job.

– You might determine that you just aren’t able to get people interested in this problem, or figure out a way to make this usable enough that your target users can realize this value. In this case, you may decide to stop there and put the idea on the shelf. Some product managers consider this a big failure. I view it as saving the company the wasted cost of building and shipping this product, plus the opportunity cost of what your engineering team could be building instead.

This whole process might sound complicated or difficult, but the remarkable thing is just how easy and effective it actually is. The best way to prove this to yourself is to take your laptop with your product or prototype on it to someone that hasn’t seen it yet and just give it a try.

I want to give credit here to two important sources (and great resources for you).

The first is the book, “Don’t Make Me Think” by Steve Krug. It’s mostly a book on interaction design, but in the back third he makes a compelling case for this sort of informal testing, and he provides many useful tips. I have long recommended this book to both product managers and designers, and I hope you’ll buy a copy and read it carefully.

Second, my favorite product testing firm is Creative Good (www.creativegood.com). These guys specialize in a form of testing they call “Listening Labs” which is a powerful form of undirected testing that can find huge issues with your product – functionality and design – that can result in pretty dramatic improvements to the business results. While most testing firms test the products on a task basis, these guys are good at looking at the big picture, which is where many of the big improvements are to be found. Several of the techniques above are adapted from their testing methodology.

Thanks also to a couple of world-class user research specialists, Kyrie Robinson and Audrey Crane, for their insights and contributions to this note.