Guide to Remote Usability Testing July 18th, 2006 by Nate Bolt :: see related comic

As more usability practitioners start conducting remote usability testing, there seems to be a demand for some tips and guidelines around this technique. New screen-sharing tools like Breeze, Co-Pilot, and GoToMeeting, and remote usability tools like Ethnio and UserVue, make it easier to conduct moderated remote usability testing. Dealing with video and audio recordings keeps getting simpler as well. But observing people remotely presents a unique set of obstacles, so this is a guide to what we’ve learned from conducting 149 remote studies with 1,213 participants over the last seven years. We can’t get that time back, but hopefully some of what we’ve picked up will be helpful.

Moderated Remote What?

Basically, there are two kinds of remote usability – moderated and automated.

• Moderated is one person watching another person use a computer, viewing their screen movements with a screen-sharing tool, and talking with them over the phone. The moderator watches and listens to where the participant runs into difficulty while interacting with the application or site, and records both the conversation and the participant’s screen. It’s pretty much the same as in-person testing, but minus the user’s facial expressions. Also known as facilitated remote research.
• Automated means hundreds or thousands of participants report their own behavior through a browser window or frame that has an open text field and survey questions. As the user navigates through a web site, they enter their feedback and answer page-specific questions in the browser frame. It’s more than a survey, because it’s still behavioral, but there is nobody watching and talking with the users. Some people call this unattended remote usability. The data is usually automatically distilled into a report with aggregate verbatim answers, click data, or exercise results. Some of the best new automated remote tools are MindCanvas, UserZoom, ClickTale, RelevantView, and KDA Revelations.

You can and should mix these methods, but this article is going to stick to moderated. If you’re just getting started with remote usability, there are articles here, here, here, and here, that can give you more background.

Recruiting Doesn’t Have to Suck

In order to get started, you’ll need some users, and there are basically three options for how you can recruit for remote testing:

• Traditional includes all the ways you would recruit for an in-person study - email campaigns, customer lists, recruiting agencies, craigslist ads, bus stop flyers, whatever.
• Online scheduled is where you use an online survey or recruiting screener of some kind to schedule participants for sessions in advance.
• Live Recruiting means you use an online screener to intercept people in the middle of their real-life tasks, and watch them live as they complete those tasks, in their native task environment. No scheduling in advance. This method is so effective that I’m going to dedicate the entire next section to it. Here we go.

Native Task Environment: A Really Big Deal

The ability to intercept someone live as they begin a task, have them quickly share their screen and comments over the phone, and then watch them continue to use their computer the way they were about to anyway, is by far the greatest benefit remote testing offers. The criteria by which participants make click decisions are totally different when they have a temporal, emotional, and logistical attachment to completing a task. Throw in their native physical and computing environments, and you are looking at the most accurate form of behavioral usability research possible. Well, that might be a stretch, but you see what I’m getting at. In order to catch people in their native task environment, you’ll have to do some form of live online recruiting.

Live Online Recruiting

This method assumes two things – you are usability testing something that has a web site associated with it, and you can convince the I.T. or Web director to let you temporarily place some code on that page. Not always possible, of course. This code can be a static link that goes to your Zoomerang survey, or in the case of Ethnio, one line of JavaScript that triggers a DHTML layer recruiting invite. Here are the details of this method:

• Got webernet? You’ll need that editorial access to a web site or web application related to the tool you are testing. Why do you need this? Because that’s where you’re going to place the invite to your live recruiting form. You can recruit live for software by asking people from your organizations web site if they are going to be using, say, QuickBooks, for anything that day. Then ask them to describe the task they will be performing. You’ll get real insight just from that alone, but then you can call them before they’ve opened up QuickBooks for the day.
• We all hate pop-ups but this is the best way to do live recruiting. We use a DHTML layer that cannot be blocked by a pop-up blocker inside Ethnio, but you can also insert a link or a feature box onto a single page.
• Call Them. Use your phone to make first contact with participants instead of email. You may lose the ability to get people to pre-setup for screen-sharing, but they have to be able to start right away if you want to get the true live behavior.
• Six live recruits per hour is the minimum response rate to do live recruiting (that we suggest). This means you need to have at least six people to choose from, in our experience, in order to not have to schedule anybody in advance for your study, ever again. If the site or web app you’re testing doesn’t have that kind of traffic, you could sit around and wait forever, but that generally isn’t much fun. With six per hour, you can conduct one session per hour and get 5-6 sessions per day in easily. Samurai moderators can do 8-10 per day. Grab a second moderator and you can do 20 users in a day.
• Use any old survey tool to put your recruiting screener questions into. Just be prepared to do a whole lot of exporting to excel to get the verbatim responses. You could also build your own live recruiting tool in Ruby on Rails. That would be mad Web 2.0. There is a certain live recruiting product out there as well.
• It’s Tricky to Call Fast. Calling Live Recruiting Respondents within a few seconds can be tricky. It involves a lot of monitoring of your live responses coming in, but when you get someone within a few seconds, it yields awesome data.
• Extremely Narrow Recruiting Quotas don’t really work with live recruiting. This means that as you narrow down your potential target audience you won’t be able to easily get live users. If you need to recruit a married database administrator from the east coast who enjoys marmoset photography and live folk music, you’re going to need to schedule them in advance.
• But I want new users. A common misconception is that you cannot recruit brand new users with an online invite of some kind from an existing web site. We’ve found that using the live recruiting method to call within 10 seconds of them arriving at the site and filling out the screener will give you nearly the same results as getting users from an agency or craigslist ad that swear they have never used your web site before. Just add a question in your screener that asks: “How many times have you visited the site before?” You can also do a mix of “new user” email recruits and live recruits and see for yourself.
• Lady Luck. In reality, live recruiting takes a lot of luck to get people before they have started actually using your site. But even calling within a few minutes can yield good results, and you can absolutely mix and match recruiting methods A common setup for us is 8 users live, and 8 users recruited by an agency.

Observing

This is where you watch someone else’s desktop behavior using a screen-sharing tool. You just have to pick the screen-sharing tool that’s best for you. No matter what tool you use, some people won’t be able successfully share their screen. We see about an 85% success rate just getting someone to successfully install a screen-sharing tool. The participant often has firewalls, no admin installation privileges, etc. Here is a list of my favorite tools, and you can find updates and a more detailed list at the remote usability wiki:

• Breeze (It’s in Flash. Its slick. Very reliable. Kinda pricey)
• UserVue (Coming soon from TechSmith, pricing not yet determined)
• GoToMeeting (Least expensive. Speed is much better than WebEx. Also, if you are using the “Reverse Morae + GoToMeeting” method to conduct remote sessions, you can use the features and flags inside Morae to log behavior, which is cool. )
• Co-Pilot (Based on open source VNC, which rocks.)
• WebEx (Slow, and they once billed us $3,000 for two sessions. I hope they fixed that glitch.)
• Ethnio (Early release, hard to get an account, and only tool with integrated audio recording and recruiting as of July, ‘06.)

Tips for Remote Moderating

Once you’ve chosen a tool for the screen sharing, you’ll have some questions about how to contact participants and interact with them over the phone. Do you prefer taking notes by hand? You can still do that with remote testing but you might consider giving the old evil machine a try for note-taking. Coming up with a system for tagging quotes, behavior, and video time code as it happens will make your life so much easier. Here is a list of things to keep in mind when you call the participant:

• Practice Muting. Come up with a phrase you can use frequently to put the participant on hold. Practice saying, “Please hang on a sec while I adjust something on my end.” You can say this to your participant for ANY reason, so that you can put them on mute. If a fire breaks out at your desk (it happens), “Can you hang on while I adjust something on my end?” works great. If an observer starts laughing hysterically and calling your participant a moron, just go “Can you hold on again one more time while I adjust something again?”, press mute, then slap the observer with a reverse open palm.
• Success Rate. Only 80-85% of the people you call will be able to participate (and sometimes less). Some will have insane firewalls that don’t even work with the workhorse WebEx. Some will have kids that trip over Ethernet cables. Many will be whispering to you from their cubicle so quietly you can’t hear a d*mn thing they’re saying. Just plan for this in the same way you recruit extra lab users to accommodate for flakes.
• Scary Willingness. Almost everyone you call will be willing to install any crazy screen-sharing tool you ask them too. This is scary, and I don’t know why people are so willing. If someone doesn’t want to, thank them and move on. Users recruited live are plentiful.
• Outside Observers. The absolute most fun way to do remote testing is in a big conference room on-site with whoever is paying for your study (client, your company, etc). Use a projector and speakers hooked up to your phone tap, so that all the engineers, designers, and stakeholders can come in to watch live. If the moderator uses a headset, people can have low-level conversations in the same room and the directional mic won’t pick it up. They can also tap the moderator on the shoulder or send them an IM if they want follow up questions or something changed about the moderating. You can also setup remote observers with most of the tools out there by having them join the meeting. Make sure observers are invisible by hiding the participant list, and make sure observers are muted if you have them call in to a teleconference to listen to the session. Your participant may guess there is more than one person listening to them, but there is no need to make it obvious by having outside observers giggle or sneeze when the user does something silly. I mean interesting. Something interesting. Set these three ground rules posted on the door:
  • No loud laughing or giggling.
  • Speak at whisper level.
  • No yelling out what they think the participant should click on.
• Study Design is Up to You. You can use the live recruiting method but have participants do all pre-determined tasks, or even measure time-on-task. You can mix online scheduled with live recruiting, or divert some users to a prototype, just as you could in person.

Recording

Ahh, recording. You’d probably like to have the video and audio from your sessions afterwards so you can view, edit, and share them with your team. The easiest way would be to use a screen-sharing tool that has built-in audio and video recording, but until the Astoria Project Beta is complete or Ethnio is fully launched, you’ll have to use a combination of methods:

• Camtasia + Phone Patch. You can use a screen recording tool, like Camtasia, combined with this phone tap from JK audio to record to AVI. You just plug this tap in between the handset and your PBX or Analogue phone, and then the microphone input on the computer you’re using to test. The JK tap is inexpensive and the best quality phone audio tap on the market. We’ve tried the ones from RadioShack, HelloDirect, and WebEx, and they all pick up interference or send low quality signals.
• Teleconferencing. Another method is to use a teleconferencing service that records the phone call for you – Breeze alledgedly does this although we’ve never got it to work. You still have to use a screen recording tool such as Camtasia.
• Speakerphone Rigged. Yet another method is to position a microphone next to the speaker on your phone, or speakerphone. This is officially jury-rigged. It can work, but it’s susceptible to all sorts of problems. Mic getting knocked over. Noises in the room. Plus, nobody wants to listen to you over a speakerphone because you just don’t sound like yourself.
• Reverse GoToMeeting + Morae is a tricky but popular method. You have to give the participant control of a computer (through GoToMeeting or WebEx, etc.) on your end that has Morae installed. Then you use Morae to record all the participants behavior and somehow get the audio in. The main problem is that the participant is surfing super slow, because they are remote controlling the Morae computer. This will be unnecessary with the Astoria Project release, I’m sure.

International Love

Many people get into remote usability strictly for the ability to test international participants. Here are the most important things you need to know:

• Plan for about double the time that a domestic remote usability session would take.
• Use a native language moderator if you can, or AT&T language line service to get a native language operator on the phone doing real-time translation between you and your user. Be prepared to put subtitles in your highlight video using a video editing program for the coolest sounding highlight clips ever.
• You can also recruit English speaking participants in any country, as long as they verify they have been a resident for more than, say, 10 years..
• Realize that your cultural biases may have a huge, and difficult to predict, impact on the results of your international remote research. Just like in-person, but easier to forget about since remote testing is just so much easier than flying to Beirut all the time.

Incentives

We use Amazon gift certificates because they only require an email address to fulfill. You can use any incentive you like, but if it’s a check or an American Express gift card, you might have to spend a considerable amount of time with every person verifying their address and name spelling. I’m sorry, could you please spell Moheekuwaka Drive for me?

Make it happen.

If you’d like to get started with remote testing, you have three easy steps. First, choose a recruiting method, then decide on a screen-sharing tool, and third, figure out if audio and video recording is important to you and decide on a method for that. Then you just have to give it a whirl, ideally with a trial study, or you can just use it for an actual project. If you’re already doing some remote usability, don’t forget to experiment with different study designs and new tools. In 1999, we did 5% of our studies remotely at Bolt | Peters using Timbuktu or PcAnywhere. By 2005, almost 95% of our studies were conducted remotely. Now we hardly ever leave the computer screen, which is everyone’s goal, right?

Nate Bolt is co-founder and CEO of Bolt | Peters where he works on remote usability and ethnography. Having overseen hundreds of moderated remote usability studies for clients like Oracle, Time Warner, Princess Cruises, and Hallmark, he led the creation of the first moderated remote usability application, Ethnio. Nate speaks regularly about remote research and created a degree titled “Digital Technology and Society,” at the University of California, San Diego, which focused on the intersection of technology and mass population usage. He also studied at the Sorbonne in Paris, where he was jailed briefly for playing drums in public without a license.

Name (required)

Mail (will not be published) (required)

Website