Enter Amy Buckner, Managing Partner and Co-Founder of AnswerLab, a user experience research company based out of San Francisco.  I’ve worked closely with Amy on a couple of usability tests and wanted to see how AnswerLab is approaching mobile usability.

BRADLEY HEBDON: Last time we chatted you mentioned AnswerLab was focusing more on usability testing for mobile. Is this still the case?

AMY BUCKNER: Yes, clients are devoting an increasing amount of resources to develop both mobile apps and WAP sites. There are 2x as many mobile devices in the world as there are TVs . . . and 4x more mobile devices than PCs. If your company doesn’t currently have a mobile strategy, it will – and very soon. At AnswerLab, we’ve been focused on exploring new research techniques to understand the user experience with prototype and live mobile sites and apps.

BH: What sorts of systems/devices are you testing?

AB: Almost everything. To get a true picture of how your WAP prototype will perform, it’s critical to conduct usability testing across a variety of devices – from Blackberry to iPhone, from touch screen to clamshell – each interface poses a series of user experience challenges. What works on one device, might break on another. Clients who are serious about mobile web design, should have a comprehensive library of devices for both user testing and QA. It’s not easy, but it’s imperative to developing a good user experience.

BH: How are you approaching the actual usability test? How is it different from usability testing for the web?

AB: Mobile testing is dramatically different from web-based testing. First, as mentioned earlier, it’s impossible to standardize on one device. In traditional usability testing, you can set up a computer at various screen resolutions and be confident you’ve covered most of the usage situations. In mobile, you need to test on multiple devices. Secondly, mobile devices are . . . well, quite mobile, making it difficult to focus a camera on the screen during testing and recording. We’ve used both remote-controlled cameras and in-room videographers to constantly keep the back-room picture in focus. Thirdly, finding the right participants is critical. When testing a WAP site, you want users who would naturally be typing in a URL on their phones and browsing WAP sites in real life.  Unfortunately, most mobile device owners have never done this . . . but think they have. It can be a real shock in the lab if you don’t have an air-tight screening process.  Also, your analysis needs to focus on findings that apply across devices vs. findings that are relevant to only one device, thus complicating the reporting process.

BH: Are you testing them in a lab setting, or “in the wild”?

AB: Both. If we’re looking for usability issues around downloading an app or for basic tasks, we typically conduct lab-based testing. This ensures we can observe users’ behavior first hand – and help guide users if a prototype is involved. Clients can watch this live from behind the glass or remotely via our streaming capability.

The ‘In the Wild’ testing is one of our favorite methods. Recently our client, Yahoo!, and I presented a case study at the UPA conference of an interesting method for understanding user behavior in context of using Yahoo! Go / OneSearch. For this study, we wanted to uncover the basic scenarios in which users would choose to search in Yahoo! Go and any user experience issues as a result. So, we sent 9 typical users ‘into the wild’ to experience Yahoo! Go for a month. They answered daily surveys via SMS, carried around a pocket-sized questionnaire (laminated like a credit card) and called a voice mailbox repeatedly throughout the month to answer questions about their experience searching. They also sent photos of their surroundings to help us better understand the context of where they were and what they were doing when they used search. We received photos ranging from Fry’s Electronics store (shopping usage scenario) to airplanes on the tarmac (travel usage scenario). The findings had deep business impact for Yahoo!, as they informed improvement of search results relevancy, better product differentiation, and opportunities to drive distribution partnerships. 

BH: What kinds of tasks are you giving participants?

AB: It really depends on the research method. For more ‘in the wild’ studies, participants develop their own tasks. They use the product however they normally would in real life. In this case, we’re trying to learn more about what the tasks would be – how are people trying to use the product and why? It helps us understand the possible use cases.

For lab-based studies, a typical session starts with either downloading an app or accessing the WAP site, followed by an open-explore task to see where users naturally gravitate and whether they understand the value proposition of the product. After the more open tasks, we move on to directed tasks to explore the usability of specific features and functions. Of course, the more directed tasks really vary by the type of site or app. As you can imagine, a financial services site will have dramatically different tasks than a consumer package goods recipe app designed for in-store usage.

BH: Are there re-occurring mobile usability issues that you have identified as a pattern?

AB: Definitely. The most common issue I’ve noticed is that WAP sites often break basic navigation conventions.  This month on Facebook and Twitter, AnswerLab is doing a series on mobile usability tips based on issues we’ve identified across studies. Here are a few:

1) Users’ expectations for mobile web browsing are heavily influenced by computer-based browsing. New conventions only confuse users.

2) Icons should be used sparingly, and if it is absolutely necessary to use them, stick to universal symbols.

3) Real estate is limited, but including a clickable breadcrumb is a pretty good investment. Find more tips at www.facebook.com/answerlab.

BH: Currently, what are the most and least usable mobile devices out there?

AB: Our recent testing has been primarily focused on a WAP site or app vs. the specific devices. However, in the process of testing, we’ve definitely uncovered devices that tend to ‘get in the way’ more than ‘aid’ in navigation. Clamshell phones tend to pose the most user experience problems when users attempt to navigate WAP sites. Of course, users seem to breeze through the iPhone. However, only 2% of the population currently has one.

BH: What device(s) do you own?

AB: Ahhh. . .you don’t want to know. I’m a very loyal Verizon Wireless subscriber. With my travel schedule, I need reliable access everywhere.  I also prefer the phone functions to be separate from the keyboard – and for the keyboard to be tactile. These requirements seriously limit the choices. My current phone is a Verizon SMT. It crashes, has horrible battery life, and is generally way uncool. I’m holding out for the Palm Pre on Verizon, which should be available in 6 months.

BH: Got to Love the Pre!  Where do you see mobile interfaces 10 years from now?

AB: Interfaces are only as good as the device. I hope we see much more convergence and standardization across mobile devices so that mobile interfaces will be more easily built and tested. Some folks talk about the growth of touch and voice-enabled commands; those seem like very near-term advancements. Thinking further in the future, our team believes mobile devices will become almost as small as a bottle cap with projection capabilities. Imagine a full-sized keyboard projected onto the table and a screen projected onto the wall. The mobile device will then become the hub for all your communications – home HVAC, keyless entry, burglar alarms, work, shopping, social networking, you name it.  Now that would be a dream come true – everything controlled by a single device in the palm of your hand – and no more black, leather laptop bags!

About Amy Buckner
Amy has been helping Fortune 500 companies improve customer experiences for over 12 years. In 2004, Amy co-founded AnswerLab, which conducts custom user experience research to improve websites, software, and mobile applications. At AnswerLab, Amy is responsible for the company’s professional services and finance. She speaks frequently at conferences such as UPA, iMedia, and the AMA – and has special expertise in mobile user experience research.

Prior to founding AnswerLab, Amy served as Director of Professional Services at Vividence Corporation (now Keynote Systems), a provider of online research products and services. In her tenure, Amy developed the company’s automotive vertical in addition to building and managing the West Coast research team. Before Vividence, Amy was a manager at SmartPlanet, an e-learning company founded by Ziff-Davis, where she was responsible for user experience research and representing the “voice of the customer” internally. Prior to SmartPlanet, Amy managed website development for a boutique management consulting firm and consulted with the Japanese Ministry of Education for three years in Hokkaido, Japan

Amy graduated Phi Beta Kappa and magna cum laude from Vanderbilt University with a degree in English and East Asian Studies. She is fluent in Japanese.

Related posts:

1. Designing the Palm Pre: An interview with Michelle Koh
2. Caution: 5 Common Usability Testing Mistakes to Avoid
3. Mobile Web Design trends for 2009

1. Recruiting Unsuitable Participants
It’s vital that you validate a design through the eyes of your user. But who are your users? Work with marketing folks to understand your target audience, and screen potential participants accordingly.  If you have recruited participants that are too advanced, they might not uncover usability issues that a more typical user might experience.  Regardless of how good the moderator might be, if you don’t have suitable and relevant participants, you won’t be getting an accurate usability assessment.

2. Not identifying the Critical Tasks
Have you identified the primary use cases and goal-oriented tasks?  Really think critically here, and focus on the  most important ones only. Hopefully you have already designed the user interface around these tasks, so bring them into the test. Write a test script that includes a moderator guide, and provide context to the tasks. Make users feel like they are in a real-life situation by telling them a story.E.g. “It’s 5 days before Christmas and you need to buy your nephew in Germany a Nintendo Wii”. Critical tasks are not only a usability concern, they’re equally impactful on the business. 

3. Testing Too Much For Too Long
Don’t feel like you have to test every single aspect of a user interface, nor every single possible task. Focus on what’s critical and if necessary, break the test into multiple phases. After 45 minutes or so, most users will tire and will have trouble focusing. So keep the sessions to under an hour if possible.

4. Allowing Participants to be Designers
As a general rule of thumb, you should be looking for what participants DO more than what they say. It’s OK to ask what their impressions are, especially when comparing a new design against an old one. Just be careful about taking their feedback literally.  E.g. A user may have a personal preference for red, and suggest that all buttons should be red and flashing. Now, taking this feedback literally could actually introduce usability issues. Observe, filter and interpret.

5. Asking Too Many Questions
Keep in mind that a usability test should be focused primarily on task completion, navigation, comprehension, and interaction. While it’s OK to ask some questions to understand how satisfied they are with the experience, most of the findings should be derived through observation and not questioning.

While these are what I consider the top 5 mistakes made, there are many more to consider. What other common mistakes would you add to this list?

Related posts:

1. Interview: Amy Buckner on Mobile Usability
2. Usability Study suggests Bing topples Google?
3. 20 User Experience Books you should own