With mobile devices being used to perform an ever increasing number of functions and tasks, mobile usability has never before been as critical as it is today. Jakob Nielson recently posted an article about mobile usability and concluded that there is a lot of room for improvement.
Enter Amy Buckner, Managing Partner and Co-Founder of AnswerLab, a user experience research company based out of San Francisco. I’ve worked closely with Amy on a couple of usability tests and wanted to see how AnswerLab is approaching mobile usability.
BRADLEY HEBDON: Last time we chatted you mentioned AnswerLab was focusing more on usability testing for mobile. Is this still the case?
AMY BUCKNER: Yes, clients are devoting an increasing amount of resources to develop both mobile apps and WAP sites. There are 2x as many mobile devices in the world as there are TVs . . . and 4x more mobile devices than PCs. If your company doesn’t currently have a mobile strategy, it will – and very soon. At AnswerLab, we’ve been focused on exploring new research techniques to understand the user experience with prototype and live mobile sites and apps.
BH: What sorts of systems/devices are you testing?
AB: Almost everything. To get a true picture of how your WAP prototype will perform, it’s critical to conduct usability testing across a variety of devices – from Blackberry to iPhone, from touch screen to clamshell – each interface poses a series of user experience challenges. What works on one device, might break on another. Clients who are serious about mobile web design, should have a comprehensive library of devices for both user testing and QA. It’s not easy, but it’s imperative to developing a good user experience.
BH: How are you approaching the actual usability test? How is it different from usability testing for the web?
AB: Mobile testing is dramatically different from web-based testing. First, as mentioned earlier, it’s impossible to standardize on one device. In traditional usability testing, you can set up a computer at various screen resolutions and be confident you’ve covered most of the usage situations. In mobile, you need to test on multiple devices. Secondly, mobile devices are . . . well, quite mobile, making it difficult to focus a camera on the screen during testing and recording. We’ve used both remote-controlled cameras and in-room videographers to constantly keep the back-room picture in focus. Thirdly, finding the right participants is critical. When testing a WAP site, you want users who would naturally be typing in a URL on their phones and browsing WAP sites in real life. Unfortunately, most mobile device owners have never done this . . . but think they have. It can be a real shock in the lab if you don’t have an air-tight screening process. Also, your analysis needs to focus on findings that apply across devices vs. findings that are relevant to only one device, thus complicating the reporting process.
BH: Are you testing them in a lab setting, or “in the wild”?
AB: Both. If we’re looking for usability issues around downloading an app or for basic tasks, we typically conduct lab-based testing. This ensures we can observe users’ behavior first hand – and help guide users if a prototype is involved. Clients can watch this live from behind the glass or remotely via our streaming capability.
The ‘In the Wild’ testing is one of our favorite methods. Recently our client, Yahoo!, and I presented a case study at the UPA conference of an interesting method for understanding user behavior in context of using Yahoo! Go / OneSearch. For this study, we wanted to uncover the basic scenarios in which users would choose to search in Yahoo! Go and any user experience issues as a result. So, we sent 9 typical users ‘into the wild’ to experience Yahoo! Go for a month. They answered daily surveys via SMS, carried around a pocket-sized questionnaire (laminated like a credit card) and called a voice mailbox repeatedly throughout the month to answer questions about their experience searching. They also sent photos of their surroundings to help us better understand the context of where they were and what they were doing when they used search. We received photos ranging from Fry’s Electronics store (shopping usage scenario) to airplanes on the tarmac (travel usage scenario). The findings had deep business impact for Yahoo!, as they informed improvement of search results relevancy, better product differentiation, and opportunities to drive distribution partnerships.
BH: What kinds of tasks are you giving participants?
AB: It really depends on the research method. For more ‘in the wild’ studies, participants develop their own tasks. They use the product however they normally would in real life. In this case, we’re trying to learn more about what the tasks would be – how are people trying to use the product and why? It helps us understand the possible use cases.
For lab-based studies, a typical session starts with either downloading an app or accessing the WAP site, followed by an open-explore task to see where users naturally gravitate and whether they understand the value proposition of the product. After the more open tasks, we move on to directed tasks to explore the usability of specific features and functions. Of course, the more directed tasks really vary by the type of site or app. As you can imagine, a financial services site will have dramatically different tasks than a consumer package goods recipe app designed for in-store usage.
BH: Are there re-occurring mobile usability issues that you have identified as a pattern?
AB: Definitely. The most common issue I’ve noticed is that WAP sites often break basic navigation conventions. This month on Facebook and Twitter, AnswerLab is doing a series on mobile usability tips based on issues we’ve identified across studies. Here are a few:
1) Users’ expectations for mobile web browsing are heavily influenced by computer-based browsing. New conventions only confuse users.
2) Icons should be used sparingly, and if it is absolutely necessary to use them, stick to universal symbols.
3) Real estate is limited, but including a clickable breadcrumb is a pretty good investment. Find more tips at www.facebook.com/answerlab.
BH: Currently, what are the most and least usable mobile devices out there?
AB: Our recent testing has been primarily focused on a WAP site or app vs. the specific devices. However, in the process of testing, we’ve definitely uncovered devices that tend to ‘get in the way’ more than ‘aid’ in navigation. Clamshell phones tend to pose the most user experience problems when users attempt to navigate WAP sites. Of course, users seem to breeze through the iPhone. However, only 2% of the population currently has one.
BH: What device(s) do you own?
AB: Ahhh. . .you don’t want to know. I’m a very loyal Verizon Wireless subscriber. With my travel schedule, I need reliable access everywhere. I also prefer the phone functions to be separate from the keyboard – and for the keyboard to be tactile. These requirements seriously limit the choices. My current phone is a Verizon SMT. It crashes, has horrible battery life, and is generally way uncool. I’m holding out for the Palm Pre on Verizon, which should be available in 6 months.
BH: Got to Love the Pre! Where do you see mobile interfaces 10 years from now?
AB: Interfaces are only as good as the device. I hope we see much more convergence and standardization across mobile devices so that mobile interfaces will be more easily built and tested. Some folks talk about the growth of touch and voice-enabled commands; those seem like very near-term advancements. Thinking further in the future, our team believes mobile devices will become almost as small as a bottle cap with projection capabilities. Imagine a full-sized keyboard projected onto the table and a screen projected onto the wall. The mobile device will then become the hub for all your communications – home HVAC, keyless entry, burglar alarms, work, shopping, social networking, you name it. Now that would be a dream come true – everything controlled by a single device in the palm of your hand – and no more black, leather laptop bags!
About Amy Buckner
Amy has been helping Fortune 500 companies improve customer experiences for over 12 years. In 2004, Amy co-founded AnswerLab, which conducts custom user experience research to improve websites, software, and mobile applications. At AnswerLab, Amy is responsible for the company’s professional services and finance. She speaks frequently at conferences such as UPA, iMedia, and the AMA – and has special expertise in mobile user experience research.
Prior to founding AnswerLab, Amy served as Director of Professional Services at Vividence Corporation (now Keynote Systems), a provider of online research products and services. In her tenure, Amy developed the company’s automotive vertical in addition to building and managing the West Coast research team. Before Vividence, Amy was a manager at SmartPlanet, an e-learning company founded by Ziff-Davis, where she was responsible for user experience research and representing the “voice of the customer” internally. Prior to SmartPlanet, Amy managed website development for a boutique management consulting firm and consulted with the Japanese Ministry of Education for three years in Hokkaido, Japan
Amy graduated Phi Beta Kappa and magna cum laude from Vanderbilt University with a degree in English and East Asian Studies. She is fluent in Japanese.