1. Recruiting Unsuitable Participants
It’s vital that you validate a design through the eyes of your user. But who are your users? Work with marketing folks to understand your target audience, and screen potential participants accordingly.  If you have recruited participants that are too advanced, they might not uncover usability issues that a more typical user might experience.  Regardless of how good the moderator might be, if you don’t have suitable and relevant participants, you won’t be getting an accurate usability assessment.

2. Not identifying the Critical Tasks
Have you identified the primary use cases and goal-oriented tasks?  Really think critically here, and focus on the  most important ones only. Hopefully you have already designed the user interface around these tasks, so bring them into the test. Write a test script that includes a moderator guide, and provide context to the tasks. Make users feel like they are in a real-life situation by telling them a story.E.g. “It’s 5 days before Christmas and you need to buy your nephew in Germany a Nintendo Wii”. Critical tasks are not only a usability concern, they’re equally impactful on the business. 

3. Testing Too Much For Too Long
Don’t feel like you have to test every single aspect of a user interface, nor every single possible task. Focus on what’s critical and if necessary, break the test into multiple phases. After 45 minutes or so, most users will tire and will have trouble focusing. So keep the sessions to under an hour if possible.

4. Allowing Participants to be Designers
As a general rule of thumb, you should be looking for what participants DO more than what they say. It’s OK to ask what their impressions are, especially when comparing a new design against an old one. Just be careful about taking their feedback literally.  E.g. A user may have a personal preference for red, and suggest that all buttons should be red and flashing. Now, taking this feedback literally could actually introduce usability issues. Observe, filter and interpret.

5. Asking Too Many Questions
Keep in mind that a usability test should be focused primarily on task completion, navigation, comprehension, and interaction. While it’s OK to ask some questions to understand how satisfied they are with the experience, most of the findings should be derived through observation and not questioning.

While these are what I consider the top 5 mistakes made, there are many more to consider. What other common mistakes would you add to this list?

Related posts:

1. Interview: Amy Buckner on Mobile Usability
2. Usability Study suggests Bing topples Google?
3. 20 User Experience Books you should own