Moderating Usability Tests is the place for new and experienced moderators to learn about the rules and practices for interacting that have never been described in one place before. Authors Dumas and Loring draw on their combined 40 years of usability testing experience to develop and present the most effective principles and practices - both practical and ethical --for moderating successful usability tests.
To help usability professionals, students, and novices understand these principles, the authors provide videos from their lab that demonstrate good and poor interaction as well as commentary from a panel of testing experts on why certain techniques succeed or fail. The videos are accessible from the publisher’s companion web site.
• Presents the ten “golden rules? that maximize every session’s value
• Offers targeted advice on how to maintain objectivity
• Discusses the ethical considerations that apply in all usability testing
• Explains how to reduce the stress that participants often feel
• Considers the special requirements of remote usability testing
• Demonstrates good and bad moderating techniques with laboratory videos accessible from the publisher’s companion web site
About the Author
Joe Dumas is a recognized expert in usability evaluation. He has 25 years experience as a usability professional. He as moderated or observed others moderate thousands of usability testing sessions and taught numerous students and usability professionals how to moderate. He is the author of A practical guide to usability testing (with Ginny Redish), Designing user interfaces for software, and numerous articles, both for researchers and practitioners. He is currently a Usability Consultant for Oracle Corporation. He was a Senior Human Factors Specialist at Bentley College’s Design and Usability Center and taught graduate courses in the college’s Human Factors in Information Design Master’s Degree program. He has a Ph.D. in Cognitive Psychology.
Read an Excerpt
Moderating Usability TestsPrinciples and Practices for Interacting
By Joseph S. Dumas Beth A. Loring
MORGAN KAUFMANN PUBLISHERSCopyright © 2008 Elsevier Inc.
All right reserved.
This book grew out of a tutorial that we presented at several Usability Professionals' Association (UPA) conferences starting in 2004. The success of the tutorial and the positive reaction of the participants led us to expand the scope and publish it as a book. It is intended for both newcomers to usability testing and experienced practitioners who want to reflect on their own practices.
1.1 WHY THIS BOOK?
Usability testing is now accepted as the evaluation method that influences product design the most (Rosenbaum, Rohm, & Homburg, 2000). To a large extent, successful usability testing depends on the skills of the person moderating the test. Most practitioners, however, learn how to moderate with little formal training and almost no feedback on their technique. They watch a few test sessions, moderate a few under supervision, and then proceed without further evaluation. It's not surprising that researchers have found that moderators, even in the same organization, don't follow the same practices (Boren & Ramey, 2000).
With the exception of short sections in Rubin (1994) and Snyder (2003), the literature on usability testing provides only general advice about interacting with test participants. In addition, there are no sources about either general principles or specific guidelines that provide a rationale for what moderators should or shouldn't do. Maybe this is because many people consider it part of the "art" of testing—a skill that is difficult to teach. The lack of relevant literature became a problem as we tried to teach people how to moderate. Therefore, in this book we go into depth about what to do (and what not to do) while interacting with test participants, and we present a set of rules of good practice.
Although one paper discusses situations that happen infrequently (that is, Loring & Patel, 2001), this book focuses on the common situations you will encounter in typical usability test sessions.
Over time, as we taught people how to moderate test sessions, we recognized the need to address the difficulties that moderators have with the following issues:
* learning how to overcome the initial anxiety that all new moderators experience * staying in control of the test session * being friendly but resisting becoming friends with participants * dealing with personal—rather than the participant's—anxiety while struggling with usability problems * balancing the trade-off between respecting participants' rights and pushing them to keep working * knowing when and how to provide assistance to participants * knowing how to probe for more information in an unbiased way
We hope that by addressing these and other issues in this book, we are making a valuable contribution to the field.
1.2 WHAT IS USABILITY TESTING?
Usability testing is a systematic way of observing actual and potential users of a product as they work with it under controlled conditions. It differs from other evaluation methods (such as quality assurance testing and product demonstrations) in that users try to complete tasks with a product on their own, with little help. Usability testing can be conducted in a laboratory, in a conference room, in the participant's environment, or remotely. Companies use this method to evaluate software, hardware, documentation, web sites, or any product with a user interface.
People are recruited, and usually compensated, to participate in the sessions. The goal of the study may be to uncover as many usability "bugs" as possible or to compare the usability of two products. A typical test session involves one participant and one moderator, lasts one to two hours, and includes these tasks:
* greeting the participant * explaining the participant's rights and having him or her sign an informed consent * explaining how the session will proceed * guiding participants through a set of carefully selected tasks using the product, usually while thinking aloud * recording data in one or more ways * asking participants to summarize their experience
1.3 THE IMPORTANCE OF MODERATING SKILLS
Usability testing is like many things—easy to do, but hard to do well, and improved only through practice. The moderator's manner of interaction with participants from first recruiting them to thanking them at the end of the session is critical to the success of the test, the validity of the data, and the reputation of the moderator's organization. A test moderator must be unbiased and neutral regarding the product while being open and approachable to participants. This mixture of neutrality and approachability can be difficult to accomplish.
1.4 THE GOLDEN RULES OF MODERATING
Because most practitioners have learned only by example, they have seldom thought about the underlying principles of interacting with test participants. This led us to create the golden rules of interacting for moderators of usability tests. Our rules attempt to capture the best practices of interacting and provide a rationale for the practices described in the rules. The golden rules are:
1. Decide how to interact based on the purpose of the test.
2. Respect the participants' rights.
3. Remember your responsibility to future users.
4. Respect the participants as experts, but remain in charge.
5. Be professional, which includes being genuine.
6. Let the participants speak!
7. Remember that your intuition can hurt and help you.
8. Be unbiased.
9. Don't give away information inadvertently.
10. Watch yourself to keep sharp.
These rules will help you deal with new situations that you may encounter. We discuss these rules in detail in chapters 3 and 4.
1.5 CULTURAL POINTS OF VIEW
As we developed the material for this book, it became clear that we have the perspective and assumptions of usability specialists from the United States. In the book, we don't pretend to be otherwise. Instead, we state our biases up front and, where possible, discuss the need to be sensitive to cultural differences. In some cases, we cite literature about ways people from various cultures might react, but we don't attempt to guess how moderators and participants from other cultures might react. While we cite the few studies that describe ways that people from other cultures react to testing situations, we don't extrapolate beyond those studies by guessing how people from other cultures may react to our golden rules. The role of culture in usability testing is a new and interesting topic that is just beginning to be explored. It would be wonderful if practitioners from other cultures would create localized sets of golden rules!
1.6 ABOUT THE SIDEBARS IN THIS BOOK
Throughout the book, we offer sidebars to supplement the text. There are two types of sidebar. The first type presents short descriptions of relevant research studies that shed light on the topic at hand. Entitled "What the Research Says," each focuses on the practical implications of one published study.
The second type presents interviews with a variety of moderators at different organizations to allow them to express their views about a topic. Some of the moderators are new to moderating and others are quite experienced. Hence, these are entitled "Interviews with a New [or Experienced] Moderator." We found that stating the principles in moderators' own words can be more informative and colorful than if we described them ourselves.
1.7 ABOUT THE VIDEOS THAT ACCOMPANY THIS BOOK
Perhaps one reason that interacting with test participants has not been adequately addressed in the literature is that it doesn't lend itself well to description only on paper. To fully grasp both the art and the craft of test moderation, it helps to observe the interactions—including facial expressions, body language, tone of voice, and numerous other factors—that can have a direct impact on the test outcome but are difficult to convey in words.
To provide another dimension, we created a number of short videos to accompany this book. The role-playing videos show both good and not-so-good moderating practices. The filmed discussion by a panel of usability experts gives their opinions about what they see in the videos and discusses trade-offs that you might consider in each situation.
The videos, available on the book's web site were filmed in the Bentley College Design and Usability Center laboratory and were carefully scripted to illustrate the points we want to make.
Chapter 11 describes the type and content of the videos and suggests how to get the most value from them.
1.8 ABOUT THE COMPANION WEB SITE
The companion web site for this book is www.mkp.com/ moderatingtests. In addition to the videos, you will find downloadable versions of the sample test materials in the book, such as the checklists, forms, and recruiting scripts.
Chapter TwoGetting started as a test moderator
This chapter provides some guidance to moderators who are just starting out, but our comments may also be useful for experienced moderators who need to refresh their memories or who are training others. We discuss the attributes of a great moderator, types of testing, moderator (sometimes conflicting) roles, the basics of running a test, and finally some ways to get started quickly as a moderator. If you're familiar with usability testing, most of this material will be a review.
2.1 WHAT MAKES A GREAT MODERATOR?
Moderating usability tests is not as easy as it looks. If you're reading this book, you have probably seen a number of test sessions and may even have moderated some (or many) yourself. Some moderators make it look easy and others are so nervous that you feel nervous just watching them. Some moderators are able to elicit wonderful insights from test participants; others simply go through the motions. Some remain neutral and others seem to lead or bias the participants. Why is there so much variation?
Becoming a great test moderator takes four things:
1. Understanding the basics of usability testing
2. Interacting well with test participants (our golden rules)
3. Ability to establish and maintain rapport with participants
4. Lots of practice
2.1.1 Understanding usability testing
To be an effective moderator, you need a firm understanding of what usability testing is all about. You need to know the following:
* the purpose of the test that you are moderating. The purpose determines how much and when to interact with participants. * how usability testing differs from other evaluation methods. The emphasis here is on obtaining valid data from a small sample of typical users, and letting the users speak. * how tests are designed. For example, design might be important in presenting the product and related tasks in the context of their intended use. * how tests are planned. Every task has a purpose and the order of tasks may be important to the test's validity. Sometimes it's important to probe for an understanding of the concept behind a user interface component. Stopping a task before the participant has completed it may be required in some tests but not in others. * how and why data is being collected. The emphasis may be on the quantitative measures or the qualitative measures, depending on the test goals.
Fundamental knowledge of usability testing is vital to understanding the material in this book, so if you need more information, we suggest that you read a book on the basics of usability tests (e.g., Dumas & Redish, 1999; Rubin, 1994) or take one of the many courses available.
2.1.2 The basics of interacting
The ways that practitioners interact with test participants has a huge effect on test results. Moderators (or their team mates) interact with participants throughout the testing process, including
* selecting them for the study.
* greeting them when they arrive.
* providing an introduction to the test.
* guiding them as they are performing tasks.
* interacting during the post-test debriefing.
* thanking them and giving them their compensation.
Excerpted from Moderating Usability Tests by Joseph S. Dumas Beth A. Loring Copyright © 2008 by Elsevier Inc.. Excerpted by permission of MORGAN KAUFMANN PUBLISHERS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of ContentsChapter 1: Introduction
Introduction to the problem
Goals of the book and videos
A brief history of usability testing practice
Part I: The rules of interaction
Chapter 2: Your roles as administrator
The gracious host
The objective observer
The tradeoffs between them
Chapter 3: The Golden Rules -
Part 1: the Five Core Rules
1. The goals of the test and your relationship with developers determine why and when to interact
2. Respect the participants? rights
3. You have a responsibility to future users
4. The participants are the experts; you are in charge
5. Being ?real? is being professional
Chapter 4: The Golden Rules -
Part 2: Five Additional Rules
6. Let the participants speak
7. Your intuition can hurt and help you
8. Be unbiased
9. Don't give away information inadvertently
10. Watch yourself to keep sharp
Part II: Moderating the test session
Chapter 5: First contact ? Setting the tone
Chapter 6: Interacting During the Session
Keeping the participant talking
How much to interact
Dealing with stress
Deciding when to give assistance
How to move the participant along
Chapter 7: Post-task activities
What to do first
Taking advantage of their knowledge
Presenting ratings and questionnaires
Making a good last impression
Part III: Special topics
Chapter 8: Interacting in a remote testing session
Preparing for the session
Getting the session started
Interacting when you can't see the participant
Chapter 9: In the Room vs. Out of the Room
Advantages and disadvantages of being in the room
Advantages and disadvantages of being out of the room
Chapter 10: Summing Up
Where do we go from here?
Advice for novice administrators