Abstract
Remote usability testing, while presenting new opportunities to discover user behavior, brings unique challenges to even seasoned usability professionals. The goal of this workshop is to discuss experiences with remote usability testing and remote technology, then to develop best practices for designing and implementing remote tests.
Workshop session timeline
The workshop will be divided into four sections, two before and two after the lunch break.
8:30am Introductions, sharing expectations, introducing format and logistics
In an open discussion presenters will ask participants to introduce themselves, briefly summarize their affiliation, and lay out the key points they have identified from their experiences.
8:45am Introduction to remote usability testing techniques as described by participants; brief review of described remote usability testing arrangements
During this first session participants share their experiences of actual remote testing situations, focusing on both the strengths and weaknesses of their set up. Based on discussions that may have already occurred on the workshop listserve, they may also identify commonalities and differences from other participants’ experiences, and lay out their initial perspectives. Participants will be given a list of questions to use as discussion starting points. Each participant will share their findings, particularly the technological design of their remote usability testing set up, with the class.
10:30am Demonstration of remote usability testing arrangements; discussion of strengths and weaknesses
In this segment, participants will be asked to demonstrate for the class their actual remote testing set up, focusing on describing the technical connection and operation of the equipment and the research pros and cons that their type of testing creates. Participants will be given a list of questions to use as discussion starting points. The facilitators will provide their equipment as possible, and can assist participants recreate or simulate their set ups.
1:30pm Discussion of processes, emphasizing strengths and weaknesses; discussion of reporting practices; possible demonstration of clip inclusion
During the first half of the session participants will work in groups to identify strengths and weaknesses of the described remote testing processes. Participants will be given a list of questions to use as discussion starting points. From these, each group will be asked to nominate potential best practices, which will be shared with the class in the next segment. During the second half of the session, participants will discuss methods used when reporting findings, and show examples of how they capture and convey usability problems, e.g. video clips, screen captures, etc.
3:30pm Collection of potential best practices; nominating and discussion of final best practices
In groups, and based on the presentations they have participated in throughout the day, participants will brainstorm a list of the best practices to be used in remote testing. Group members will then list their best practices (on a board, overhead, flipchart, etc) so that, at the end, all are visible and open for discussion. In a class discussion, using the various groups generated lists, participants will come up with a final list of best practices, noting consensus areas, areas where there still needs to be more research done, and areas where there are conflicts and why.
4:30pm Review: synthesize conclusions and potential ways to develop them
In a class discussion presenters will synthesize the key points brought up in the workshop. Based on what has been discussed throughout the day, the session presenters will draw conclusions as they highlight the main points and ask for final comments from workshop participants.
Applying to Participate in This Workshop
A workshop is a closed session. Admission to a workshop requires an approved position paper from you addressing the issues suggested by the coordinator(s). Please send your position paper (which should be roughly 1 to 3 pages) to Amy Thurston, athurston@userworks.com. Position papers received by April 25 will be accepted or rejected by April 30, in time for you to register before the early registration deadline on May 2. Position papers received by May 24 will be accepted or rejected by May 29, in time for the May 31 registration discount. Papers received after May 24 will be evaluated at the facilitator's discretion.
Participant selection criteria
The facilitators will invite position papers from usability practitioners who have designed, developed, implemented, and used remote usability testing technologies and techniques. Remote usability testing, for the purposes of this workshop, could include same time-different place studies, such as telephone interviews, controlled desktop studies, etc., as well as different time-different place techniques, such as online surveys, pop-up questionnaires, clickstream monitoring, etc.
The purpose of this workshop is to compare and discuss the relative values of different types of usability testing, as well as share and compare best practices. For example, are remote participants, especially those in different time-different place studies, more likely to be deceptive than participants interacting with a live usability professional? How important, if at all, is it to have live dialogue, live screen views, and live video of the participant, and the ability for all parties to seen the same screen? Does the usability evaluator lose critical information by not seeing the body language and hearing speech subtleties or typical office interruptions by not sitting with the participant in person? Have other evaluators systematically compared results from remote and in-person tests to show differences, especially advantages and disadvantages? Might remote testing be comparable, or even preferred, for some applications and discouraged for others? If so, what would the criteria for selecting these techniques be? What are the reasons for selecting remote testing – geographical diversity, participant diversity, cost, convenience? Does remote testing support or discourage third-party observers?
The facilitators particularly welcome papers from people who are working with established remote testing arrangements, and who have conducted remote usability tests on a range of products with a variety of users in several geographically- and temporally-diverse environments. We discourage position papers from those who have not actually performed remote usability testing, but have done only face-to-face usability testing (or none at all), as the goal of this session is to develop lessons learned from the remote experiences. Participants will be selected based on the quality of submitted 3-5 page position papers describing their remote usability testing experience, and containing the following pieces of information:
- Biographical Information. Participants will be asked to describe their educational background, work experience and particular interests in the field of HCI in general and with remote usability testing in particular.
- Relation of experience to workshop objectives. Participants will be asked to describe the remote usability testing that they have performed, including the types of technologies they have used, the types of users involved in the testing, and the types of systems or products tested. Participants could also comment on technologies which they themselves have not implemented but which they believe hold promise for this type of testing.
- Best practices, common procedures, and questions. Participants will be asked to describe their customary process for designing a remote usability test, from selecting the type of technology to reporting the results, and to comment on areas that are and are not appropriately or adequately addressed by their process or which they think might hold promise. Participants will also be asked to discuss decisions that led to difficulties and/or successes in implementing remote usability testing.
- Desired outcomes of the workshop. In the final section of their position paper, participants will be asked to describe what they expect to learn and get out of the workshop.
Sample Position Paper
Biographical Information The workshop leaders are both currently employed as usability specialists, dedicating 100% of their time to usability evaluation activities. One, employed with the US Federal government, has supervised and led a wide range of remote usability tests, while the other, a contractor, has assisted and led many similar studies. The workshop leaders have recently collaborated on a series of remote usability tests using a state-of-the-art dedicated remote usability testing laboratory. These tests have evaluated various areas of a public government Web site as well as access-controlled portions of an Intranet. Nationally-based Information Centers and Regional Offices in half a dozen US cities have served as remote sites.
Relation of Experience The Web sites and applications selected for usability testing are reviewed by 8 to 12 members of the target audience. The usability goals for these applications are defined in the three categories of satisfaction, efficiency, and accuracy, and reported using a modified Common Industry Format as proposed by the National Institute for Standards and Technology. Quantitative assessment of user performance is calculated in each of these areas, and the report includes design recommendations for all usability violations.
The test environment is as follows:
Testing Facilities The users sit in a small office at each remote location, facing a computer, television, and a Polycom videoconferencing system. The user and test administrator communicate via the audio component of the videoconferencing system. (Note: See attached schematic.)
Computing Environment The user’s workstation consists of a Dell Dimension GX150 personal computer with a Pentium III processor, a Dell flat screen monitor set at 800 by 600 resolution with true color, a standard keyboard, and a PS2 IntelliMouse with a wheel. The access to the tested application is via an Internet browser, connecting to the Internet over a dial-up modem or network connection to the organization’s Wide Area Network.
Audio and Video Connection As the user completes the initial questionnaire and signs the consent form, the connection specialists in the Usability Lab and in the remote office establish a videoconferencing connection via the Polycom videoconferencing system and engage the NetMeeting software for application sharing. The connection specialist in the Usability Lab adjusts the settings, mixing the user’s desktop view and a view of the user into a picture-in-picture format, which is recorded by a Sony DSR-20 digital Videocassette Recorder on 124-minute, Sony PDV metal-evaporated digital videocassette tape. After the connection is established, the Usability Lab starts videotaping, and the test administrator begins the testing session.
Logging A note-taker logs notes onto a computer file using Noldus Observer V4.0 logging software. The note-taker types a two key code to indicate the observation category and enters a more detailed description in the comment section. The software adds a time code to each observation. Time codes and comments are used in data analysis and report preparation.
Event Sequence The typical session progressed as follows:
- Remote connection was established
- User completed pre-questionnaire
- User signed a videotaping consent form
- Recording equipment was turned on and operating
- Test administrator greeted the user
- Test administrator read an introductory explanation
- Logger began taking notes of the video/audio proceedings
- Test administrator directed user to begin tasks and administered the testing session
- User gave running commentary
- Test administrator asked questions as necessary
- Test administrator thanked the user for completing the session
- User filled out a user satisfaction questionnaire
- Test administrator dismissed the user
- Remote site manager faxed forms to test administrator.
Best Practices and Questions There have been many positive factors in remote usability testing. Overall, the sites and the participants are willing and enthusiastic, and most have been able to accomplish the tests without major inconvenience. We have been able to collect amounts and types of usability data similar to that collected in a traditional face-to-face usability test, which have led to constructive and informative usability recommendations for sites tested. The Polycom set-up, while difficult to manage setting up a connection and establishing a new session, has been amazingly reliable during an initiated test, failing only once during a testing situation. The combination of teleconferencing and computer sharing easily lends itself to traditional usability testing, yet may just as easily be applied to card-sorting, paper prototyping, or other low-fidelity testing situations.
The challenges that we have faced with the remote usability testing have been twofold. First, and most importantly, working closely with the sites has been hampered by lack of communication, training, and an apparent disconnect in expectation. The remote sites have avoided or delayed contact with our scheduler, either through email or telephone, despite repeated attempts to contact the sites and find out how they are doing with the recruiting effort, and what we can do to help if there are troubles. Their previous experience did not seem to prepare them for the task of managing the paperwork and logistics of usability testing, and although they were willing to go through dry run testing to prepare the site coordinators for each test, overall site administration has been inconsistent.
While questions about screening criteria were raised by the remote sites and answered, test participants were often found to have not met the stated criteria -- but not until the test had already begun. Our expectation to begin sessions promptly and constrain ourselves to the test protocol needed to be more flexible when working with locations whose primary function is not usability testing. Tests often started late; were interrupted by office needs, noise, and congestion; and have suffered from inconsistent relay of test materials both to and from the sites. One critical expectation that we did not express - the need to have the test participant be a fluent speaker of English - caused the postponement or canceling of some tests. The recruits themselves likewise have been challenging, and the remote sites required additional time and resources from us to locate and identify appropriate users for the testing. It seems that while the remote locations can easily find "general public" Internet users, they are not able to identify specialty users -- such as those who routinely use specific data or specific Web sites. Upon the remote sites’ initial feedback that they found this to be a difficult recruit, we compiled a list of additional recruiting resources. Due to the lack of communication described above, we haven’t heard whether these resources have been helpful.
The second problem, noted briefly above, has been to train staff in the specialty of managing the Polycom set-up. The many options available to test, diagnose, and fix problems when encountered during an attempted connection are complex enough to warrant the training of not only logistics staff, but also usability lab staff and IT support staff. The process of establishing a Polycom connection, preparing for NetMeeting, establishing NetMeeting, initiating and accepting calls, and beginning the remote desktop sharing can easily go off-track, and require extensive trouble-shooting and backtracking. Support, coordination, and good humor from logistics, usability, and IT staff has reduced (although not eliminated) and alleviated the tension inherent in these experiences.
Desired Outcomes
The workshop leaders would like to explore the following questions in this workshop: How influential is it to have live dialogue, live screen views, and live video of the participant, versus creating a face-to-face usability study? Does the slight time-delay that is created by using a remote testing set up influence the user’s perception of the quality of the application tested? Does the usability evaluator lose critical information by not seeing the body language and hearing speech subtleties or typical office interruptions by not sitting with the participant in person? What are the pros and cons of usability testing without same time and place connection? Are there potential concerns (variability of download time, user confusion about tasking, or lack of follow-up questions from the administrator) that make different time-different place so different as to be ineffective? Learning how other tests have been completed, and how other usability practitioners have addressed these questions, would greatly increase and enhance the quality and effectiveness of both the usability recommendations that we make to site designers and the reports from usability testing that affect future design.
Pre-workshop participant activities
Respondents who have been selected for the workshop will be asked to bring a schematic diagram of the hardware used in remote usability testing, and a sample of reported results. It would be helpful, but not necessary, to provide a written description of the testing arrangement and procedures. Also, the workshop leaders will establish a listserve for the attendees, on which selected participants will be encouraged to participate.
Pre-workshop facilitator activities
Prior to the workshop, the facilitators will have reviewed each participant’s position paper for information about planning, implementation, and reporting. The facilitators will transfer the information describing each of the technology arrangements to PowerPoint slides so that they may be viewed during group discussion. We will also prepare packets for each participant with copies of each participant’s sample reports, and a collected list of proposed best practices for discussion by the group.
Presentation of results during the conference
The workshop facilitators will present the best practices generated by this workshop as a poster session during the UPA Conference.
Post-conference dissemination of results
The workshop facilitators will produce a report from the workshop for publication in User Experience.
Post-conference activities
The workshop facilitators will maintain and moderate the listserve for workshop participants through the time of the publication of the workshop report. Additional workshops and group discussions can be planned as adjunct activities to other user-experience conferences.
Facilitators:
Amy Thurston has supported a variety of UserWorks' clients on projects involving accessible Web site design, strategic Web site development, and usability evaluation. She has also supported several Federal agencies move toward compliance with Section 508 accessibility requirements.
Erica Olmsted has a Master’s degree in technical communication and works as a usability specialist in the Census Bureau’s usability lab. She has conducted a number of remote usability studies and has presented information on the Census Bureau’s remote testing capabilities. She is a member of the Society for Technical Communication (STC) and is also a UPA member.
|
