Switching Between Tools in Complex Applications
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 173-188
Article Contents
Methods of observation
Studies of multi-tasking and task switching in the literature provide a framework for categorizing data and controlling the test environment. A general taxonomy for modeling multitasking (Wild et al., 2004) enumerates possible sources of error and distortion in our test environment and protocol. The test environment provides that test participants do not inhabit different roles (developer, manager, leader, or contributor), that there are no interruptions from outside the task, that there are no proximate external forces such as job or peer pressure, that there are no events hindering one tool in favor of another, and that there is no change in work environment. The following are the most important points to consider:
- No distinct goals separate operations and actions into different but interwoven, workflows. Tasks in this study have a single goal. No other tasks compete for attention.
- The character of the goal and how it affects activity does not differ from participant to participant. The goal in this study—to finish as many subtasks as possible in the allotted time—engages all equally.
Compared to most multi-tasking studies, usability testing is clean-room control.
Learning about tool switching begins with an accurate picture of what actually happens. So, to perform the necessary basic studies, we need a technique that extracts and summarizes pertinent (to tool changes) data and metrics from testing and focuses on critical incidents with little or no "dog work" to encumber the practitioner's day job.
Logging and reporting
Microsoft used silent logging extensively and famously to gather design requirements for Vista (Harris, 2008). A web site, Hackystat, provides an open source framework that enables anyone to log their own activity on a remote server and retrieve regular reports.
Recording operating system focus and user input captures time spent with each tool and an indication of the activity. The log record's time signal indexes into a recording of screen activity and users' commentary (if any). Additionally, success in tasks and subtask steps can be measured using completion, time to completion, user satisfaction, and the System Usability Scale (Brooke, 1996; Tullis & Stetson, 2004). We use Techsmith's Morae®, which captures some tool switching information, to record screen and voice.
A Java applet logs window activity and user input from within MATLAB® capturing all activity that can be precisely distinguished. A Visual Basic for Applications (VBA) script on the back end of the logger produces timeline pictures of tool use and changes, and statistical descriptions (counts and charts) that summarize the patterns with exhibits such as those presented in this paper.