Unexpected Complexity in a Traditional Usability Study
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205
Article Contents
Users were satisfied by the first simple solution
A closer examination of the process users followed may help to account for some of this difference, but the principal point to be made here is, users assumed that they were dealing with a simple problem, so once the handbooks suggested to them that they had found the simple solution, they didn't look any further for more complex answers. Yet, deciding whether or not a comma is required after the introductory clause "In America" required that users needed "to be aware of the entire situational context in order to make good decisions" (Albers, p. 263). Whether or not the comma is required depends on the rhetorical situation in which the sentence occurs (it is required in a formal piece of discourse such as a business proposal and may be omitted in an informal medium such as a letter to a friend). However, because the handbooks functioned acontextually, they never signaled to users that context might be a factor in their decision-making processes.
For example, the DK users would read the sentence in question and identify the introductory clause pattern (a few users actually called it an "intro clause"). They would then quickly decide that they were dealing with a question about commas and they were very successful at identifying where they should go in the text to locate more information. Because printed pages were used in this study, eye-tracking systems that could have confirmed where users were looking on the page were not available to the researchers. Nevertheless, all of the users appeared to read the large bold heads that said, for example, "Use commas to make numbers, place names, and dates clear." Often these headers were all that users needed to decide whether or not they would find information they were seeking. In the main, users skipped prose passages. Once users found headers that suggested they might be close to the type of comma use they were seeking, they only scanned the examples on the page. They looked at examples in a very specific way-i.e., to decide if they could match the syntactic pattern they were seeking to the examples. Once they found the example that matched the pattern, the use of commas provided by the example was the only answer they felt they needed, and they rarely read any prose text to confirm the accuracy of their decisions. Unless there were additional examples or some other visual clues to suggest that the decision might be more complex, users assumed it was simple and read no further.
Users scan headers and examples and skip text
Although this scanning behavior was new to our clients, it will come as no surprise to usability professionals who have observed users of computer or software user guides. Given that users read headers and scanned for examples in order to decide whether or not to read a page or a section, it might be easy to assume that the visual approach used by the DK prototype would be far easier for users to scan (see Figure 2). However, it should be observed that the logic of Hacker's pages also enabled users to aggressively seek out examples on a page and then slavishly follow those examples. Hacker's pages use a maroon colored header to state the comma usage (e.g., "Use a comma after an introductory word group"), and the pages set off examples from the rest of the text with a maroon bullet, a different font, indentation, and double-spacing. Users complained, sometimes vociferously, about how they "hated" Hacker's small fonts and the fact that the page design made them flip around and read too much. But they were also careful to qualify this by saying that they were, ultimately, able to find the information they wanted. However, their performance on the question of whether introductory commas were required or optional was essentially the same as the DK prototype. This is startling because, even though Hacker states on the same page that there are exceptions when an introductory comma may be omitted (see Figure 1), these users did not read the exception to the rule. They read the header, they looked at the example, and they decided that they had all the information they needed in order to make a judgment.