Unexpected Complexity in a Traditional Usability Study
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205
Article Contents
The visual handbook tried to combine too much in one graphic
Another observation that appeared to have contributed to the problems users had with inappropriate citations and with users' problems with correct comma usage had to do with the overuse of a single graphic or visual. It may be that the authors of the DK manuscript were limited to a strict page count, or they may have been restricted in the number of graphics they could use. However, the authors appeared to be attempting to force as much information as possible into graphics and visual elements like those illustrated in Figures 2 and 3. This led to visuals
- that actually produced errors,
- that confused users so they ignored the information, or
- that led users to complain that the book was "tangled up" and visually "messy."
For example, user 11 complained that the visual shown in Figure 2 "threw" her because it was trying to do too much, and she went on to explain that the information around the base sentence "I make time to play outside" is trying to illustrate five different syntactic structures in the same visual. Similarly, user 10 complained that the "pattern pages were distracting" because the "pattern" pages were those that used visuals like those shown in Figure 2 and 3.
Because of its complexity, the graphic in Figure 2 appeared to contribute significantly to the performance errors and decisions not to read the text described in the previous section. In fact, it should be observed that few users actually looked at or commented on the graphic during the actual think-aloud protocol. The test administrator often had to take users to the pages in the post-test interview and ask them about the visual. During these reflective moments, users explained that they did, actually, glance at the visual; however, they complained that they decided that the graphic on 432 was "too busy" and would require too much effort to understand. Consequently, they skipped it. This was an unfortunate decision since six of the eight comma uses from the punctuation scenario needed information described on page 432.
It should be noted that this was not the case for the visual shown in Figure 4. Users observed the pattern, understood it, and commented favorably on its clarity. This led us to question why Figure 2 was ignored and Figure 4 was success.
Figure 4. Successful single-purpose visual
Part of the answer we believe lies in the scanning behavior described in the previous section. Users scanned almost exclusively to match syntactic patterns in the examples. In Figure 2, the most prominent feature of the visual is the base clause or main idea, "I make time to play outside." The clauses or additional information that are added to the main idea are all visually subordinated to the large, uncolored, and uncluttered main idea. Yet, it is precisely this subordinated material which users needed in order to match the syntactic structures they were seeking. Additionally, instead of having to examine one structure and then decide if it matched the pattern they were seeking, Figure 2 required that they consider at least five patterns in the same visual. Conversely, the more successful Figure 4 does not subordinate elements, and it does not attempt to combine multiple patterns into one visual.
The visual as shown in Figure 3 was even more problematic than Figure 2, however. This is ironic because users were nearly unanimous in their positive comments about Figure 3 and how effectively they thought that the visual would help them to produce a properly formatted MLA works cited entry. However, this visual actually produced errors because it failed entirely to alert users to the problems faced when citing an article from a book, an edition, a corporate author, an edited collection, etc. Users were overwhelmed by the large font, yellow highlighting, and underlining of the pattern provided and failed to recognize that additional information would be required. Although once again the information they needed was provided, the treatment of the material in the visual misled users into thinking that they had adequately responded to the task. Users believed that, if they supplied the "Author's Name," "Title of the Book," "Place of Publication," "Publisher," and "Year," then they had provided all the data needed for the works cited entry, when, in fact, they had not. As a result, users commented positively on the visual; yet, every single user failed the task. It would be unfair to assert that the visual in Figure 3 was entirely responsible for this failure, and the next section discusses other factors that appeared to lead users of both the Hacker and DK texts to produce incomplete works cited entries. However, asking the visual in Figure 3 to accomplish too much certainly appeared to contribute to the problem, and once again, we observe that the failure of the information product to signal to users that they were dealing with a complex problem resulted in positive user evaluations and poor user performance.