upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents


This section discusses the general findings and observations of the study.

Users preferred the visual approach

As Redish has suggested, traditional ease of use measures alone gave us no real sense of the complexity involved in the tasks. Overall, users reported that they preferred the DK prototype's visual ease of use to the more verbal approach used in Hacker. When asked to rank the "overall" ease of use for the two texts after they had actually used both handbooks, 9 of the 12 users preferred the DK prototype, and 9 of the 12 indicated that they would recommend it to their teachers for their entire class. Users recommending the DK prototype also appeared to have a stronger preference for their recommendations than the 3 recommending Hacker. Users were asked to indicate the strength of their preference on a scale from 1 to 10 where 1 indicated that they thought the text was "slightly better" and 10 indicated "vastly superior." The 9 users recommending DK averaged 7.44 (standard deviation was 2.35), and the 3 recommending Hacker averaged 6.00 (standard deviation was 1.0). See Table 2 for the range of scores users gave.

Table 2. Strength of Preference
  Strength of Choice Strength of Choice
  DK Hacker
  7 6
  5 5
  3 7
Avg. 7.44 6.00
Std. Dev. 2.35 1.00

During the post-test, users were also asked to give their overall ranking of the ease of use for both texts in terms of finding information (see Table 3). On a 4-point scale where 4 was "Easy" and 1 was "Difficult," the DK text received an average score of 3.33 (with a standard deviation of 0.78), and the Hacker excerpt received an average score of 2.33 (with a standard deviation of 0.89). And once again, the low standard deviation for the DK scores here are noteworthy since only 2 of the 12 users gave the DK text a score of 2 (or "somewhat difficult"), while the remaining users either gave the text a 3 ("somewhat easy") or a 4 ("easy"). Users' evaluations of the Hacker excerpts were slightly more varied, resulting in the larger standard deviation. However, the difference here is notable when one considers that only 5 users gave Hacker a score of either "easy" or "somewhat easy," and the remaining 7 users gave Hacker a score on the "difficult" side of the scale.

Table 3. Overall Ease of Use for Finding Info
  Overall Overall
  DK Hacker
  4 3
  4 2
  3 3
  2 3
  4 3
  3 2
  4 2
  4 2
  3 1
  2 4
  3 2
  4 1
Avg. 3.33 2.33
Std. Dev. 0.78 0.89

Users failed at tasks, but didn't realize it

Yet, while users' clear preference for the DK text and their overall "ease of use" scores are suggestive, it would be an error to conclude that the DK prototype's visual approach was more "usable" than a verbal approach. The ease of use evaluations above do not give a complete picture of the usability of the texts because the users' evaluations must also be considered in light of the question of whether or not users were actually able to complete the task "successfully" or not. In other words, users may initially give a positive evaluation of the ease of use for a product that they thought had helped them complete a task, but if a text misled them by allowing them to believe that they had finished the task when, in reality, the task was only partially completed, then the users' initial assessments are less valuable as a measure of usability. It is at this point that the issue of complex problem solving manifested itself in our study, and it is by means of content experts' assessment of the quality of the users' performance that researchers who are working with new clients can observe when complex problems may be disrupting the findings in a traditional usability study.

In this study, both the DK prototype and the Hacker excerpts failed the users when it came to successfully completing acceptable works cited entries for the works provided. All 12 users failed to provide a works cited entry that would have been judged satisfactory by college-level composition instructors. Even if one takes into account some of the information many instructors would consider optional-such as the use of "et al" for multiple authors or editors, the omission of words like "Press" and "Inc." from publishers' names, or the decision of whether or not to include the initials for the state after giving the city's name-even without these, users in this study omitted critical information necessary for a complete, acceptable citation. For example, users failed to list the authors of an article in an anthology, they failed to list a title of the essay, they failed to include the number of an edition, they failed to provide the page numbers for articles, and so on.

Previous | Next