Examining Users on News Provider Web Sites: A Review of Methodology
Journal of Usability Studies, Volume 3, Issue 3, May 2008, pp. 129-148
Article Contents
Methodology review
In this section, the following methods used to collect observational data are reviewed:
- recording of task performance while thinking aloud (with measures of time, mouse and page events, and search);
- identification of information seeking behavior including (a) information seeking trails; (b) interaction variance; (c) Web pages recurrence; (d) URL frequency; (e) Behavior identification; and (f) the sequence of browsing behaviors.
Recording of task performance while thinking aloud
The author found the think aloud protocol extremely useful in understanding user actions. It did not appear to impede task performance. Users were not hesitant about the method and none exhibited or expressed difficulties with it during task performance.
When reviewing the Morae recordings, participants often explained actions enabling the reviewer to make more informed observations. For example, when reading a column of text, a participant encountered an embedded ad that he immediately skipped to continue reading below it. However, at the bottom of the ad, the column of text shifted slightly causing the user to lose his place and to perceive the text below the ad as an entirely new story. He discontinued reading and moved off the site. Because he was thinking aloud, one could readily determine why he left the page and how the ad and the column shift affected behavior. In another instance, an ad covered the entire screen. While the user appeared to be reading the ad, it was clear from her verbalization that she was frustrated by it and was actively seeking a button to remove it. However, without her verbalization, her physical expression suggested that she was interested in reading the ad.
With the appropriate hardware and software, this approach to data collection is straightforward and yields a rich data set. The author used the Morae software from which it was easy to compile data about general behavior patterns, such as time on task and navigational events. The software provides a spreadsheet-like display of interactions and time. Events are linked to the recording so when an event is selected in the spreadsheet, the software accesses it in the recording enabling one to easily discern user actions. A disadvantage of this approach is that when making detailed observations about interactions, a researcher will likely have to sort through the data, which is time consuming. For example, with Morae, one can quickly identify the number of mouse clicks in a recording but it is more time consuming to identify when a specific button or link has been clicked.
Information seeking trails and interaction variance
White and Drucker's (2007) method for examining variance among information seeking tasks was useful in three primary ways: (a) observing how a user's information seeking varied across tasks as well as how information seeking varied among users; (b) labeling and describing navigational actions to visually depict how users traversed the informational space. While other methods provided a general sense of amount (number of mouse clicks), this method helped express directionality (as did Web Behavior Graphs) and temporal order. In other words, labeling depicted navigation direction and sequence; (c) creating Web Behavior Graphs enabled information seeking trails to be visualized. The author found that once the behaviors were labeled they could be used to check WBGs and, in some cases, assist in their development.
Using data exported from Morae, the author wrote a program that identified the information seek trails and labeled them. However, in some cases, because ads and superfluous page changes were logged, it was necessary to review the recording and to prepare the data, which was time consuming.
Web pages recurrence and URL frequency
In addition to the Web Behavior Graphs and data collected about information seeking trails, Web page recurrence and URL frequency were useful to identify patterns of browsing. As noted by other researchers (Catledge & Pitkow, 1995; Jenkins, Corritore, & Wiedenbeck, 2001; Tauscher & Greenberg, 1997a, 1997b), a breadth-first pattern was observed in which the users did not stray more than a few clicks beyond a hub (home or SERP). This pattern has been associated with Web novices but in this project, participants were experienced Web users. Calculating Web page recurrence and URL frequency helped make this pattern apparent and supported data presented in the WBG. The aforementioned program written by the author was used to calculate page recurrence and URL frequency. Again, it was necessary to review the recordings and to prepare the data, to weed out ads and superfluous page changes.