A Modified Delphi Approach to a New Card Sorting Methodology
Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30
Article Contents
Generating information structures from card sorting study results
Once the two studies were completed, information structures were generated based on pre-defined guidelines that guided objective creation and limited expert interaction. The same guidelines and analysis methods were used for the results of the Modified-Delphi and Open card sorting studies.
The results were recorded in a spreadsheet, similar to the popular spreadsheet template published by Joe Lamantia (2003). For each card sorting study, a separate spreadsheet was created to record the results. Categories generated in a study were given a separate column and similar categories labels were combined. For each participant, the card topics were listed in the category it was placed by the participant. When the card topics from all of the participant sessions were placed, an agreement weight (see Formula 1) was calculated for each card in each category. The agreement weight for a card topic is calculated by dividing the number of occurrences of a single card topic in a category by the total number of topic cards.
An agreement weight is a way to describe the strength of a card in a single category. This calculation is used instead of a correlation because a correlation finds the relationship between multiple variables. For example, a correlation would be used to find the relationship of a card between two specific categories (two variables). The agreement weight finds the single strongest category (one variable) for a card.
Formula 1. Calculating agreement weight
Result structures were created by organizing cards with greater than 50% agreement weight into categories in the final information structure. Greater than 50% agreement weight means that more than half of the participants agreed on the location of the card. These high percentage agreement cards also helped determine the strongest categories in the information structure. Cards with less than or equal to 50% agreement weight were organized based on their highest agreement. Additional categories were created as cards with lower agreement percentage that did not fit in to existing categories were added as needed. Agreement weight ties between categories were decided based on one of the following heuristics, in order of importance:
- If the category to be organized in was created based on inclusion of cards with greater than 50% agreement weight, that choice was selected.
- If the selection of a category would result in the creation of a new category that was not popular with the results, that choice was omitted.
- If the organization of a card in a specific category was obviously illogical, that choice was omitted.
The third heuristic was rarely relied on because of its need for expert opinion, but necessary in order to prevent an anomaly that may affect the later studies. The goal of these guidelines was to create information structures that explicitly represented the results of the card sorting study and not the expertise or opinion of the analyst.