upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30


Open card sorting is used by information architects to gather insights from users to incorporate feedback into an information architecture. In theory, it is one of the more inexpensive, user-centered design methods available to practitioners, but hidden costs make it less likely to be conducted properly and affect the quality of results produced. The following proposes a new card sorting method called the Modified-Delphi card sort to replace the Open card sort. The Modified-Delphi card sort is based on a well-known forecasting technique called the Delphi method. Instead of producing individual models that are then analyzed as a whole, participants work with a single model that is proposed and modified throughout the study. The Modified-Delphi card sorting method produces more useful results to aid in the design of an information architecture than the Open card sorting method.

A series of studies were conducted to directly compare the Modified-Delphi and Open cart sorting methods. First, two parallel studies using both methods were conducted with the same dataset and number of participants. Then, two studies were conducted on the results of the parallel studies: a heuristic review and ranking with information design experts and an Inverse card sort with additional users of the proposed architecture. The Modified-Delphi card sorting method produced results that were found to be at least as good as the Open card sorting method results and in some cases, better.

Practitioner's Take Away

The Modified-Delphi card sort is an exciting new method that promises to replace Open card sorting as a pre-design method in information architecture. The following are some of the discussed benefits learned by this study:

Article Contents

Download Article

A Modified Delphi Approach to a New Card Sorting Methodology