Tutorials

Qualitative user research, such as field studies, is essential to user-centered design, but findings from these studies are often merely anecdotal or impressionistic summaries, with vague or even misleading implications for design. This tutorial will teach you techniques to improve the validity and credibility of your findings, to keep them focused on design, and to help you avoid drowning in your data.

PARTICIPANT KNOWLEDGE AND EXPERIENCE EXPECTED

This tutorial is of interest to practitioners who want to improve the credibility of their findings from qualitative research, by learning practices and skills to ensure rigor and validity. It is intended for people with experience in usability, fieldwork and observation of users, user experience research, or any other user research that generates qualitative data. People with some background in user-centered work who are planning new field research projects. It will be especially useful for people who want to provide best practices leadership for multi-disciplinary teams involved in field research.

This tutorial does not focus on how to carry out basic field research techniques such as contextual inquiry and artifact walk-throughs. Rather, it focuses on how to manage and draw design conclusions from the data these methods yield. Although many of the issues we address and skills we teach are applicable to qualitative data other than field research data, such as data from formative usability evaluations, people interested primarily in analysis of open-ended questionnaires will probably not find the exercises and examples relevant.

GOALS FOR THE SESSION:

Participants in this tutorial will learn skills they can apply immediately to improve the quality of any fieldwork currently under way, or planned, including:

• Principles for maintaining scientific rigor when working with qualitative data
• Proactive strategies to make sure field data provides targeted and useful input for guiding design
• Ways to manage data to maintain accountability and facilitate analysis
• Analysis techniques including coding strategies, clustering, extracting dimensions, examining relationships, network diagramming
• How software tools can help in archiving field data and exploring patterns
• Avoiding pitfalls in interpreting and drawing conclusions from field data

HOW THIS TUTORIAL WILL BE CONDUCTED

This tutorial uses a mix of lecture, demonstration, and hands-on exercises to show how qualitative data from field research, can be used rigorously to guide crucial product definition and design decisions. The progression of topics and exercises are built around a simulated early product development. We therefore begin by presenting the product concept, the idea of developing an information appliance to take over the “communication” and “information” functions that some families (in some countries more than others) seem to take care of by posting things on their refrigerators. As is typical in the product development lifecycle, this early product concept embodies many assumptions and raises many questions, some of which the class evaluates in the course of the exercises.

As materials for the exercises, we use printouts of digital photographs of refrigerators we have collected from many countries, along with stories about the artifacts that are posted on them. We have found that these images and stories, which could have come from a true field study, point to very complex and rich behavioral and contextual dynamics. As a result, they work very well as stimuli for exercises in which we can closely examine the process of formulating and evaluating inferences from qualitative data. These new exercises have proven extremely engaging and fun for audiences, and generate lively discussion.

We will draw examples from our own extensive professional research and design experience to illustrate various analytical methods and design approaches. We will also introduce the use of various types of software to facilitate data management and analysis. We have found that the mixture of lecture, demonstration, discussion, and exercises is very effective in maintaining participants' interest and engagement, and in providing them with practical experiences that they will be able to apply when they return to their respective jobs.

The first topics we cover include an overview of the nature and challenges of qualitative data; the role of an explicit initial focus structure in organizing the research and data management; and the foundations of scientific rigor in working with qualitative data. We cover these early, because they are themes that we bear in mind throughout the rest of the day. We move on to cover the topics of

• Capturing and documenting data
• Data archiving and coding, including use of Computer Assisted Qualitative Data Analysis Software (CAQDAS)
• Integrating data across cases to explore typologies, via different types of clustering approaches.
• Higher Order Analyses, such as defining and scaling dimensions, and validating them
• Understanding and avoiding pitfalls in interpreting qualitative data and communicating it in organizations

TUTORIAL SCHEDULE WITH TIME ALLOCATION

DETAILED DESCRIPTION OF TUTORIAL

Introduction: (15 minutes)

Instructors' backgrounds, focus of class, agenda for the day, polling audience to understand their background . Introduce the early product concept that will provide the organizing theme for our exercises throughout the day.

Lecture: The Challenge of Qualitative Data and Scientific Rigor (25 Minutes)

Qualitative data from field research is essential in User-centered design. We will identify the inherent challenges qualitative data presents for drawing sound conclusions. We describe the core principles and practices for maintaining scientific rigor when doing qualitative research. These include applying the concepts of reliability and validity to analysis of qualitative data; recognizing and compensating for common sources of bias; and actively maintaining a stance of hypothesis testing of interpretations throughout the process of research. We also introduce topics related to scientific rigor that we will focus on in more depth in the remainder of the tutorial. These include maintaining data accuracy, integrity, and accountability; appropriate interpretation of qualitative data; and responsible communication of findings. The rest of the tutorial will teach specific methods to apply these principles in study planning, data recording and archiving, and analysis. Because Grounded Theory is one well-known paradigm for defining validity in qualitative research, we will give a short overview of its core concepts and point out where our approach differs.

Lecture: Creating an initial focus structure (20 Minutes)

We will discuss the need for a workable initial conceptual focus structure prior to conducting the research, and will also discuss the controversial issue of how to balance the need for pre-planning of a focus structure with the need to remain open to opportunistic discoveries and patterns at all levels. Developing a preliminary focus structure helps to ensure buy-in from the stakeholders who contribute to it, provides a shared “mental map” of the research space that enables the team to work together in the field. It provides a conceptual baseline that makes discoveries in the field more salient. It can provide a foundation for organizing data collection, archiving, coding, and retrieval. Finally, it helps keep the research focus on things likely to have practical product implications. We will cover techniques for eliciting and organizing researchable topics from the team, for structuring them at an appropriate level of abstraction, and for using the resulting focus structure as a tool to organize the research.

Exercise: Create an initial focus structure (20 minutes)

We will lead the group in a brainstorming exercise to identify issues and questions suggested by the initial product concept, which must be addressed to evaluate the concept and to guide its initial specification. This exercise tends to generate a very long list of topics at very heterogeneous levels of abstraction. We will then guide the group in transforming these topics into researchable topics at an appropriate level of abstraction, and show how they can be organized to provide a “conceptual baseline” for the research. Depending on the number of attendees, we will do this exercise in subgroups. The brainstorming portion of the exercise can be down in subgroups, depending on the number of attendees.

Lecture: Capturing and Documenting Data (10 minutes)

In field research, a tremendous amount of crucial information can be lost between the observation and the documentation. This brief section focuses on practical aspects of field data collection and recording to ensure that data records are useful, accessible, and meaningfully interpretable. We discuss tips and skills for effective and accurate note taking. We will show how the data structure can be used for focusing attention on key items for documentation. Finally, we will discuss ways to manage the post-visit debriefing to improve data accuracy of documentation.

Exercise 2 Capturing Relevant Information in a Concise Narrative (10minutes)

The goal of this exercise is to understand what makes a concise narrative data element rich and interpretable, so that it will be meaningful when retrieved later for analysis. In the exercise, we practice taking notes that isolate the core information from stories about artifacts from the simulated field study.

Lecture: Data Archiving and Coding—Methods and Tools (30 minutes)

We will discuss the challenges of and strategies for developing coding systems for classifying, retrieving, and facilitating the interpretation of the narrative data. We will show how initial coding systems can build upon the initial focus structure. We will introduce word processing, spreadsheet, database, and Computer Assisted Qualitative Data Analysis Software (CAQDAS) as contrasting tools for storing, indexing, coding, and exploring qualitative data. We illustrate this with case examples from our experience, and provide visual examples of different approaches.

Demonstration: Coding Narrative Records Using Atlas.ti (25 minutes)

We demonstrate the use of one CAQDAS tool (Atlas.ti) to code a sample visit report related to our product development simulation. This demonstration not only introduces the functionalities of Atlas.ti, but also presents the process of coding as an intellectual process of analysis and exploration of the data. We will briefly compare Atlas.ti with another CAQDAS tools such as Nudist.

Exercise: Coding as a Form of Analysis (25 minutes)

Attendees will practice coding excerpts of sample visit reports from the simulated field study, with different subgroups using different subgroups will adopt different focuses from the initial focus structure. In the discussion, we will explore areas where the narrative data compels modification of the focus structure.

Lecture: Classification and Typology (20 minutes)

Meaningful classification and the identification of key distinctions is essential for understanding variations in user populations, usage contexts, functions, concepts, and so on. Making robust, measurable, and relevant distinctions has a tremendous impact on the conceptual design of systems, and are essential for ensuring fit between the product and its customers. We will discuss the importance of using multiple complementary methods for developing and testing classification schemes to identify robust distinctions in the data. We will introduce the techniques of clustering.

Exercise 3: Clustering (25 minutes)

We will conduct an exercise in affinity diagramming as an example of a type of clustering approach. In this exercise, the group works to identify a typology of the types of artifacts people post on their refrigerators. The post-exercise discussion will explore the process of making the organizing principles of the typologies explicit and what makes a typology useful for product planning and design. We will also explore some of the pitfalls of affinity diagramming.

Lecture: Higher order analysis: Defining Dimensions (20 minutes)

We will show how the process of clustering can lead to the identification of more abstract dimensions for defining the “design space.” We will discuss the process of operationally defining dimensions and of scaling on dimensions, and show how these constructs can be evaluated. We will also explore ways of examining dimensions in combinations, via typologies, graphs, and matrices. These will be illustrated with examples from our simulated product field research. Finally, we discuss how this type of analysis contributes toward developing a more general model of the product or user space, and influences product planning and design.

Demonstration (20 minutes)

We will present samples of novel types of graphs and matrices that we have used in previous qualitative studies to search for and depict patterns in data, and to test hypotheses. These demonstrate various ways of extracting and validating concepts such as dimensions to describe the design space and the user space. We will show how a sequence of analyses relating factors can be used to evaluate higher order concepts and hypotheses.

Lecture: Higher Order Analysis: Networks And Flows (30 Minutes)

In order to model dynamics, it is useful to sort variables in ways that show their interdependencies and influences on each other. We will

Lecture: Pitfalls and Issues in Interpreting and Communicating Qualitative Field Data (30 minutes)

We discuss and provide guidelines for addressing classic issues in interpreting qualitative data. Issues covered include the small sample problem, “outliers,” appropriate and inappropriate forms of generalization, guarding against implicit quantitative inferences. We also discuss the tendencies towards distortions that can arise when research findings are communicated within organizations, and how to manage these.

Wrap up Discussion: (30 minutes)

We will lead the class in a discussion of the product and design ideas they have developed in the course of the day, and identify ways to formulate them as hypotheses that could be evaluated in further research. The purpose of this discussion is to integrate the themes and topics from the day. There will also be time for general questions.

SPEAKER BIOS

David A. Siegel, Ph.D.

Vice President

Dray & Associates, Inc.

David has worked as Vice President of Dray & Associates since 1993. He consults on usability, user interface design, and user-centered design processes. He specializes in conducting contextual user studies, as well as formal usability testing, expert evaluation of interface designs, and interface design consultation. He has experience carrying out studies both in the US and overseas.

He has published numerous articles and taught on a variety of user-centered design topics, including tutorials at professional conferences in the U.S., Europe, and Africa. He is the co-editor of the Business Column in ACM's magazine, interactions. He has recently co-authored (with Susan Dray) a chapter on planning international user studies, for Usability and Internationalization of Information Technology, edited by Nuray Ayken, and to be published by Lawrence Erlbaum Associates, Inc. this coming year.

David has helped product teams from a long list of clients to make pivotal improvements in their designs and product strategies based on user-centered research. Together with Susan, he conducted a series of iterative longitudinal field trials of successive prototypes of Microsoft's Tablet PC. These trials led to significant modification of the interactions design, form factor, and market positioning of the product. He played a leading role in designing the interface for Hewlett-Packard's first digital camera and for its desktop software, based on a process of iterative rapid prototyping with user testing at each stage. In awarding the resulting product "Best Buy" status, PC Magazine specifically praised it for its ease of use.

His education includes a B.A. in psychology from Princeton University, and a Ph.D. in psychology from UCLA.

Susan M. Dray, Ph.D.

President

Dray & Associates, Inc.

Since 1979, Susan has worked in the field of human factors to increase the quality and intuitiveness of user interface designs for users around the world. She consults internationally on interface design and usability. She combines expertise in contextual and ethnographic research and usability evaluation with a cross-cultural perspective, gained from consulting and teaching in 18 countries.

Susan is a leader in the Human Factors profession nationally and internationally. She has given over 100 talks at conferences and symposia in the U.S., Europe, and Australia, including plenary and keynote addresses at many international professional conferences. She is also a frequent presenter of tutorials at these conferences. In addition, she has published numerous papers and book chapters. Her most recent book chapter, written with David Siegel, is on planning and conducting international user studies. Together with David Siegel, she currently co-edits the Business Column of the Association for Computing Machinery's magazine interactions.

Susan was elected a Fellow of the Human Factors and Ergonomics Society and has chaired both the Organizational Design and Management Technical Group and the Computer Systems Technical Group of this same organization, as well as the Computers and Communications Scientific and Technical Committee of the International Ergonomics Association. Susan also served as North American editor of the prestigious international journal Behaviour and Information Technology.

Before starting her consulting firm, She was Director of Human Factors at IDS (now American Express Financial Advisors), where, in 1988, she developed one of the first corporate usability labs outside the computer industry. Previously, at Honeywell, she was involved in evaluating usability of consumer product hardware and software, as well as military technologies. Dr. Dray received her doctorate in Psychology from UCLA and is a Board Certified Human Factors Professional.