Participant Knowledge and Experience Expected

Participants are not required to have any particular knowledge other than a general knowledge of a user-centered design process.

Goals for the Session

Attendees will learn:

• Pros and cons of the traditional semantic differential method.
• Applications of the traditional semantic differential method to product/interface design
• Techniques for addressing the limitations of the traditional methods in their own business environment by bounding the scales and understanding the valence by using ideal and stereotypical ratings.
• A clear, data-driven method for designing products that stand out from other products and have a greater appeal to consumers.
• Techniques for evaluating the emotional impact of products.

The information taught will be immediately applicable in the participant’s business environment. Participants will leave the session with all of the materials, examples, and explanation needed to make use of this technique.

Previous Publication or Use of This Material

This material has not been published or presented at any conferences. The topic of semantic differential analysis is not new. The presenters have collected several techniques for improving the usefulness of semantic differential analysis for design (product and interface) and have used their own experiences to create a repeatable approach that can have a strong impact on design.

Presenter Background in this Material

The presenters, Ross Teague and Art Swanson have a combined 15 years of product research and design experience and work for HumanCentric Technologies, a consulting company that supports many different industries and product design efforts. The presenters are involved in all aspects of product design from requirements gathering (user, business, and technology) through to the build and testing phase. This work has brought to the forefront the need for understanding user perceptions and product aesthetics and how to feed those perceptions back into the design process. The techniques to be taught in the tutorial are a result of three years of investigation and their experiences with the deficiencies of traditional product evaluation tools and the difficulty applying the results in an iterative design process.

Ross C. Teague, Ph.D. – Dr. Teague is passionate about understanding people and their use of products and services. As a Senior Human Factors Psychologist and Manager of Design Research at HumanCentric Technologies, Dr. Teague gets as close as possible to end-user’s likes, dislikes, and limitations by going beyond product functionality to create differentiated products. He works to integrate user specifications with business and technology requirements to create a product that exceeds user expectations.

At HumanCentric, Dr. Teague is focused on supporting specialized domains such as medical products, consumer products, and communication devices. He has applied his explorative research methods to maximize design development capabilities and product ease of use for companies such as Johnson & Johnson, SunTech Medical, Source Technologies, Sparton Medical, Amerock, and Cisco Systems. His tool kit of methods for understanding what people need and their reactions to products has been derived from a variety of sources including clinical psychology, advertising, market research, ethnography, and sociology.

Understanding and applying design research is Dr. Teague’s forte. He is fluent in all aspects of contextual inquiries, field observations, focus groups, and requirements gathering. Dr. Teague often invents methodologies to collect information required to facilitate innovation.

Before joining HumanCentric Technologies, Dr. Teague spent 6 years with Intel Corp. helping companies' design and build world-class consumer products in the areas of children’s technology, sports, and entertainment. At Intel, his roles included project management and design research consulting to create innovative Internet and software products.

Dr. Teague received his B.A. in Psychology from North Carolina State University and earned both a Ph.D. and M.A. in Applied Cognition and Human Factors from George Mason University.

Art Swanson – Mr. Swanson is a Human Factors Engineer for HumanCentric Technologies specializing in human-based innovation. He utilizes anthropometric, biomechanical, and human factors data as creative seeds to drive innovative product design strategies. Tempering his engineering background with a creative streak, he enjoys using data to feed understanding rather than benchmarks. Mr. Swanson believes that good design comes from fully understanding people and he strives to provide that understanding to companies.

At HumanCentric, Mr. Swanson has focused on several different domains during his six-year tenure. He has extensive experience in automotive interior design, consumer products, and medical device design. Mr. Swanson has evaluated and improved products from Johnson Controls (automotive), General Motors, BE Aerospace, Thomson Consumer Electronics, IBM, Apple, Synaptics, Motorola, Ohmeda Medical, Johnson & Johnson, Sparton Medical, and Abbott Laboratories. His approach to design research draws not only from Human Factors and Cognitive Psychology literature, but also from such diverse arenas as Physics, Biology, Computer Science, Business, and Engineering.

Mr. Swanson received his BS in Mechanical Engineering from North Carolina State University.

How the Tutorial will be Conducted (summary)

The session will begin with all of the participants filling out semantic differential rating scales focused on their view of a “stereotypical” UPA (or other conference) tutorial and another set of semantic differential rating scales focused on their “ideal” conference tutorial. This will be done to familiarize all participants with semantic differential scales and the data collected will be used during the tutorial to show how the results can be analyzed. At the end of the session participants will fill out another semantic differential questionnaire evaluating the tutorial they just attended. We will post the compilation of the final questionnaire and our discussion of the results at the conference, how they compare to the “stereotypical” and “ideal” questionnaires, and how the data can be used to improve the tutorial.

After the initial questionnaires are filled out, the presenters will introduce the topic and lead a discussion with the attendees related to their own use of semantic differential scales (domains, scale anchors, purposes). Following this, the presenters will talk in depth about semantic differential limitations and weaknesses and ways that the data collected from semantic differential scales can be better applied to design. Participants will rate and discuss several products during the session.

The session will conclude with a working discussion time where attendees walk through the entire process of creating, analyzing, and understanding the results of a product evaluation and using the results to create design recommendations for improving the product. The presenters will have a downloadable program available for participants available after the session is complete that provides them the scales, factor loadings, and calculations needed to conduct their own semantic differential evaluations with the improved analysis techniques.

Tutorial Schedule with Estimated Times

* Approximately 1.5 hours of tutorial involves attendee participation

Detailed Description of Tutorial

1. Initial rating scale completion

When the session begins, once we have introduced ourselves and the agenda for the next 3 hours, we will describe to participants what we would like for them to evaluate first. That will be a UPA (or other conference) tutorial session. We will ask them to first consider and rate the “stereotypical” tutorial followed by consideration and rating of their “ideal” conference tutorial, using the semantic differential scales that we provide.

Once this is complete, we will collect the ratings and move into our introduction of the topic. While we are continuing the talk, we would like to have a conference volunteer (if one is assigned to our session) enter the data into an analysis program so that we can refer back to the data at the end of the session.

2. Introduction of the topic

We will discuss bridging the gap between usability evaluation and design and how it is one of the most difficult problems in product development. It is one thing to collect data on the evaluation or usability of a product, but it is something completely different to turn those data into actionable points for product designers. This is especially apparent when trying to understand the emotional or aesthetic impact of a design. The semantic differential technique has been used frequently by designers, marketing researchers, and usability professionals to evaluate products. The usefulness and transfer of this data, however, has traditionally been lacking. Traditional approaches to the semantic differential method have been lacking in two key ways:

• Lack of standardization (and factorization) of the scales so that the results lack specific meaning and direction for the designers. For example, if you were designing a product that marketing wanted to be described as "high tech" which was actually rated as "low tech" by users, what actionable information does that result hold for a designer? At a minimum it is not enough information, and at worst it can be misinterpreted. And what if it was rated as a 3 on the 7 point scale (1 - low tech, 7 - high tech)? Does that rating justify spending $2.00 more to upgrade the screen quality to affect a more "high tech" look?"
• Another problem with the traditional semantic differential method is the true valance of the scales is rarely captured or understood. When applying traditional semantic scales to a product, the designer has to infer a correct “direction” for the scale (i.e. should the product be more “hard” or more “soft”). This direction often comes from marketing information, user personas, or similar successful products, all of which are indirect measures of preference, which may or may not be correct.

We will talk about shortcomings in the application of semantic differential scales to improving and designing products.

3. Group semantic differential discussion

To understand the different needs people have for using semantic differential rating scales and to discover the breadth of scales used, we will solicit from the attendees the scales that they use, why they are using the ones they do, and discuss the differences between them. This will start our conversation on how many of the semantic differential scales are used to evaluate products and services from a marketing or branding perspective and how to incorporate those types of scales in our analyses.

4. Detailed presentation of materials

The semantic differential technique has been validated for a number of different applications ranging from the evaluation of political concepts to capturing the difference between user perceptions of sound and music. The semantic differential technique receives much of its generality from the fact that it is truly capturing the “meaning” of the stimuli to the user. This “meaning” has been found to lie somewhere within a three-dimensional space called semantic space. The three dimensions (as determined by factor analysis) are Evaluation (E), Potency (P), and Activity (A) (Osgood, C.E., Suci, G.J., & Tannenbaum, P.H., 1957). These three dimensions capture the majority (but not all) of the “meaning” in our internal semantic space, with the Evaluation dimension being the most dominant of the three. The EPA semantic space has been found to be persistent in a number of studies over the last 50 years in wide ranging areas of application, including aesthetic evaluations (Tucker, W.T., 1955).

The idea of semantic space, and the relationships that can be observed by understanding the positioning of products in semantic space, is rarely leveraged in design. For a true picture of semantic space, however, we need several pieces of information. First, we need to understand the relative weight of each of the factors for each scale. For example, if we use the scale good – bad, it is pretty apparent that that scale will weight almost exclusively on the evaluative dimension, but what about the scale high tech – low tech? Well, the answer to high tech – low tech is that we do not know what the weighting is without running a separate factor analysis on that scale and extracting the factor loadings. There are many scales that have already been factor analyzed, however, and we can use these scales where we understand the relative weightings on each of the three primary factors to allow us to plot the position of our product in semantic space.

Second, we need to have some reference points in semantic space for comparison. These reference points can be competing products (to understand how your design compares with your competition) or they can be the ratings for an “ideal” product and a “stereotype” product. When we add these reference points to semantic space, we can calculate the distance between these points, which will give us a wealth of information. For example, the distance between the “ideal” product and the “stereotypical” product can be defined as the design opportunity. If there is a large distance between those two points, it means that users are unfulfilled, and they expect more than they are getting from their products. However, if that distance is small, it means that the product is a commodity, and that there is little room for incremental design improvement (but that does not mean it cannot be improved. We’ll discuss this more on that later). The distance between your product and the stereotype is the “novelty” of your product, and the distance between your product and the “ideal” is the perceived room for improvement of your design.

Understanding your product relative to the “ideal” product and your competition is certainly valuable, but the semantic space also provides some design direction for improvement. For example, if your product is rated lower on the potency scale than the “ideal” product, there might be some benefit in changing your materials to be harder (one way to effect the hard – soft scale, and consequently, the potency dimension).

As you can imagine, understanding the semantic space is only part of the solution. We would also like to understand the emotional impact of the product on the user. Emotion in design has been a hot topic over the last couple of years, and everyone agrees that emotional design is key to building a great user experience. But capturing the emotional response to products has been limited to ad-hoc methods of ratings scales such as: “how happy does this product make you feel?”, etc. Fortunately, the semantic differential method has also been applied to understanding the emotional impact of products. If we slightly alter the three primary dimensions of the semantic differential from Evaluative, Potency, and Activity to include emotional aspects, it has been shown that three emotional dimensions can be generated: Pleasure (the emotional counterpart to Evaluative), Arousal (Activity), and Dominance (Potency) (Mehrabian, A., 1995b). Using semantic scales that are designed to capture the emotional impact of design allows us to also plot our product in emotional space. The dimensions in emotional space have been quantized into different emotional states which allows us to infer a specific emotional state based on the three dimensional location in emotional space. For example, the emotional state “bored” has been calculated to have coordinates of (-0.65, -0.62, -0.33) on a normalized scale of -1 to +1 (Mehrabian, A., 1995b). These dimensions would indicate that “bored” indicated fairly displeasureable (- pleasure), fairly un-arousing (- arousal), and slightly submissive (- dominance) reactions to the stimuli. Therefore the current design can also be understood for its emotional impact on the user. Note that the emotional ratings are a measure of the user after interacting with the product, not the product itself. Therefore the utility of the “ideal” and “stereotype” ratings is greatly reduced. These ratings will likely be consistent across any range of products as they are more representative of the “ideal” and “stereotypical” emotional state of the user, independent of the product.

The search for a way to quantify aesthetics has been a constant struggle between the arts and the sciences for decades (if not millennia). While the engineers and scientists want a numerical measure of the aesthetics of a product (using things like symmetry, balance, density, etc.), the artists (including designers) have argued that true art is not quantifiable, because it affects the user in so many different ways. This technique serves as a bridge between those two camps by offering some quantification of the user perception of aesthetics (how well the artist’s message is received), while still leaving the process of design and redesign in the hands of the artist. The output from the semantic differential exercise is likely to be on the order of: “the design needs to be more masculine, harder, and wetter”. That data is then put in the hands of the designer to manifest the needed changes, after which the design is reevaluated. Therefore, semantic differential does not take the artistic process out of design, but merely provides the designer with two key pieces of information:

1. How does the user perceive your design? The designer’s intent may have been that the design seems aggressive and powerful to appeal to male customers, but the user’s perception may actually be that the design seems fragile.
2. What aesthetic direction would the user prefer? Designer’s often have to rely on marketing data, personas, and idea boards to infer what the users in a particular segment are searching for aesthetically, but the semantic differential method provides direct information on the user requirements for aesthetics.

The methods to be described during this workshop will give the usability professional more and better tools with which to evaluate products from a design perspective. These techniques will also serve as a bridge between the usability evaluation and the design to allow evaluators and designers to understand the impact of aesthetics on the overall design, as well as the emotional impact of a design on the user.

5. Break (~:20 minutes)

6. Working session

To this point, we will have spent our time talking about the use of Semantic Differential scales and techniques for improving their use in the evaluative and design environment. At this point and for the rest of the session, we are going to show by hands on example, how to take the data from semantic differential scales, apply pre-determined factor loadings to the data, chart the data, interpret the findings, and make design alterations to improve the product.

We will break the attendees into small groups. Assuming 30-35 people in attendance, we will create smaller groups of 5-6 people each. We will show a product to the entire group have them evaluate it, and tell them that the product has already been evaluated by a larger group of people and we have the data for them to use.

Once this is completed, we will give each group a summary of the compiled data (product rating, “ideal” rating, and “stereotype” rating. Each group will get slightly different data, though they will not be told of this yet. The differences will relate to the strength of the dimensions and how each product rating compares to the “stereotypical” and “ideal” ratings. We are making the data different to give attendees an understanding of the various stories the data can tell and ways it impacts design and recommendations.

The data that attendees are given will have the factors loadings associated with each scale included, except for a few. Attendees will have already been told about applying factor loadings to the data and in this portion of the working session will be given factor loading tables for the different dimensions and asked to fill in the blanks using the table. This shouldn’t take more than a minute or two but will give everyone familiarity with the actual procedure for doing this activity. The small groups will work together on this.

We are going to create a downloadable program for each participant that has all of the information and loadings they need to be able to conduct the evaluations when they are back at their job. We are going to build a program that can be used by attendees on their jobs to enter data and see the results. We will talk about this program in this part of the session and show how it can be used (and more importantly, what it is telling you about the data). We will then provide each team with the output they would receive from entering their data into the program and have them begin their interpretation of the information.

The groups will be told at this point that their data was slightly different from each other and that their interpretations will also be slightly different. Though this topic will be discussed earlier, the presenters will describe how to interpret the findings and determine what dimensions they can change to improve the product’s overall rating and make it closer to the ideal.

Using the interpretations of the data for the product, stereotype, and ideal ratings, the groups will start making design change recommendations (some specific, others general) and once complete, present the information to the larger group. This will be a valuable part of the session because all groups will be able to see how small changes in the ratings can result in vast differences in the recommendations made to improve the design. Each group will discuss the changes that they would recommend. Some groups will provide only recommendations based on the data, while others will provide specific design changes based on their skills in design. The presenters will share their own ideas and ideas from our design teams how to make changes to the product to improve the design and more approach the “ideal” product that people want to have.

7. Final session evaluation

At the beginning, we had attendees rate their stereotypical and ideal conference tutorials. Now that they have completed this tutorial, we will ask them to rate this tutorial on the same semantic differential scales as used at the start. We will not have time to enter the data, but we will post it at the conference (location TBD) and interpret the data and discuss how the data suggests that we could improve the tutorial. We will also make the findings available to the UPA Conference Committee as a potential analysis tool for other sessions in the future.

References

Mehrabian, A. (1995b). Framework for a comprehensive description and measurement of emotional states. Genetic, Social, and General Psychology Monographs, 121, 339-361.

Osgood, C.E., Suci, G.J., & Tannenbaum, P.H. (1957). The measurement of meaning. Urbana, IL: University of Illinois Press.

Tucker, W.T. (1955). Experiments in aesthetics communications. Unpublished doctor’s dissertation, University of Illinois, 68-70, 291-295.