Abstract
Usability practitioners who have justified or measured usability ROI distill their experiences to explore how we choose what factors to measure. During the workshop, we’ll extrapolate from peoples’ ROI successes (and failures) and design a model process—or at least general guidelines—for measuring and applying usability ROI.
Workshop Description (Facilitators’ Initial Position)
According to a recent Computerworld article, a survey of 1,200 businesses revealed that only 9% of U.S. companies required ROI analysis to justify IT investments in March 2001. By December 2001, that number had increased to 23%. The article predicted that by the end of 2002, 65% of companies will require ROI analysis because of pressures from top management, executive boards, and shareholders.
Thus it’s not surprising that “return on investment” is a phrase increasingly heard in the usability community. But turning “usability ROI” and “business case” into the latest buzzwords isn’t enough. We need processes for both measuring usability ROI and communicating it to management.
Measuring usability ROI is happening more often as practitioners are faced with more usability work than they can manage on current budgets, and as they compete with other groups within their organizations for resources. A few years ago, we could enroll senior management sponsors with anecdotes about the benefits of usability. Today, as Computerworld predicted, the holders of the purse strings want hard numbers which often don’t yet exist. As a result, many practitioners over the past year have attempted to measure and apply usability ROI, with diverse approaches and varying degrees of success.
The goal of this workshop is to analyze the participants’ experiences with usability ROI to create a model process and guidelines for practitioners. Building on the UPA 2002 panel on “Measuring ROI for Usability,” this workshop invites experienced usability practitioners and managers to share their experiences with usability ROI. During the workshop, we’ll discuss all the participants’ approaches to usability ROI and attempt to create from them categories of situations and general guidelines for measuring and using ROI in different organizational contexts.
The facilitators want this workshop to be more than just a series of case histories or war stories, however enlightening. The added value of this workshop will be to facilitate the interactive analysis by the participants that will “mine” our combined experiences and create worthwhile results for the practitioner community, during and after the conference. The central theme of our initial position is that we’re trying to extrapolate from people’s experiences to design a model process—or at least general guidelines—for measuring and applying usability ROI.
A key question for the workshop is: How should we choose factors or variables to consider or measure? For example, how do we quantify:
- Development cost reductions. Examples might be shortened time to release, or fewer releases needed, or more useful functionality in each release because of less time needed for bug-fixing.
- Support cost reductions. Examples might be fewer problems reported by users, shorter calls to tech support, fewer (or more) escalations of tech support calls, cost of fixing usability bugs in design versus in test versus in patches versus in releases.
- Training cost reductions. Examples might be fewer courses needed or shorter courses needed—or training might be a profit center and thus a controversial candidate for usability ROI.
- Increased sales due to usability. Examples might be feedback from new customers or sales rep productivity improvements because product is easier to sell (more benefits, more competitive).
- User productivity improvements. It’s easier to measure productivity improvements for internal systems than for commercial products. How can we translate improved customer productivity into numbers that affect profits (higher sales, lower costs)?
- Increased customer satisfaction. How can we identify what aspects or what percentage of customer satisfaction is due to usability, and how do we translate this into numbers that affect profits?
We plan to discuss and analyze when and why these factors—and others suggested by the participants—should be incorporated into usability ROI calculations. In addition to this key question, some of the issues we plan to explore during the workshop include:
- What kinds of usability projects or programs are most suited to cost-justification through ROI? Where should usability practitioners begin with ROI?
- Why did the participants choose the variables or factors they used in the ROI calculations?
- How did they address the issue of “soft” versus “hard” dollars?
- What would the participants do differently the next time they measure usability ROI? Why?
To help us agree on terminology, the workshop facilitators will begin with an overview of ROI calculations and cost-benefit analysis, which we’ll revise as appropriate during a short group discussion. Because the group will read the position papers before the workshop, we don’t plan to use valuable workshop time “presenting” each position paper. Rather, we’ll use a more interactive approach:
- We’ll post the position paper titles around
the walls of our meeting room, and ask participants
to write their questions under each title.
- Each participant will moderate a short discussion
of the questions about his/her paper.
- In a group interactive discussion, we’ll identify
commonalities between experiences and we’ll then
create a list of “burning issues” or questions
of general interest to measuring usability ROI that
arose during the individual paper discussions.
- We’ll discuss each of the burning issues, with
our goal to design a model process and guidelines
for usability ROI calculations.
- By the end of the day, we should describe a model
process that includes likely successful paths—based
on our participants’ experiences—and pitfalls
to avoid.
- If the participants agree, we may also define some
follow-on assignments that will help the workshop
facilitators with the post-conference dissemination
of results.
Applying to Participate in This Workshop
A workshop is a closed session. Admission to a workshop
requires an approved position paper from you addressing
the issues suggested by the coordinator(s). Please send
your position paper (which should be roughly 1 to 3
pages) to Stephanie Rosenbaum, stephanie@teced.com.
Position papers received by April 15 will be accepted
or rejected by April 30, in time for you to register
before the early registration deadline on May 2. Position
papers received by May 15 will be accepted or rejected
by May 29, in time for the May 31 registration discount.
Papers received after May 24 will be evaluated at the
facilitator's discretion. If you want to register early
for UPA and have not completed your position paper by
these deadlines, you may register for the rest of the
conference and add the workshop fee later.
Participant Selection Criteria
Participants will be selected based on the quality of submitted position papers. Each position paper should describe one or more case histories or experiences with justifying or measuring usability ROI. Some of the issues participants should address include:
- What led you to choose this project or projects for measuring ROI?
- What ROI calculations did you perform? What were the variables or factors?
- Why did you choose those variables?
- How did you choose a time frame for the return?
- How did you collaborate with stakeholders in making these choices?
- How did you apply the ROI calculations to obtain funding?
- What was most successful in your application of usability ROI? What worked less well?
- What would you do differently the next time you measure usability ROI? Why?
To help the facilitators evaluate your position paper, please also include biographical information about your usability experience and a description of the organizational context for your case history (that is, tell us about your company: size, age/maturity, industry segment, target customers, etc.).
Because our plans for post-conference dissemination of results include an article in User Experience, participants should be willing to contribute excerpts from their position papers for such an article. (If necessary, the case histories can be “anonymized” for the article.)
Pre-Workshop Participant Activities
Participants must submit position papers as described earlier under “Participant Selection Criteria.” Accepted participants will receive all the position papers. Before the workshop, participants are expected to read the position papers and formulate questions about them.
Pre-Workshop Facilitator Activities
The workshop facilitators will review and evaluate
the submitted position papers, and select up to 20 participants
who provide a variety and balance of experiences from
which we can create a model process and guidelines for
the UPA community. Based on the position papers, the
facilitators will begin identifying themes that will
focus our development of the model process and guidelines
during the workshop.
The facilitators will inform participants of their acceptance, and may request some or all of them to expand or revise their position papers. Then, we’ll send each participant copies of all the papers, and ask that participants read the papers and formulate questions about them before the workshop.
Presentation of Results During the Conference
The facilitators, assisted by participants who volunteer, will create a chart of the model process and list of the guidelines developed during the workshop. We will post these results in the conference Commons area.
Post-Conference Dissemination of Results
The facilitators will publish the results of the workshop in an article submitted to User Experience. This article will summarize the case histories in the position papers and will describe the model process and guidelines developed during the workshop.
Facilitators:
Stephanie Rosenbaum is president of Tec-Ed, Inc., a 15-person firm specializing in usability research and information design. Tec-Ed clients include Sun Microsystems, AOL/Netscape, Cisco Systems, and many smaller firms. Stephanie’s publications include a chapter in John Carroll’s volume on Minimalism Beyond the Nurnberg Funnel and a chapter on “Making Usability Research Usable” in Klaus Kaasgaard’s book on Software Design and Usability.
One of Tec-Ed’s usability specialists in Silicon Valley, Roberta Shroyer has 15 years of experience in usability research and information design for a wide variety of systems. Roberta holds an M.S. in technical communication from the University of Washington, where her studies included cognitive psychology and human computer interaction. Stephanie moderated the panel and Roberta co-authored the paper on “Measuring ROI for Usability” for the UPA 2002 Conference.
|
