Abstract

Cost-justifying usability activities or measuring the business value of usability enables better project implementation decisions, resulting in solutions with the greatest ease of learning and ease of use. However, finding the optimal measurement technique and understanding the financial impact to the company's bottom line can be challenging.

Workshop description

Facilitator methodology presentation (9 a.m. to 10:45 a.m.)

Facilitators will discuss the workshops purpose, rationale, and present their experiences and paper. Together with the group the facilitators will then evaluate the methods used for those contexts.

Participant paper presentations and discussion (10:45 a.m. – Noon)

Selected papers will be discussed as a group. Participants will be asked to discuss their experiences and then the facilitators will open the topic for discussion and questions. In the event that there are more than eight participants, facilitators will prepare summaries of the presentations for discussion to enable more topics to be covered in the time allotted.

Small group discussion and sharing of results with the rest of the groups (1:30 p.m. – 4:30 p.m.)

Small groups will discuss and problem-solve selected topics in measuring the value of usability and share their results with the rest of the other groups.

Measuring the Value of Usability

Measuring the business value of usability solutions can take place in a variety of scales and contexts. There are three categories that can be used to describe the scale of usability measurement activities. First, usability measurement activities can involve measuring the value to an organization of a single usability activity, such as a usability test, and the impact that activity will have on the productivity of that organization, to justify the expense of doing the usability test. Second, measuring the value of usability solutions can involve measuring the impact of involving usability professionals through multiple phases of the product lifecycle and understanding the productivity benefits vs. the costs of employing usability professionals throughout the development of the project. These two types of measurement activities have often been used by consultants in an effort to justify to a client why the usability consultant should be hired into a project. They have also been used to justify to an organization why usability professionals should be employed and engaged in the development process. Usability involvement, especially in the early stages of the development process, can not only make incremental changes in the ease of learning and ease of use for the user, but can also result in restructuring how the user does their job in a way that greatly enhances ease of use and therefore the overall productivity of the employee. Measurement activities can also be utilized after the fact to clearly demonstrate the benefit of that restructuring, or of any other types of improvements.

A third type is one where traditionally there has been little or no usability involvement. This type involves measuring the business value of the entire application solution. This can be a software application or some combination of hardware and software. Decision makers in internal corporate development organizations are under increasing pressure to provide better justification for the solutions they provide. We, as usability professionals, can participate in these endeavors by making use of our unique skill set in measuring the value in terms of the ease of learning and ease of use of these solutions. These solutions, in some cases, can not only make evolutionary improvements to the users' ease of learning and ease of use, but can sometimes also result in a fundamental restructuring of how the users work, greatly improving the employee productivity. In addition, usability measurement activities can not only impact a specific project, but can also impact strategic decisions that a company makes. In these scenarios, usability professionals work alongside finance and other business analysts in an effort to measure the business value of the entire solution to the organization.

In this workshop we hope to be able to examine measurement activities from each of the three levels because many of the same problems apply to all three situations. In addition, there are a variety of different contexts where value can be measured, and different techniques will be needed depending on the context. Below are two examples of case studies that illustrate how different techniques can be used in different measurement contexts.

Case study 1: Measuring the value of wireless Pocket PCs in a factory setting

This study measured the benefit of wireless Pocket PCs in a factory setting using field study methods and task analyses.

Observational Methodology

• Semi-structured interviewing to obtain task-analysis and task-frequency information and to enable the development of usage scenarios
  
• Determination of task measurability for both baseline and pocket PC conditions
  
• Measurements of tasks
  

Measurement of tasks for baseline and post conditions

Multiple methods were used for this research because of the variance in the ability to measure tasks. The method chosen depended on the nature of the task and the feasibility of taking direct measures

• Method 1: Timed measurements of tasks being done or of simulation of tasks being done with the same individual for both the baseline and the Pocket PC conditions. This proved most feasible with tasks that were small in nature such as looking up contacts.
• Method 2: Time measurements were taken of baseline data and directly compared to timed measurements from people who had pocket PCs.
• Method 3: Timed measurements of similar or identical sub-tasks were re-used.
• Method 4: Where simulations or actual measurements were not feasible, self-reports of estimated time in the context of semi-structured interviewing were used, after a step-by-step understanding of the procedure was obtained.

Non-typical circumstances that would add extra time, such as a workstation not being available, requiring the user to go to another workstation, were not included in the data. In all of these cases it was stressed to participants that conservative estimates or underestimates should be given. Task frequencies were obtained by self-report. In some cases, the task could be generalized to other groups or to the same type of worker. In other cases, the estimates applied only to a limited group of workers.

The goal of this methodology was somewhat different from traditional statistical research. This methodology attempted to not overestimate the measured value rather than to precisely estimate the value. Because of this, additional adjustments were made downward in the estimated values. Plus, the small number of workers involved made using traditional statistical procedures unfeasible.

Example task: Voice over IP using the pocket PC vs. using a pager

• Six pages were received per shift (self-report data)
• Conservative reduction - use "four times per shift" as the estimate for number of pages received
• Factory worker paging other people inside the factory an estimated two times per shift (self-report) data

Scenario: Factory worker paging someone else (measured by simulation)

1. Walking to find a phone: 38 seconds - estimated by simulation inside factory. Pocket PC is carried on belt - time to find it is estimated to be 0 seconds.
2. 30 seconds to dial page vs. estimated 10 seconds on a pocket PC. 30-10 = 20 seconds dialing using voice over IP
3. Waiting for page to be returned was measured by 9 actual pages at 3:18. As a conservative reduction, time saved is 1:39 - 99 seconds. Note: Pages that were not returned were not included in analysis. Time to answer voice over IP estimated to be 5 seconds. 99-5 seconds - 94 seconds (38 sec saved walking to phone) + (20 sec saved dialing) + (94 seconds saved waiting for page to be returned) = 152 seconds saved for person paging another

(152 seconds saved) X (2 times per shift) X (14 times per month) X (12 months per year) = 51,072 seconds saved per person per year / 3,600 = 14.18 hours saved per factory worker, per year, paging others

Scenario: Factory worker answering page

1. 38 seconds to find a phone
2. 7 seconds to call person who paged you = 45 seconds vs. 5 seconds to answer Pocket PC phone with voice over IP - net savings 40 seconds.
3. (40 seconds saved) X (4 times per shift) X (14 shifts per months) X (12 months per year) = 26,880 seconds saved per person per year / 3,600 = 7.5 hours saved per year, per factory worker answering pages

Total estimated savings for all factory workers per year

(7.5 hours saved answering pages) + (14.18 hours saved paging others) = 21.68 hours saved per year per factory worker as a result of voice over IP

(21.68 hours saved per year in the factory) X (1000 factory workers) X ($35.00 burden rate) = $758,800 estimated benefit of voice over IP to factory workers per year

This savings is a time-savings or headcount productivity benefit. It is not clear how this benefit can be tied to the bottom line of the company. Sixteen other tasks were identified as headcount productivity savings benefits. Possible benefits to the company bottom line could be greater output capacity in times where demand is greater, reduction in headcount, and fewer errors.

Case study 2: Using field study techniques to measure the business value of a call center workflow change

The usability of an application is measured in part by the efficiency with which the user can perform tasks. Time to complete the task is one measure used to objectively evaluate the usability of a tool. Task completion time can be reduced, and thereby the usability of the tool enhanced, not only by making the several process steps more intuitive and efficient, but also by reducing the total number of steps. We have performed a study to measure the impact of reducing the number of steps in a business process workflow and how it reduces the task completion times.

A field study was conducted in a customer support call center with 5 users, involving 11 tasks. The users were customer support personnel who respond to user requests for creating, modifying, and deleting accounts. The users represented a broad spectrum of workers in terms of years of work at customer support. The objective of the study was to compare the time to complete the tasks in the current workflow with that in the revised workflow, and to evaluate the savings in task completion time and headcount. The users were given the tasks to complete using the current workflow and were asked to time themselves using a stopwatch. The users picked the customer request items randomly from a queue and completed the transactions.

The revised workflow seeks to achieve improved user performance by eliminating some of the manual steps. The time to complete the task in the revised workflow scenario was calculated by adding the time to complete only the remaining steps after the eliminated ones had been removed and by adding manual steps needed to complete the transactions whenever the automated transactions failed. In this case, the benefit was actually calculated before the proposed solution to eliminate the steps had been implemented. The following table gives the details about the semi-annual volume of transactions for each task, and the man-years needed to complete the transactions in the current and revised workflows.

On an average, it was found that the users completed the tasks 85% faster in the revised workflow scenario as compared with the current workflow. The man-year requirement was calculated as a benefit of the reduced task completion time. The total time requirement for each task was calculated by multiplying the task completion time with the average frequency of each task's annual volume and was summed up across all the tasks. As customer support is a highly structured and routine office environment, time savings at the level of tasks can be aggregated into daily time savings as the sum of time savings for all the tasks in a day. It was found that the man-year requirement with the new workflow would be 0.93 as compared to the current man-year of 6.21. The study demonstrates that there is a strong business case for upgrading to a revised workflow. The substantial reduction in the headcount requirement was used as a basis on which the project was approved.

Applying to Participate in This Workshop

A workshop is a closed session. Admission to a workshop requires an approved position paper from you addressing the issues suggested by the coordinator(s). Please send your position paper (which should be roughly 1 to 3 pages) to Steven Graves, steve.e.graves@intel.com. Position papers received by April 25 will be accepted or rejected by April 30, in time for you to register before the early registration deadline on May 2. Position papers received by May 24 will be accepted or rejected by May 29, in time for the May 31 registration discount. Papers received after May 24 will be evaluated at the facilitator's discretion.

Participant selection criteria

Participants are selected based on the quality of submitted position papers containing the following pieces of information:

We invite participants to submit a 1-3 page position paper describing their usability measurement experiences and how those measurements can be translated into greater productivity and business value. These experiences can be from any range of activities where measuring ease of learning and ease of use can be translated into greater productivity and into business value. These activities can include measuring the impact of a single usability activity, measuring the impact of multiple usability activities, and measuring the impact of an entire development project on the ease of learning and ease of use of a user. Participants are asked to describe their attempts to measure business value and the methods that were used. The measurement activities need not have been successful as long as lessons learned can be shared. Participants should describe aspects of their approach that were successful, as well as aspects of their approach that did not go well, and what they might have done differently in hindsight to make the approach more successful. Suggested topics to discuss are:

• Situation or contexts where measurements were to be taken
  
• Metrics and tools used
  
  • Advantages and disadvantages of using headcount productivity measures (time savings)
    
  • Reductions in task completion time
  • Reductions in error rate
  • Increase in successful completion task rate
  • Other metrics
• Comparing baseline and post intervention conditions
  
• Limitations of the situation in which they were trying to take measurements
  
• Obstacles that needed to be overcome to take measurements
  
• Method or methods used to measure business value
  
  • Lab study vs. field study
    
  • Performance vs. satisfaction measures
    
  • Direct and indirect measurements
    
• Why this method was used instead of other methods.
  
• After executing this method did it turn out that this method was optimal for this situation? Why or why not?
  
• What other approaches might have made the experience more successful?
  
• Valuing headcount productivity time savings
  
  • Understanding the impact of headcount productivity to the company’s bottom line (profit or revenue).
    
  • Materiality threshold - how large does a time savings have to be to be meaningful?
    
  • What do people do with the time they have saved?
  • Structured vs. non-structured task workers and how that difference affects your ability to value tasks
    
  • Impact on short term (tactical) vs. long term (long-term) goals and finances of a company
    
  • Impact of headcount productivity in a constrained situation vs. a non-constrained situation (supply and demand)
    
• Ability to tie measured metrics to the companies bottom line financially
  
• Follow up measurements to see if forecasted impact actually occurred and why it did or did not.
  
• Lessons learned
  
• Unsolved issues
  
• Any questions they have for other participants
  

Pre-workshop participant activities

Participants are asked to submit a position paper of 1-3 pages describing their measurement experiences or experiences and evaluate the methods used to achieve them. Participants should also be prepared with questions to facilitate discussion and the exchange of ideas.

Pre-workshop facilitator activities

Prior to the workshop, the facilitators will review each participant’s position paper for information about methodology, impact on the organization and the usage of a particular technique for a particular context. The facilitators will prepare a list of questions and discussion topics for selected papers. In addition, the facilitators will prepare a set of example scenarios and topics to be used in the small group discussions.

Presentation of results during the conference

Workshop results will be presented as a poster summarizing the key findings of the workshop with the list of participants and summaries of their experiences and key lessons learned.

Post-conference dissemination of results

An article will be written for the UPA Voice describing the results of the workshop and key lessons learned.

Post-conference activities

Post conference activities will be determined at the end of the workshop by consensus of the participants.

Facilitators

Steven Graves is a senior human factors engineer at Intel Corporation. He is currently working on several projects as part of Intel's "IT business value" program. Steve has worked in the field of human factors engineering since 1989 and has a Ph. D. in cognitive psychology from New Mexico State Univ.

Majorkumar Govindaraju is a senior human factors engineer at Intel Corporation. He is currently working on projects aimed at finding the business value of Information Technology products and solutions. Majorkumar has worked in the field Human Factors Engineering since 1993 and has a Ph. D. in Industrial Engineering from the University of Cincinnati.

David Sward manages a team that applies user-centered design to the development of products and services. He is engaged with several companies to develop processes that capture the business value of IT organizations. David has worked in the field of human factors engineering since 1991 and is an Adjunct Faculty member at ASU East.