Dec 2006 Contents
Thumbnail: Rolf Molich
By Cliff Anderson
Rolf Molich is “curious.” He cites that curiosity as the impetus behind his CUE, or Comparative Usability Evaluation, studies (www.DialogDesign.dk/cue.html). These now-world-famous studies look at how usability tests and expert reviews are actually carried out in practice – and about how reproducible they really are. There’s a lot more to Rolf, though, than his CUE studies.
Running into Jakob Nielsen also helped. The two worked on a project for the Danish government together. Out of that came a course, a book, and a lifelong friendship.
Others he ran into in those early years included Clayton Lewis, Jef Raskin, Ben Shneiderman, and Patricia Wright. He was an inveterate conference-goer, especially when “the IT field was so small you could have one conference and cover everything.” He also brought some of the speakers at these conferences back to Denmark.
After many years with a number of different companies, Rolf “dared to take the risk of becoming independent.” In 1997, he started Dialog Design. In addition to “wanting to work full-time with usability,” he also was intrigued by the free time and independence owning his own company might give him. And that – and his curiosity – led him to his first CUE study.
He shared his thoughts on a usability list serv, got four volunteers, then launched CUE-1. This study looked at a Windows application, having the participants test it in any way they saw fit to try and identify usability problems.
The results were presented at UPA 1998. Though “everyone in the crowd thought it was interesting,” there was also some resistance, something that Rolf would become used to over the years. He notes, “People said, ‘This just can’t be true. The results just can’t be this diverse.”
But they were. In CUE-2, Rolf found that 75% of the problems generated by nine teams running simultaneous, independent tests of Hotmail were unique, something very similar to what he had found in CUE-1.
The next study, CUE-3, took a different approach. Instead of doing tests, he had 12 teams do an expert review on a car reservation site (avis.com). Though this study “has not been published in any great detail,” it did serve as a springboard for CUE-4.
That study, the largest to date, was part of a workshop at CHI 2003. It divided up 17 teams, with half doing tests and half doing expert reviews. Rolf notes, “One of the things we wanted to find out was, ‘Is there a difference?’ Tests are more expensive, but do they give better results – more quality, more reliability, more reproducible results? We found that not to be the case.”
He continues, “It would be very difficult, if not impossible, to make a distinction based on what the teams came up with. There was one difference though. The teams that used an expert review were a bit more efficient. They invested fewer resources. Otherwise, there were few differences.”
CUE-5 was done at User Interface Engineering’s 2005 User Interface Conference. It pitted 13 teams against an Ikea tool that lets users customize their own furniture. This time, though, not all the teams were experts (who Rolf had relied on in previous studies). Nonetheless, except for “some who quite clearly didn’t have the time to do a thorough review, some of them did as well as the experts in CUE-4.”
The last study, CUE-6, was also done as part of a UIE conference, this past October. Thirteen teams took a look at another car reservation site (enterprise.com) before the conference. Rolf then discussed the results with the participating teams at a Master Workshop at the conference.
An Ounce of Prevention
What’s the solution? Interestingly, Rolf doesn’t see it as increasing the number of users, or other methodological changes. “The solution is a much more radical one,” he states firmly. “It’s prevention. Many of the usability problems we have seen should have not occurred on the website in the first place.”
But how to provide that prevention? Rolf recommends that “the development team know certain basic rules, like the heuristics that Jakob and I have been teaching for 30 years now.” Indeed, though many now associate those famous heuristics with Nielsen, they first appeared in print in a jointly written paper at CHI 90.
Rolf sees another possible problem with testing. Much too often, project teams do not actually implement the agreed-upon recommendations. Once again, Rolf was curious: “Why is it that much of what usability professionals say and can demonstrate tends to get ignored? During and immediately after the test sessions, everyone’s very enthusiastic, then they turn to other things, and the results tend to be forgotten. But it’s a very important message – usability problems don’t correct themselves.”
Rolf thinks he has an idea of where part of the problem may lie – with the usability reports themselves. “Sometimes I get shocked when I see some of the reports being produced from a professional usability test,” he confides. “They deal with usability, but the usability report itself is unusable, which I think is not a very good idea.”
Rolf recommends “making sure your presentations are extremely usable, so the project team doesn’t have that excuse.” Some things he recommends include prioritizing issues (he counsels 30 to 40), including a one-page executive summary (“I’ve seen executive summaries of 10 pages, which I think is ridiculous”), and making sure to include positive results (“to show that you’re not only critical, but you can also administer praise”).
He continues: “The things that usability professionals have to promote themselves on these days are that they have the knowledge. It’s really a specialist job. It’s something that requires a considerable time investment to keep abreast of developments in the field. It also requires special skills and you have to master these skills.” Being curious probably helps too.
|Contact the Voice|