Adapting Web 1.0 Evaluation Techniques for E-Government in Second Life

Alla Keselman, Victor Cid, Matthew Perry, Claude Steinberg, Fred B. Wood, and Elliot R. Siegel

Journal of Usability Studies, Volume 6, Issue 4, August 2011, pp. 204 - 225

Abstract

As Web technology evolves, information organizations strive to benefit from the latest developments. Many academic and government organizations develop applications in Second Life, an online virtual world that allows users to interact with one another and the virtual environment via graphical personas, to support education and information outreach. This is accompanied by growing interest in evaluation methods for Second Life applications. While no special methods have been developed for Second Life or other virtual worlds, the field of Web evaluation is mature and likely to offer metrics, methods, and tools that might be adaptable to Second Life. The goal of this project was to analyze how existing Web measures of Internet performance, Web usage, usability, and user feedback could be adapted, expanded, and modified for Second Life. The project employed two facilitated expert panel discussions, followed by an empirical pilot-test of the experts’ suggestions. The findings suggest that prevailing methods and metrics of four key evaluation dimensions can be adapted to Second Life. Specific recommendations are made for their adaptation. Challenges involve lack of universal Second Life design practices and user expectations, influence of other avatars on user experience, complexity of 3-D topography, high technical requirements for data collection, and the proprietary nature of Linden Lab’s data.

Practitioner’s Take Away

The following are the practical recommendations from this project:

To monitor performance of user applications in Second Life, create simple software tools for collecting availability and response-time statistics using scripts. At the present time, no companies offer these services, but the task is fairly easy for sufficiently motivated developers.
To obtain usage statistics about your Second Life applications, use commercial in-world analytics companies that build products and services on top of the virtual platform.
To conduct usability testing in Second Life, employ user testing and focus groups methods and follow the testing recommendations summarized in Figure 4 of this manuscript. Be sure to test placement and helpfulness of navigational aids and the effect of your 3-D space metaphor on the ease of finding information.
To ease the process of moderating of multi-player Second Life user testing sessions, assign separate moderators to teams or pairs of participants, create chat groups to reduce the number of communication channels for the moderator to attend, and employ pre-written chat macros for in-world interviewing.
To be able to obtain transcripts of typed communications during the session, make sure Second Life text chats are set to be recorded.
To handle video data of user testing sessions, convert Morae recordings to a format that significantly reduces the files’ size and import notation separately. Configure the Second Life settings not to use the internal Second Life browser for following Web links, because Morae cannot record it.
To obtain user feedback, employ in-world or linked-out surveys, focus groups, and polls by staffed avatars or drones.

Article Contents

Download Article

Adapting Web 1.0 Evaluation Techniques for E-Government in Second Life

Adapting Web 1.0 Evaluation Techniques for E-Government in Second Life

Abstract

Practitioner’s Take Away

Article Contents

Download Article

Usability Professionals' Association