Methods of Assessment
Focus groups
Focus groups are primarily used by market research companies as a method of eliciting people’s opinions about a certain topic by bringing a group of people together in a room to discuss the issue at hand. Focus groups have also been used in social sciences as a qualitative research method to complement traditional quantitative methods. And recently Lunt & Livingstone (1996) have made a case for using them specifically in media and communications research.
Focus groups ideally consist of 8-12 people of either similar or different demographics. The trend is towards selecting a focus group from people who already know one another or are from similar backgrounds as this will help spur the discussion. A moderator will be present to ensure that the group discussion does not go astray and that unclear points are clarified. The moderator, however, does not function as an interviewer.
An informal chat about something will often bring out valuable opinions and reveal conflict areas. According to Moscovici (cited in Lunt & Livingstone 1984) a focus group is a simulation of everyday conversation that is socially generated rather than a "readout" of individual minds. A focus groups will therefore produce the kind discussion that takes place around the coffee machine in a more controlled environment. Focus groups can be used in connection with requirements capturing and for assessing usability of computer systems. However, as users do not always do what they say they do (Nielsen, 1997), focus groups should always be used in conjunction with observational studies for evaluating usability.
In the following we report on two cases where focus groups have been used in connection with usability in multimedia communication. In the first one, focus groups were used to capture requirements for a new interface to rat. In the second one, focus groups were used to evaluate a series of ReLaTe field trials. And finally, we shall suggest how focus groups can be adapted to suit the hci community.
Rat Focus Group
In this case, focus groups were used in conjunction with traditional hci methods to capture requirements for a new rat interface. Two focus groups identified what the participants particularly liked and disliked about the current interface, which functions they used etc. The first group consisted of 7 current rat users whilst the designer functioned as moderator. Rat was kept running on a large screen throughout the focus group to provide inspiration for the discussion and as a memory aid. The focus group was recorded on both videotape and audio tape and lasted approximately an hour.
Nielsen (1997) suggests using email, Web sites, or Usenet newsgroups as a cheap way to approximate a focus groups. The second rat focus group went one step further by having the focus group as a multicast conference. The 9 rat users who took part in the second group, were all experts working on the Merci project. The designer, again, functioned as moderator, and the session was recorded using the Merci mmcr tool.
The session was announced on sdr and the tools used were rat, vic and wb (Wb was used for writing down comments). The meeting lasted approximately one and a half hours.
ReLaTe Focus Group
In the ReLaTe project, focus groups were used as part of the evaluation for a year’s field trials. The students and tutors involved in the project were brought together for a workshop, and as part of the workshop divided into two groups, a student group and a tutor group. Their experiences from using ReLaTe and what they would like changed in future versions were discussed. These focus groups were very small, with only two students and two tutors participating, apart from the members of the ReLaTe project who functioned as moderators.
Use of focus groups in multimedia communication
From our experiences in the rat and ReLaTe projects, we consider focus groups as a viable method of capturing user requirements as well as evaluating field trials. In particular, we would be interested in exploring the use of Mbone conferences for conducting focus groups for multimedia communication tools, as this would allow us to reach a much wider user population. However, the method would need to be refined somewhat for this purpose. First of all, the task of moderating an Mbone conference focus group is much harder than a traditional one. There are the usual problems with bad audio and no video from all participants, but more importantly, the fact that having a focus group where either the participants and/or the moderator are not confident using multimedia conferencing tools, makes it hard to moderate. Another issue that needs to be explored, is the use of shared workspaces. In the rat focus group, the wb was used mainly by an assistant appointed by the moderator for writing down points that came up in the discussion. After a while, however, other people started to write down their own points and small discussions were taking place on the wb concurrently with the spoken discussion. The use of a wb is potentially a valuable tool for Mbone focus groups if participants are encouraged to take notes during the discussion.