At Eurospeech ’97 Elsnet provided the possibility for participants to test ten different spoken dialogue systems. After each test, participants were asked to complete a questionnaire (18 questions). This questionnaire was designed to assess the major properties of the spoken dialogue systems in a system-independent way. The total number of completed evaluation forms returned was 255, and for eight of the ten systems the number was large enough to permit meaningful statistical processing. A factor analyses was performed on the pooled data; the five main factors explained 75% of the total variance. Three of these five factors could be interpreted to reflect user opinions on (1) general appreciation, (2) functional capabilities of the system, (3) intelligibility of the output speech, while two were user-oriented: (4) proficiency in the system’s language, and (5) familiarity with spoken dialogue systems. Comparable sets of main factors were also obtained in separate factor analyses on the data for each system. This indicates that the first three main factors play a dominant role in the user's opinion of a spoken dialogue system, independently of the complexity or purpose of the system and independently of the user's knowledge of the system's language or the user's familiarity with this kind of information service. Some recommendations are given for the design of subjective evaluation of spoken dialogue systems by means of questionnaires.

As the number of spoken dialogue systems is growing, the need to evaluate these systems becomes important. Since spoken dialogue systems are complex and include a combination of speech recognition, natural language processing, dialogue management, and speech synthesis, assessment of the complete system is not a simple task. Objective evaluation may address the length of the dialogue, the number of turns, the number of diagnostic turns by the system, the number of speech recognition errors, the number of successful database queries, and so on. Besides this objective evaluation, subjective assessment can be used to address the perceived quality of the dialogue system (Walker et al., 1997). The only way to investigate the perceived quality of a system is by asking users questions about it. These questions may be related to the overall acceptability of the system or may have a more diagnostic purpose in order to address specific features of a system. In the first case it is more appropriate to ask open questions like "How did you get along?" and "What do you think of this system?" In the second case it is more appropriate to use Likert scales (e.g. "The system did understand me well": I fully agree [1]-[2]-[3]-[4]-[5] I fully disagree). In this paper we concentrate on the latter type of diagnostic evaluation.

At the ELSNET Olympics, Eurospeech '97 participants had the unique opportunity to test ten different dialogue systems. All systems were evaluated using the same questionnaire. For naïve subjects it will probably be difficult to answer diagnostic questions. However, the Eurospeech population may be considered capable of answering these types of questions about spoken dialogue systems. In this paper we investigate whether different spoken dialogue systems can be assessed by a single questionnaire. This is only possible if the questionnaire (and the individual questions) address relevant features that are present in all systems. If that proves to be the case, we may hope to be able to compare subjective evaluations of dialogue systems for different tasks. Even more importantly: we would then be in a position to transfer knowledge and experience gained with one system to speed up the development of other systems.

The ten systems that participated in the Elsnet Olympics differed in task, lexicon size, type of speech recognition, dialogue design, availability of barge-in, type of system speech and language. Our hypothesis was that irrespective of these differences it is still possible to assess systems subjectively by putting the focus on whether systems do what they claim to do from the perspective of the user. Therefore, the questions were formulated in a general way (see table 1 for the questionnaire). Most questions were of the closed type using a Likert scale, few other ones were of the open type in order to be able to interpret the meaning of the answers to the closed questions more clearly. Next to the questions related to the spoken dialogue systems, two questions were included addressing the users’ proficiency in the system’s language and the familiarity of the user with spoken dialogue systems, to see whether these factors influence the diagnostic evaluation.

Two of the eight systems could not be tested properly. The Aizula system (Japan) (Ward, 1996) was not an information system but a demonstration of simulated back-channel feedback. The Actis system (KDD, Japan) provides information about area codes, country codes and time differences between Japan and the destination of an international call. The lexicon covers the names of 300 countries and a thousand cities all over the world. Due to line problems the system was only tested a few times.

Table 1: Text of the questionnaire

The remaining eight information systems were:

Jupiter (MIT, Cambridge MA, USA) (Zue et al., 1997): The version of Jupiter used at the ELSNET Olympics was a US English conversational system that gives current weather forecasts over the telephone for over 500 cities worldwide. The system can answer queries about general weather forecasts, temperature, humidity, wind speed, sunrise/sunset times, and weather alerts (such as flooding and hurricanes). It obtains its weather information from Web sources. Jupiter has a vocabulary of 1400 words, and will soon be able to handle calls in German, Mandarin, and Spanish. It uses text-to-speech synthesis for output speech.

STACC (University of Granada, Spain) (Rubio et al., 1997): This system-driven service allows students to consult their marks. Students have to enter one of two degrees and one of six courses they want to consult, and say their full name as well as an eight-digit identification number. When the name and identification number match, the system provides the required mark. The lexicon contains about 300 words.

O-tel (University of Maribor, Slovenia). This automatic reverse directory service can handle Slovenian, English, and German. The user enters digits in one of these languages; the digits must be separated by short pauses, since isolated-word recognition is used. The system repeats each digit, and when a digit is not recognized correctly, the caller can erase it. The most distinguishing feature of this system is that it has talk-through capability. The output is given by word-based synthesis.

EVAR (University of Erlangen, Germany) (Boros et al., 1997): Developed within the Sundial project, this system provides information on German Intercity timetables. It is run mainly for research purposes. Research emphasis is on the (relatively free) dialogue, on robust recognition, parsing of spontaneous speech and detection of out-of-vocabulary words. The dialogue manager can cope with anaphora and ellipses, and has a variety of recovery strategies for unusual situations. The user can always go back and change information. The system has a vocabulary of 1600 words, as well as a spelling mode for

when standard dialogue strategies fail.

Dialogos/Italian Arise (CSELT, Torino, Italy) (Albesano et al., 1997): Like the three systems below, this continuous-speech dialogue system forms part of the Arise project, a European research project partly funded by the EC under the LE sector of the Fourth Framework Telematics Application Programme. It has a vocabulary of 3,500 words, including 3,000 Italian station names. The dialogue module interprets the content of the user's utterances by taking into account both previous utterances and data pertaining to the application. The system can support different clarification and correction subdialogues, and is able to detect repairs initiated by the user. It uses text-to-speech synthesis for output speech.

LIMSI Arise (Orsay, France) (Lamel et al., 1996): This system provides information on train schedules, fares, reductions, and services in French. It uses continuous speech recognition with task-dependent acoustic models. The lexicon contains about 1500 words, 680 of which are station names. It is possible to interrupt system prompts (barge-in); speech output is handled by synthesis, through concatenation of about 2000 prerecorded units. The system uses a very open mixed-initiative dialogue: the caller is free to ask any question at any time. The system asks the user to provide the information it needs for database access, but can deal with the caller providing different information.

IRIT Arise (Toulouse, France) : This system uses speech recognition and dialogue management technology developed by Philips. The lexicon contains 1500 words (500 of which are station names). It is a conversational system, with concatenation of prerecorded speech being used for the system's output.

Dutch Arise (NS/OVR, Philips, KPN Research, University of Nijmegen, the Netherlands): This conversational system has a lexicon of 1380 words (680 station names). Its speech recognition component uses context-dependent acoustic models (triphones). The system has been trained by more than 11,000 dialogues, and uses concatenation of prerecorded units for speech output.

We analysed the data from the Likert scales by means of a factor analysis which was followed by varimax rotation for more easily interpretable links between questions and factors. Since only completed questionnaires could be used for factor analysis we had to discard quite a few questionnaires (see table 2). Some users did not encounter errors and consequently could not answer error-related questions, others failed to communicate meaningfully with a system and did not get any impression of details of the system's capacities, while some users simply forgot to turn the page of the questionnaire. There was no specific relation between a system and the type of missing questions, with the exception of the multi-lingual O-tel system for which some users failed to pass the language choice.

Table 2. Total number of questionnaires returned for each system and the total number of fully completed questionnaires used for factor analysis.

Table 3 shows the results of the factor analysis for the pooled data. Five main factors can be distinguished, which together explain 75% of the total variance. The first three factors are related to system properties, the last two are user-oriented. The first factor refers to the general appreciation of a system and addresses how a user perceives the interaction with the system. Questions about overall satisfaction, task completion, error recovery and error messages have their main loading on this factor.

Table 3. Loadings of all closed questions on the first five factors after factor analysis (followed by varimax rotation). Boldface typed numbers show loadings that exceed 0.70. Factors 4 and 5 correspond to a single question.

The second factor refers to the functional capability of the system, i.e. what is the purpose of the system? It can be noticed that most questions that are mentioned under the first two factors have their major loadings on both factors 1 and 2, which implies that most of these questions measure some combination of general appreciation and the functional capabilities of the system. The third factor deals with properties of the system prompts and output speech. These three factors are independent dimensions in the user’s assessment of a system.

The fourth and fifth factors (language proficiency of the user and the familiarity with spoken dialogue systems) are user-related. Interestingly, there is no interaction between these user-oriented factors and the factors dealing with the assessment of system properties.

A proper factor analysis presupposes a single subject group evaluating all systems. Obviously, this could not be the case because of the different languages of the systems. We therefore also performed factor analyses on data for each of the systems separately. If the same factor pattern as for the pooled data shows up for the individual systems, this would support the general validity of the overall analysis. Table 4 summarizes the results of the separate analyses. For most questions the relation to the main factors was comparable for all systems. A slight complication was that the question on wording of the system utterances now got a separate factor for four systems, which in two cases was also related to the question on error messages. The question on dialogue structure loaded on factor 1 or factor 2, while the question on dialogue length was linked to various factors for the different systems. The scatter in the loadings of the latter questions over the factors in the pooled data obviously originates from system-dependent usage of these questions.

Table 4. Factor associations per question after separate factor analysis (and varimax rotation) per system. A question was associated with a factor when its loading exceeded .70 (otherwise - is inserted). The factor numbering follows Table 3, but higher numbers were introduced when for some system the factor pattern did not match the general pattern. The right column shows the most common factor per question.