The Evaluation of Two NLP Components in a Spoken Dialogue System

Gert Veldhuijzen van Zanten, Gosse Bouma, Gertjan van Noord, Remko Bonnema 
 
The NWO Priority Programme ``Language and Speech Technology'' is a
research programme aiming at the development of spoken language
information systems. Its immediate goal is to develop a demonstrator
of a public transport information system, which operates over ordinary
telephone lines.
 

In this Programme, two alternative natural language processing modules
are developed in parallel: a `grammar-based' (conventional,
rule-based) module and a `data-oriented' (statistical, probabilistic,
DOP) module.  Both of these modules fit into the system architecture
of OVIS.

 
In order to compare the NLP modules, a formal comparative evaluation
has been carried out. We describe the set-up of this evaluation, and
present the evaluation results. It turns out that the `grammar-based'
module obtains much better results for word accuracy and concept
accuracy, whereas less computational resources are required.