本书是"计算语言学与语言科技原文丛书"中的一册。文本和语音系统是计算语言学和语言技术的主要应用方向，涉及的范围很好广泛，几乎包括所有的语言技术应用系统。本书涵盖的内容很好丰富，既详细介绍了各种文本和语音系统及其评测的基础理论，又结合大量实例针对系统评测进行了明确的说明和解释。内容深入浅出，适合有一定背景知识的初学者或相关领域的科研工作者阅读，也可作为计算语言学专业教材。
戴布克耶Dybkjaer，丹麦南丹麦大学教授。
导读1
Preface 15
Contributing Authors 17
Introduction 21
Laila Dybkjar， Holmer Hemsen and Wolfgang Minker
1 SPEECH AND SPEAKER RECOGNITION EVALUATION 1
Sadaoki Furui
1.Introduction 1
2.Principles of Speech Recognition 2
3.Categories of Speech Recognition Tasks 3
4.Evaluation of Speech Recognition Systems 6
5.Principles of Speaker Recognition 12
6.Categories of Speaker Recognition Tasks 13
7.Normalization and Adaptation Techniques 15
8.Evaluation of Speaker Recognition Systems 17
9.Factors Affecting the Performance and Evaluation Paradigm
Design for Speech and Speaker Recognition Systems 22
10.Svstem-Level Evaluation of Speech and Speaker Rognition23
11.Conclusion 24
References 24
2 EVALUATION OF SPEECH SYNTHESIS 29
Nick Campbell
1.Introduction 29
2.Components of Computer Speech 30
3.Evaluation Methodologies 36
4.Organised Evaluations and Assessment 41
5.Speaking to (and on Behalf of) People 45
6.Conclusion 45
References 48
3 MODELLING AND EVALUATING VERBAL AND NON-VERBAL
COMMUNICATION IN TALKING ANIMATED INTERFACE AGENTS
Bjarn Granstram and David House
1.Introduction 65
2.KTH Parametric Multimodal Speech Synthesis 67
3.Data Collection and Data-Driven Visual Synthesis 69
4.Evaluating Intelligibility and Information Presentation 71
5.Evaluating Visual Cues for Prominence 77
6.Evaluating Prosody and Interaction 80
7.Evaluating Visual Cues to Sentence Mode 84
8.Evaluation of Agent Expressiveness and Attitude 85
9.Agent and System Evaluation Studies 87
10.Future Challenges in Modelling and Evaluation 91
References 92
4 EVALUATING PART-OF-SPEECH TAGGING AND PARSING 99
Patrick Paroubek
1.POS Tagging 99
2.Parsing 102
3.Evaluation and Natural Language Processing 105
4.POS Tagging Evaluation Methodology 110
5.Methodology and Evaluation Measures for Parsing 114
6.Conclusion 117
References 118
5 GENERAL PRINCIPLES OF USER-ORIENTED EVALUATION 125
Margaret King
1.A Historical Note 126
2.What is User-Oriented Evaluation？ 128
3.A First Principle： Quality is Decided by Users 129
4.A Second Principle： Users do not Have the Same Needs 130
5.A Third Principle： Quality can be Characterized 135
6.A Fourth Principle： Quality can be Measured 148
7.Combining the Particular and the General： The Ideal 153
8.Conclusion 154
References 156
6 AN OVERVIEW OF EVALUATION METHODS IN TREC AD HOC INFO-
RMATION RETRIEVAL AND TREC QUESTION ANSWERING163
Simone Teufel
1.Introduction 163
2.Evaluation Criteria 169
3.Evaluation Metrics 172
4.Real-World Performance 182
5.Conclusion 183
References 185
7 SPOKEN DIALOGUE SYSTEMS EVALUATION 187
Niels Ole Bernsen， Laila Dybkjar and Wolfgang Minker
1.Introduction 187
2.Evaluation Methods and Criteria 188
3.Evaluation of the NICE Hans Christian Andersen Prototype 193
4.Evaluation of the SENECA Prototype 207
5.Conclusion 216
References 218
8 LINGUISTIC RESOURCES， DEVELOPMENT，AND EVALUATION OF
TEXT AND SPEECH SYSTEMS221
Christopher Cieri
1.Introduction 222
2.The Linguistic Resource Landscape 222
3.Background on Linguistic Data and Annotation 234
4.Data Planning for Technology Development and Evaluation 237
5.Finding Resources 240
6.Building Resources 242
7.Conclusion 259
References 260
9 TOWARDS INTERNATIONAL STANDARDS FOR LANGUAGE
RESOURCES263
Nancy Ide and Laurent Romary
1.Introduction 263
2.Background 265
3.The Linguistic Annotation Framework 268
4.Putting it All Together 278
5.Conclusion 282
References 283
Index 285
Akers, G.and Lennig, M.(1985).Intonation in Text-to-Speech Synthesis: Evaluation of Algorithms.Journal of the Acoustical Society of America, 77(6):2157-2165.
Benoit, C.and Pols, L.C.W.(1992).On the Assessment of Synthetic Speech.In Bailly, G., Benoit, C., and Sawallis, T., editors, Talking Machines: Theories, Models and Designs, pages 435-441, Elsevier, North Holland, Amsterdam, The Netherlands.
Benoit, C., van Erp, A., Grice, M., Hazan, V., and Jekosch, U.(1989).Multilin- gual Synthesiser Assessment Using Semantically Unpredictable Sentences.In Proceedings of the European Conference on Speech Communication and Technology (Eurospeech), volume 2, pages 633-636, Paris, France.
Bernstein, J.(1982).Evaiuating Synthetic Speech.In Proceedings of the NBS Workshop on Standardization for Speech I/O Technology, pages 87-91, Gaithersburg, Maryland, USA.
Bernstein, J.and Pisoni, D.B.(1980).Unlimited Text-to-Speech System: Deion and Evaluation of a Microprocessor Based Device.In Proceedings of the Internarional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 576-579, Denver, Colorado, USA,
Bladon, A.(1990).Evaluating the Prosody of Text-to-Speech Synthesizers.In Proceedings of Speech Tech, pages 215-220, New York, USA.
Boeffard, O., Cherbonnel, B., Emerard, F., and White, S.(1993).Automatic Segmentation and Quality Evaluation of Speech Unit Inventories for Concatenation-Based, Multilingual PSOLA Text-to-Speech Systems.In Proceedings of the European Conference on Speech Communication an.d Technology (Eurospeech), volume 2, pages 1449-1452, Berlin, Germany.
Boogaar, T.and Silverman, K, (1992).Evaluating the Overall Comprehensibility of Speech Synthesizers.In Proceedings of the International Conference on Spoken Language Processing (ICSLP), pages 1207-1210, Banff, Canada.
