Published Articles >> Table of Contents >> Abstract

Fourth IEEE International Conference on Multimodal Interfaces (ICMI'02)   p. 241
Multi-Modal Translation System and Its Evaluation
Shigeo Morishima, Seikei University and ATR, Spoken Language Translation Research Laboratories
Satoshi Nakamura, ATR, Spoken Language Translation Research Laboratories.

Full Article Text:

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ICMI.2002.1167000
Send link to a friend

Abstract
Speech-to-speech translation has been studied to realize natural human communication beyond language barriers. Toward further multi-modal natural communication, visual information such as face and lip movements will be necessary. In this paper, we introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. We conduct subjective evaluation tests using the connected digit discrimination test using data with and without audio-visual lip-synchronization. The results confirm the significant quality of the proposed audio-visual translation system and the importance of lip-synchronization.
Back to Top Additional Information
Citation:  Shigeo Morishima, Satoshi Nakamura, "Multi-Modal Translation System and Its Evaluation," icmi, p. 241,  Fourth IEEE International Conference on Multimodal Interfaces (ICMI'02),  2002