|
Published Articles >> Table of Contents >> Abstract
Fourth IEEE International Conference on Multimodal Interfaces (ICMI'02)
p. 161
Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures
Sanshzar Kettebekov, Pennsylvania State University
Mohammed Yeasin, Pennsylvania State University
Rajeev Sharma, Pennsylvania State University
Full Article Text:
  
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ICMI.2002.1166986
Send link to a friend
| Abstract |
|
Although recognition of natural speech and gestures have been studied extensively, previous attempts of combining them in a unified framework to boost classification were mostly semantically motivated, e.g., keyword-gesture co-occurrence. Such formulations inherit the complexity of natural language processing. This paper presents a Bayesian formulation that uses a phenomenon of gesture and speech articulation for improving accuracy of automatic recognition of continuous coverbal gestures. The prosodic features from the speech signal were co-analyzed with the visual signal to learn the prior probability of co-occurrence of the prominent spoken segments with the particular kinematical phases of gestures. It was found that the above co-analysis helps in detecting and disambiguating small hand movements, which subsequently improves the rate of continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected from the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.
|
 Additional Information
|
Index Terms- Multimodal fusion, gesture recognition, gesture speech co-occurrence, prominence, prosody
Citation:
Sanshzar Kettebekov, Mohammed Yeasin, Rajeev Sharma,
"Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures,"
icmi,
p. 161,
Fourth IEEE International Conference on Multimodal Interfaces (ICMI'02),
2002
|
|
