http://www.computer.org/tac The IEEE Transactions on Affective Computing is intended to be a cross disciplinary and international archive journal aimed at disseminating results of research on the design of systems that can recognize, interpret, and simulate human emotions and related affective phenomena. The journal will publish original research on the principles and theories explaining why and how affective factors condition interaction between humans and technology, on how affective sensing and simulation techniques can inform our understanding of human affective processes, and on the design, implementation and evaluation of systems that carefully consider affect among the factors that influence their usability. Surveys of existing work will be considered for publication when they propose a new viewpoint on the history and the perspective on this domain. en-us Wed, 4 Jan 2012 11:00:01 GMT http://csdl.computer.org/common/images/logos/tac.gif List of recently published journal articles http://www.computer.org/tac http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.40 Human emotional expression tends to evolve in a structured manner in the sense that certain emotional evolution patterns, i.e., anger to anger, are more probable than others, e.g., anger to happiness. Furthermore the perception of an emotional display can be affected by recent emotional displays. Therefore, the emotional content of past and future observations could offer relevant temporal context when classifying the emotional content of an observation. In this work, we focus on audio-visual recognition of the emotional content of improvised emotional interactions at the utterance level. We examine context-sensitive schemes for emotion recognition within a multimodal, hierarchical approach: bidirectional Long Short-Term Memory (BLSTM) neural networks, hierarchical Hidden Markov Model classifiers (HMMs) and hybrid HMM/BLSTM classifiers are considered for modeling emotion evolution within an utterance and between utterances over the course of a dialog. Overall, our experimental results indicate that incorporating long-term temporal context is beneficial for emotion recognition systems that encounter a variety of emotional manifestations. Context-sensitive approaches outperform those without context for classification tasks such as discrimination between valence levels or between clusters in the valence-activation space. The analysis of emotional transitions in our database sheds light into the flow of affective expressions revealing potentially useful patterns. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.40 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.41 We study the affective states exhibited by students using an intelligent tutoring system for Scatterplots with and without an interactive software agent, Scooter the Tutor. Scooter the Tutor had been previously shown to lead to improved learning outcomes as compared to the same tutoring system without Scooter. We found that affective states and transitions between affective states were very similar among students in both conditions. With the exception of the "neutral state", no affective state occurred significantly more in one condition over the other. Boredom, confusion, and engaged concentration persisted in both conditions, representing both "irtuous cycles" and "vicious cycles" that did not appear to differ by condition. These findings imply that -- although Scooter is well-liked by students, and improves student learning outcomes relative to the original tutor -- Scooter does not have a large effect on students -- affective states or their dynamics. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.41 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.39 The widespread availability of digitized music collections and mobile music players have enabled us to listen to music during many of our daily activities, such as physical exercise, commuting, relaxation, and many people enjoy this. A practical problem that comes along with the wish to listen to music is that of music retrieval, the selection of desired music from a music collection. In this paper we propose a new approach to facilitate music retrieval. Modern smart phones are commonly used as music players, and are already equipped with inertial sensors that are suitable for obtaining motion information. In the proposed approach, emotion is derived automatically from arm gestures, and is used to query a music collection. We derive predictive models for valence and arousal from empirical data, gathered in an experimental setup where inertial data recorded from arm movements is coupled to musical emotion. Part of the experiment is a preliminary study confirming that human subjects are generally capable of recognizing affect from arm gestures. Model validation in the main study confirmed the predictive capabilities of the models. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.39 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.38 Convincing conversational agents require a coherent set of behavioural responses that can be interpreted by a human observer as indicative of a personality. This paper discusses the continued development and subsequent evaluation of virtual agents based on sound psychological principles. We use Eysenck's theoretical basis to explain aspects of the characterization of our agents, and we describe an architecture where personality affects the agent's global behaviour quality as well as their backchannel productions. Drawing on psychological research, we evaluate perception of our agents' personalities and credibility by human viewers (N=187). Our results suggest that we succeeded in validating theoretically grounded indicators of personality in our virtual agents, and that it is feasible to place our characters on Eysenck's scales. A key finding is that the presence of behavioural characteristics reinforces the prescribed personality profiles that are already emerging from the still images. Our long-term goal is to enhance agents' ability to sustain realistic interaction with human users, and we discuss how this preliminary work may be further developed to include more systematic variation of Eysenck's personality scales. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.38 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.37 This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional content from movies and online resources. Then EEG responses and eye gaze data were recorded from 24 participants while watching emotional video clips. Ground truth was defined based on the median arousal and valence scores given to clips in a preliminary study using an online questionnaire. Based on the participants' responses, three classes for each dimension were defined. The arousal classes were calm, medium aroused and activated and the valence classes were unpleasant, neutral and pleasant. One of the three affective labels of either valence or arousal was determined by classification of bodily responses. A one-participant-out cross validation was employed to investigate the classification performance in a user-independent approach. The best classification accuracy of 68.5% for three labels of valence and 76.4% for three labels of arousal were obtained using a modality fusion strategy and a support vector machine. The results over a population of 24 participants demonstrate that user-independent emotion recognition can outperform individual self-reports for arousal assessments and do not underperform for valence assessments. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.37 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.36 Measures of emotion should accurately characterise the nature of an emotional experience, and determine whether that experience is universal or unique to a subgroup or culture. We investigated the value of assessing emotion through skin conductance (an easy-to-interpret physiological measure), and sliders (frequently used and direct measures of perceived emotion). This paper describes findings from two experiments. The first evaluated various slider configurations and found that measured emotions successfully characterized the emotional nature of short videos. The second experiment collected the slider-skin conductance measures of emotion while one sample of Japanese participants, and another sample of Canadian participants, viewed longer videos. The measures were sensitive enough to identify cultural differences consistent with existing literature, and were also able to identify parts of the experience where members from different cultures reacted consistently, pinpointing content that provoked a universal experience. We offer a toolkit of data interpretation techniques to gain more insight into the implicit and explicit emotion data: analyses for expressiveness and agreement that can infer states such as engagement and fatigue. We summarize the aspects of our measurement approach and toolkit in a model: the ability to distinguish the emotional nature of stimuli, individuals and affective interaction. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.36 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.35 Blue-pill robots are engineered to deceive (perhaps in an attempt to secure desirable ends). Red-pill robots, on the other hand, are built to do no violence to truth. While "taking the blue pill" is an option some select, this path, in the context of present and future robotics, is an exceedingly bad one by our lights, and we herein defend this position by attempting to show that the production of blue-pill robots via engineering as we know it should be avoided. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.35 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.34 This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in interpreting and producing the emotional and non-verbal behaviours required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and non-verbal behaviour, since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on non-verbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling etc. We first report on three prototype versions of the SAL scenario, in which the behaviour of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behaviour, dialogue management, and synthesis of speaker and listener behaviour of a SAL character displayed as a virtual agent. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.34 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.33 The task of automatically detecting emotion in text is challenging. This is due to the fact that most of the times, textual expressions of affect are not direct -- using emotion words -- but result from the interpretation and assessment of the meaning of the concepts and interaction of concepts described in the text. This article presents the core of EmotiNet, a new knowledge base (KB) for representing and storing affective reaction to real-life contexts, and the methodology employed in designing, populating and evaluating it. The basis of the design process is given by a set of self-reported affective situations in the International Survey on Emotion Antecedents and Reactions (ISEAR) corpus. We cluster the examples and extract triples using Semantic Roles. We subsequently extend our model using other resources, such as VerbOcean, ConceptNet and SentiWordNet, with the aim of generalizing the knowledge contained. Finally, we evaluate the approach using the representations of other examples in the ISEAR corpus. We conclude that EmotiNet, although limited by the domain and small quantity of knowledge it presently contains, represents a semantic resource appropriate for capturing and storing the structure and the semantics of real events and predicting the emotional responses triggered by chains of actions. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.33 http://opac.ieeecomputersociety.org/opac?year=2011&volume=2&issue=03&acronym=tac IEEE Transactions on Affective Computing http://www.computer.org/portal/site/tac/ http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.32 In this paper, a new approach to the generation and the role of artificial emotions in the decision making process of autonomous agents (physical and virtual), is presented. The decision making system proposed is biologically inspired and it is based on drives, motivations, and emotions. The agent has certain needs or drives, that must be within a certain range, and motivations are understood as what moves the agent to satisfy a drive. Considering that the wellbeing of the agent is a function of its drives, the goal of the agent is to optimize it. Currently, the implemented artificial emotions are happiness, sadness, and fear. The novelties of our approach are, on one hand, that the generation method and the role of each of the artificial emotions are not defined as a whole, as most authors do. Each artificial emotion is treated separately. On the other hand, in the proposed system it is not mandatory to predefine neither the situations that must release any artificial emotion, or any other motivation, nor the actions that must be executed in those cases. Both, the emotional releaser and the actions, can be learnt by the agent, as happens in some occasions in nature, based on its own experience. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.32 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.31 During the previous years, the field of emotional content analysis of speech signals has been gaining a lot of attention and several frameworks have been constructed by different researchers for recognition of human emotions in spoken utterances. This paper describes a series of exhaustive experiments which demonstrate the feasibility of recognizing human emotional states via integrating low level descriptors. Our aim is to investigate three different methodologies for integrating subsequent feature values. More specifically we used the following methods: a) short-term statistics, b) spectral moments and c) autoregressive models. Additionally we employed a newly introduced group of parameters which is based on the wavelet decomposition. These are compared with a baseline set comprised of descriptors which are usually used for the specific task. Subsequently we experimented on fusing these sets on the feature and log-likelihood levels. The classification step is based on hidden Markov models while several algorithms which can handle redundant information were used during fusion. We report results on the well-known and freely available database BERLIN using data of six emotional states. Our experiments show the importance of including information which is captured by the set based on multiresolution analysis and the efficacy of merging subsequent feature values. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.31 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.30 This paper reports on a new methodology for the automatic assessment of emotional responses. More specifically, emotions are elicited in agreement with a bi-dimensional spatial localization of affective states. i.e. arousal and valence dimensions. A dedicated experimental protocol was designed and realized where specific affective states are suitably induced while three peripheral physiological signals, i.e. ElectroCardioGram (ECG), ElectroDermal Response (EDR), and ReSPiration activity (RSP), are simultaneously acquired. A group of 35 volunteers was presented with sets of images gathered from the International Affective Picture System (IAPS) having five levels of arousal and five levels of valence, including both a neutral reference level. Standard methods as well as non-linear dynamic techniques were used to extract sets of features from the collected signals. The goal of this paper is to implement an automatic multi-class arousal/valence classifier comparing performance when extracted features from non-linear methods are used as alternative to standard features. Results show that, when non-linearly extracted features are used, the percentages of successful recognition dramatically increase. A good recognition accuracy (>90%) after 40-fold cross-validation steps for both arousal and valence classes was achieved by using the Quadratic Discriminant Classifier (QDC). http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.30 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.29 A common objection to 'emotional' robots is that they are deceptive. This intuitive response assumes (1) that these robots intend to deceive, (2) that their emotions are not real, and (3) that they pretend to be a kind of entity they are not. We use these criteria to judge if an entity is deceptive in emotional communication: good intention, emotional authenticity, and ontological authenticity. They can also be regarded as 'ideal emotional communication' conditions that saliently operate as presuppositions in our communications with other entities. While the good intention presupposition might be an illusion we really need for sustaining the social life, in the future we may want to dispense with the other conditions in order to facilitate cross-entity communication. What we need instead are not 'authentic' but appropriate emotional responses -- appropriate to relevant social contexts. Criteria for this cannot be given a priori but must be learned -- by humans and by robots. In the future we may learn to live with 'emotional' robots, especially if our values would change. However, contemporary robot designers who want their robots to receive trust from humans better take into account current concerns about deception and create robots that do not evoke the three-fold deception response. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.29 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.28 Emotion modeling and recognition has drawn extensive attention from disciplines such as psychology, cognitive science and lately engineering. Although a significant amount of research has been done on behavioral modalities, less explored characteristics include the physiological signals. This work brings to the table the ECG signal and presents a thorough analysis of its psychological properties. The fact that this signal has been established as a biometric characteristic calls for subject dependent emotion recognizers, that capture the instantaneous variability of the signal from its homeostatic baseline. A solution based on the empirical mode decomposition is proposed for the detection of dynamically evolving emotion patterns on ECG. Classification features are based on the instantaneous frequency (Hilbert-Huang transform) and the local oscillation within every mode. Two experimental setups are presented for the elicitation of active arousal and passive arousal/valence. The results support the expectations for subject specificity, as well as they demonstrate the feasibility of determining valence out of the ECG morphology (up to 89\% for 44 subjects). In addition, this work differentiates for the first time between active and passive arousal, and advocates that there are higher chances of ECG reactivity to emotion when the induction method is active for the subject. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.28 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.27 Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This article is the first survey of the domain that jointly considers its three major aspects, namely modeling, analysis and synthesis of social behaviour. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behaviour via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead. Furthermore, the survey identifies the most important application domains where Social Signal Processing is likely to have a major impact. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.27 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.26 For many years psychological research on facial expression of emotion has relied heavily on a recognition paradigm based on posed static photographs. There is growing evidence that there may be fundamental differences between the expressions depicted in such stimuli and the emotional expressions present in everyday life. Affective computing, with its pragmatic emphasis on realism, needs examples of natural emotion. This paper describes a unique database containing recordings of mild to moderate emotionally coloured responses to a series of laboratory based emotion induction tasks. The recordings are accompanied by information on self-report of emotion and intensity, continuous trace-style ratings of valence and intensity, the sex of the participant, the sex of the experimenter, the active or passive nature of the induction task and it gives researchers the opportunity to compare expressions from people from more than one culture. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.26 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.25 MAHNOB-HCI is a multi-modal database recorded in response to affective stimuli with the goal of emotion recognition and implicit tagging research. A multi-modal setup was arranged for synchronized recording of face videos, audio signals, eye gaze data, and peripheral/central nervous system physiological signals. 27 participants from both genders and different cultural backgrounds participated in two experiments. In the first experiment, they watched 20 emotional videos and self-reported their felt emotions using arousal, valence, dominance and predictability as well as emotional keywords. In the second experiment, short videos and images were shown once without any tag and then with correct or incorrect tags. Agreement or disagreement with the displayed tags was assessed by the participants. The recorded videos and bodily responses were segmented and stored in a database. The database is made available to the academic community via a web-based system. The collected data were analyzed and single modality and modality fusion results for both emotion recognition and implicit tagging experiments are reported. These results show the potential uses of the recorded modalities and the significance of the emotion elicitation protocol. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.25 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.24 The human machine interface (HMI) is a main communication method between human and computer. Through current HMI, machine receives and accurately responses to the commands instructed by the users. In the next generation of HMI, machines are required to deal with more challenging problems/decisions (such as affective evaluations, ethical quandaries, and other innovations) in a self-governing manner. Thus, future HMI should be able to provide information about users' emotion to machine for affective evaluation. In this article, we focus on the natural connection method that can improve machines to make acquaintance with the users. However, connecting sensors scattered on the human body poses serious problems concerning comfort and convenience. Therefore, the authors introduce the Intra Body Communication (IBC) for connecting various physiological sensors on the human body such that the physiological information can enrich the capability of the computer in cognition of the user's emotion. In addition, the authors also reported two pilot studies: using the IBC for connecting the physiological sensor on the human body and using the physiological parameters in estimation of the fatigue degree of the user. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.24 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.23 Marketing strategies without emotion will not work. Emotion stimulates the mind 3000 times quicker than rational thought. Such emotion invokes either a positive or a negative response and physical expressions. Understanding the underlying dynamics of users' emotions can efficiently help companies formulate marketing strategies and support after-sale services. While prior work has focused mainly on qualitative aspects, in this paper, we present our research on quantitative analysis of how an individual's emotional state can be inferred from her historic emotion log and how this person's emotional state influences (or is influenced) by her friends in the social network. We statistically study the dynamics of individual's emotions and discover several interesting as well important patterns. Based on this discovery, we investigate the spread of emotional states of individuals and study how collective sentiments can be gauged. In both mobile-based social network and online virtual network we verify the effectiveness of our proposed approach. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.23 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.22 The relationships between consumer emotions and their buying behaviors have been well documented. Technology savvy consumers often use the web to find information on products and services before they commit to buy. We propose a semantic web usage mining approach for discovering periodic web access patterns from annotated web usage logs, which incorporate information on consumer emotions and behaviors through self reporting and behavioral tracking. We use fuzzy logic to represent real-life temporal concepts (e.g. morning) and requested resource attributes (ontological domain concepts for the requested URLs) of periodic pattern-based web access activities. These fuzzy temporal and resource representations, which contain both behavioral and emotional cues, are incorporated into a Personal Web Usage Lattice that models the user's web access activities. From this, we generate a Personal Web Usage Ontology written in OWL, which enables semantic web applications such as personalized web resources recommendation. Finally, we demonstrate the effectiveness of our approach by presenting experimental results in the context of personalized web resources recommendation with varying degrees of emotional influence. Emotional influence has been found to contribute positively to adaptation in personalized recommendation. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.22 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.20 SEMAINE has created a large audiovisual database as part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally coloured conversation. Data used to build the agents came from interactions between users and an 'operator' simulating a SAL agent, in different configurations: Solid SAL (designed so that operators displayed appropriate non-verbal behaviour) and Semi-automatic SAL (designed so that users' experience approximated interacting with a machine). We then recorded user interactions with the developed system, Automatic SAL, comparing the most communicatively competent version to versions with reduced nonverbal skills. High quality recording was provided by 5 high-resolution, high framerate cameras, and 4 microphones, recorded synchronously. Recordings total 150 participants, for a total of 959 conversations with individual SAL characters, lasting approximately 5 minutes each. Solid SAL recordings are transcribed and extensively annotated: 6-8 raters per clip traced five affective dimensions and 27 associated categories. Other scenarios are labelled on the same pattern, but less fully. Additional information includes FACS annotation on selected extracts, identification of laughs, nods and shakes, and measures of user engagement with the automatic system. The material is available through a web-accessible database. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.20 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.19 Subjectivity analysis systems automatically identify and extract information relating to attitudes, opinions, and sentiments from text. As more and more people make their opinions available on the Internet, and as people increasingly consult the Internet to ascertain other people's opinions about products, political issues, and so on, the demand for effective subjectivity analysis systems continues to grow. Information extraction systems, which automatically identify and extract factual information relating to events of interest, remain critically important in this day and age of increasingly vast amounts of text available on-line. In this work, we discover that these research areas are mutually beneficial. Information extraction techniques may be used to learn informative clues of subjectivity. Then, by bootstrapping from a lexicon of subjectivity clues, we can build a subjective-objective sentence classifier that does not require annotated data as input. This classifier may then be used to improve information extraction performance, on data which has not been annotated for subjectivity, by improving precision with little loss in recall. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.19 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.18 In this study we tested whether visual complexity (VC) of web pages influences a viewer's affective reactions. In a laboratory experiment, 48 students viewed 36 web pages varying in VC while subjective feelings, behavioral, and cardiovascular responses were recorded. Less complex web pages were associated with more positive affect, decreased eye movements (specifically in the first few seconds of viewing), a triphasic heart rate response, and increased finger pulse amplitude. Results suggest that affective responses to web page viewing differ as a function of VC and that web page displaying could be made adaptive to a user's emotions. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.18 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.17 Most research efforts dealing with recognition of emotion-related states from the human speech signal concentrate on acoustic analysis. However, the last decade's research results show that the task cannot be solved to complete satisfaction, especially when it comes to real life speech data and in particular to the assessment of speakers' valence. This article therefore investigates novel approaches to the additional exploitation of linguistic information. To ensure good applicability to the real world, spontaneous speech and non-acted non-prototypical emotions are examined in the recently popular dimensional model in 3D continuous space. As there is a lack of linguistic analysis approaches and experiments for this model, various methods are proposed. Best results are obtained with the described bag of n-gram and character n-gram approaches introduced for the first time for this task and allowing for advanced vector space representation of the spoken contents. Further, string kernels are considered. By early fusion and combined space optimisation of the proposed linguistic features with acoustic ones, the regression of continuous emotion primitives outperforms reported benchmark results on the VAM corpus of highly emotional face-to-face communication. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.17 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.15 We present a multimodal dataset for the analysis of human affective states. The electroencephalogram (EEG) and peripheral physiological signals of 32 participants were recorded as each watched 40 one-minute long excerpts of music videos. Participants rated each video in terms of the levels of arousal, valence, like/dislike, dominance and familiarity. For 22 of the 32 participants, frontal face video was also recorded. A novel method for stimuli selection is proposed using retrieval by affective tags from the last.fm website, video highlight detection and an online assessment tool. An extensive analysis of the participants' ratings during the experiment is presented. Correlates between the EEG signal frequencies and the participants' ratings are investigated. Methods and results are presented for single-trial classification of arousal, valence and like/dislike ratings using the modalities of EEG, peripheral physiological signals and multimedia content analysis. Finally, decision fusion of the classification results from the different modalities is performed. The dataset is made publicly available and we encourage other researchers to use it for testing their own affective state estimation methods. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.15 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.13 Facial expression is an important channel for human communication and can be applied in many real applications. One critical step for facial expression recognition (FER) is to accurately extract emotional features. Current approaches on FER in static images have not fully considered and utilized the features of facial element and muscle movements, which represent static and dynamic, as well as geometric and appearance characteristics of facial expressions. This paper proposes an approach to solve this limitation using 'salient' distance features, which are obtained by extracting patch-based 3D Gabor features, selecting the 'salient' patches, and performing patch matching operations. The experimental results demonstrate high correct recognition rate (CRR), significant performance improvements due to the consideration of facial element and muscle movements, promising results under face registration errors, and fast processing time. The comparison with the state-of-the-art performance confirms that the proposed approach achieves the highest CRR on the JAFFE database and is among the top performers on the Cohn-Kanade (CK) database. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.13 http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.12 The main goal of the study at hand is the development of a multimodal, ensemble based system for emotion recognition. Special attention is given to a problem often neglected: the problem of missing data. In off-line evaluation the problem can be easily solved by excluding those parts of the corpus where one or more channels are corrupted or not suitable for evaluation. In real applications, however, we cannot neglect the problem of missing data and have to find adequate ways to handle it. To address this issue, we do not expect examined data to be completely available at all time in our experiments. Therefore various standard fusion methods as well as a newly developed fusion scheme - called cascading specialists - will be explained and enriched with strategies on how to overcome problems with temporarily unavailable modalities. We will show that the cascading specialists approach is well suited for practical applications, as it aims at a well balanced recognition performance among classes, especially supporting weakly recognised or rarely occurring classes. Evaluation is carried out on the CALLAS Expressivity Corpus, featuring facial, vocal and gestural modalities. http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.12