The discriminability(d') of facial expressions within gaze-contingent moving foveal windows

Greene, Harold H., Christopher Koch, and Yen-Ju Lee

The discriminability(d') of facial expressions within gaze-contingent moving foveal windows Harold H. Greene1, Christopher Koch2, Yen Ju Lee1 1Psychology Department, University of Detroit Mercy 2Psychology Department, George Fox University (This work is to be presented at the 2011 convention of the American Psychological Association) Background. Affective computing (AC) involves enabling computers to have (artificial) emotional intelligence, such that they may be able to sense the mood of the user, and respond appropriately (Picard & Klein, 2002). One approach is to give computers the ability to interpret users' facial expressions. Towards the successful accomplishment of AC, an important goal is to understand the process by which humans identify facial expressions. The specific issue of concern in the experiment described here is whether facial expressions are identified by feature-driven integration of eyes, mouth, nose, etc; or by a global analysis of faces. A global analysis may be executed by processing spatial (i.e. configural) relationships among individual facial features (e.g. Searcy & Bartlett, 1996), or by a holistic (i.e. Gestalt-like) processing of the face (e.g. Tanaka & Farah, 1993). Within the framework of feature-driven integration models, facial expressions may be identified by individual features, irrespective of the context. For example, the mouth`s shape may suffice for the identification of some expressions, even in visually noisy or inverted faces. In the case of holistic models, the whole face is different from the sum of its individual features. Thus, any distortion to the typical face may be expected to disrupt the identification of facial expressions. For configural models, analysis of the spatial relations among multiple facial features supersedes analysis of individual features. Configural models may be implemented as feature-driven integration of face parts, or as top-down capture of adjacent face regions. It is difficult to rule out configural processing even when there is strong evidence of feature-driven integration in human identification of expressions (e.g. Fiorentini & Viviani, 2009). The problem is that the methods used thus far to address the issue are not disambiguating. Generally, one modifies the facial context in which an individual feature occurs, and predicts no disruption to the expression identification process. Problematically, a finding of no disruption may reflect either local configural processing, or bottom up integration. We propose a method that preserves the configuration of the face, and manipulates only access to feature-driven integration strategies. The method utilizes gaze-contingent moving foveal windows (GCMW; e.g. McConkie & Rayner, 1975). Here, the experimenter defines a window around the observer's fixation point, and this window moves in synchrony with the observer's fixations. Parts of the face that are outside the window are invisible to the observer. If feature-driven integration is most influential for the identification of facial expressions, we hypothesize that sequential integration of facial features through a GCMW should not hinder the identification of facial expressions. In the experiment described, we also explored the hypothesis that face processing strategies may not be similar for all facial expressions. Method. Thirty adult females participated in a 3 Window X 6 Facial Expressions experiment. The faces were from the JACFEE stimulus set (Matsumoto & Ekman, 1988), and they subtended a visual area of 11 deg wide X 14 deg high on the computer screen. An Eyelink II eyetracker was used to implement GCMWs. Facial expression responses (Anger, Sadness, Happiness, Surprise, Disgust, Fear) were indicated by mouse clicks on a screen-based 6 AFC response display. For the Window factor, 12 participants judged facial expressions with the whole face visible (i.e. NW condition), 10 viewed the faces through a large, 2 deg wide X 3 deg high GCMW (i.e. LW condition), and 8 viewed the faces through a small, 1 deg wide X 2 deg high GCMW (i.e. SW condition). In the LW condition, participants had access to 4 % of the face within any single fixation. In the SW condition, 1 % of the face was available per fixation. Participants in the NW condition could use holistic or configural processing, or feature-driven integration strategies. The LW condition allowed configural processing and feature-driven integration but not holistic processing of the faces. Given its size, it is reasonable to purport that the SW condition allowed only feature-driven integration. If feature-driven integration is most influential in the identification of facial expressions, we expected no effect of window size on the discriminability (d') of the six facial expressions presented. Otherwise, if holistic processing is most influential, discriminability should be significantly superior in the NW condition compared to the LW and SW conditions. If configural processing is most influential, then the LW condition should be superior to the SW condition, but similar to the NW condition. Results. Mean d' values in all conditions were greater than 1.40 and less than 5.60. There was a main effect of Window [F(2, 27) = 7.72, p < .01]. Post hoc analyses revealed significantly superior performance in the NW relative to the SW condition (p .05) lower in discriminability compared to Happiness and Sadness. The Window X Facial Expressions interaction was not significant (F= 1). Conclusion. The results of the experiment described here support neither a holistic nor a feature-driven integration account of how facial expressions are identified. A configural account appears to be more defensible. Interestingly, the lack of a significant Window X Facial Expressions interaction indicates that no expression was significantly more susceptible to one kind of processing over another kind of processing. We shall discuss the results and conclusion in depth at the APA conference. References Matsumoto, D., & Ekman, P. (1988). Japanese and Caucasian facial expressions of emotion (JACFEE) and neutral faces (JACNeuF) [Slides]. San Francisco, CA: Department of Psychology, San Francisco State University. McConkie, G. W., & Rayner, K. (1975). The span of the effective stimulus during a fixation in reading. Perception and Psychophysics, 17, 578-586. Picard, R.W., Klein, J., 2002. Computers that recognise and respond to user emotion: theoretical and practical implications. Interacting with Computers 14, 141'169. Searcy, J. H., & Bartlett, J. C. (1996). Inversion and processing of component and spatial-relational information in faces. Journal of Experimental Psychology: Human Perception & Performance, 22, 904-915. Tanaka, J. W., & Farah, M. J. (1993). Parts and wholes in face recognition. Quarterly Journal of Experimental Psychology, 46, 225-245.