I designed a system for automatic recognition of facial expressions. The idea is to automatically detect facial Action Units (AUs) and their temporal segments in frontal-view face videos. I used a non-rigid registration technique to determine the motion in the input videos. Each video was then segmented in to a set of regions, from which features were extracted. A combination of an HMM and a boosting algorithm then detects the presence of AUs. I presented this work at the Face and Gesture Recognition 2008 conference, where it was awarded the best student paper award.

An extended version of this work was published in the IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) and also formed a chapter in my PhD Thesis.

1. Affective and Implicit Tagging using Facial Expressions and Electroencephalography. pdf bibtex S. Koelstra. Queen Mary University of London, 2012. PhD Thesis

   @PhdThesis{Koelstra2012affective,
       title = "{Affective and Implicit Tagging using Facial Expressions and Electroencephalography}",
       author = "S. Koelstra",
       institution = "Queen Mary University of London",
       month = march,
       year = "2012",
       note = "PhD Thesis",
   }

2. A Dynamic Texture based Approach to Recognition of Facial Actions and their Temporal Models. pdf bibtex S. Koelstra, M. Pantic and I. Patras. In IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, number 11, 2010.

   @article{Koelstra10,
     author = {S. Koelstra, M. Pantic and I. Patras},
     title = {A Dynamic Texture based Approach to Recognition of Facial Actions and their Temporal Models},
     journal = {IEEE Trans. Pattern Analysis and Machine Intelligence},
     pages={1940--1954},
     year={2010},
     volume={32},
     number={11},
     abstract = "In this work we propose a dynamic-texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modelling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Non-rigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2\% for the MHI method and of 94.3\% for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener dataset.",
   }

3. Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics. pdf bibtex S. Koelstra and M. Pantic. In Automatic Face & Gesture Recognition, 2008. FG'08. 8th IEEE International Conference on, 2008.

   @inproceedings{koelstra2008non,
     title={Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics},
     author={S. Koelstra and M. Pantic},
     booktitle={Automatic Face \& Gesture Recognition, 2008. FG'08. 8th IEEE International Conference on},
     pages={1-8},
     year={2008},
     organization={IEEE}
   }

4. Using appearance-based features in the recognition of facial actions and their temporal dynamics. bibtex S. Koelstra. Delft University of Technology, 2007. MSc. Thesis

   @mscthesis{Koelstra2007msc,
           author = "S. Koelstra",
           title = "Using appearance-based features in the recognition of facial actions and their temporal dynamics",
           institution = "Delft University of Technology",
           year = {2007},
           note = "MSc. Thesis"
   }