Multimodal Behavior Processing

Jun. Prof. Dr. Hanna Drimalla

Multimodal Behavior Processing Group

group picture

The research group Multimodal Behavior Processing, headed by Jun.-Prof. Dr. Hanna Drimalla , is dedicated to the automatic analysis of social interaction signals (e.g., facial expression, gaze behavior, voice etc.) using machine learning as well as speech and image processing. Three aspects are the focus of our research: the detection of positive and negative affect, the measurement of stress, and the analysis of social interaction patterns. All three have in common that they are multimodal and time-dependent phenomena. To address this complexity, we collect innovative training data and develop novel analysis methods. Our goal is Empathic Artificial Intelligence, that recognizes and adapts to a user's mental state


Automatic affect recognition

Previous solutions for affect recognition are based on non-representative and unrealistic data sets. The first step towards better emotion recognition is therefore the collection of a balanced video data set in a situation as natural as possible using standardized test procedures. For this purpose, we have developed two paradigms: The Berlin Emotion Recognition Test (BERT) is a computer-based task for sensitively assessing emotion recognition of a person. The Simulated Interaction Task (SIT) is a simulated social interaction that recods non-verbal interaction behavior. With this data we develop machine learning algorithms for affect recognition, using innovative fusion approaches to integrate different modalities (voice, facial expression, gaze behavior).

Computer-based stress measurement

The measurement of stress has so far focused mainly on self-report or individual parameters of the physiological response. In different stress paradigms we record the non-verbal behavior of test persons together with physiological markers. Based on these multimodal data, we develop algorithms for automatic stress detection and validate them in natural environments. To investigate stress and its effects, psychologist use classical stress induction paradigms like the Trier Social Stress Test (TSST), which are cost- and time-intensive. As scalable solutions to induce stress are still missing, we have developed a Digital Stress Test (DST) .


Analysis of social embedding

To assess the social integration of a person, clinicians and researchers often use questionnaires. In order to capture the social integration of a person more sensitively and objectively, we have developed Dona4Me , a data-donation method, where the participant uploads their social media interaction data from a social platform such as Facebook or Whatsapp. As social media data is highly personalized, we extract only meta-data of these files. In a large online study, we compare the sensitivity of this approach to classical questionnaires. Furthermore, we identify characteristic and helpful interaction patterns using machine learning to predict the social embedding and resilience of a respondent.

NLP on Clinical Psychiatric Notes

Clinical notes of psychiatric patients are a rich resource for gaining a better understand of mental disorders and developing better phenotypes. In a master project at the Hasso-Plattner-Institute of winter term 19/20, we used natural language processing on clinical notes of electronic health records (EHR) from Mount Sinai hospital system to develop meaningful language-based representations of patients with depression. With unsupervised machine learning, we aimed to find categories that are closer to underlying mechanisms as well as subcategories that could inform treatment decisions. In a follow-up project we are now working on Using Natural Language Processing (NLP) to explore gender differences in clinical descriptions of mental health conditions. It is well known that psychiatric symptoms either differ or are perceived differently by doctors, depending on gender or ethnical background of patients. In hospitals, doctors notify these symptoms in clinical notes. Using Natural Language Processing on these notes allows to compare, on a big scale, descriptions of mental conditions between gender and ethnical groups. This information could inform and improve the diagnostic process of mental health conditions in the future.