Advanced search
Start date

Transfer Learning in Multimodal Learning For Video Emotion Recognition

Grant number: 22/14903-1
Support Opportunities:Scholarships in Brazil - Scientific Initiation
Effective date (Start): February 01, 2023
Effective date (End): January 31, 2024
Field of knowledge:Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
Principal Investigator:Ricardo Marcondes Marcacini
Grantee:Gabriel Natal Coutinho
Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil


Automatic emotion recognition is a relevant task for many purposes. purposes, such as opinion polls and identification of hate speech. We know that an emotion we feel has an impact on interpersonal relationships and social interactions, so it is possible to use sentimental analysis for the purpose of also improving human-machine interaction. Although the emotion recognition task has often been treated unimodally, identifying emotion depends on multiple stimuli, as in identifying facial expressions and voice sounds. In this context, multimodal learning is the area of science that studies the relationship between data from different modalities. The goal of this scientific initiation project is to investigate transfer of learning to recognize emotions in videos. Regarding the transfer learning, it is worth mentioning that recently different pre-trained models for images, texts and audio have been made available. While most existing methods focus on generating new unimodal or multimodal pre-trained models, which has a high computational cost, in this project we raise the following question: how to take advantage of different pre-trained unimodal models to obtain a new and unified multimodal feature space to recognize awareness of emotions in videos? Answering this question will allow not only to reduce the computational cost of multimodal learning, but advancing in transfer and reuse of pre-trained models on different data sets.

News published in Agência FAPESP Newsletter about the scholarship:
Articles published in other media outlets (0 total):
More itemsLess items

Please report errors in scientific publications list using this form.