Multilingual and multimodal learning for Brazilian portuguese

Abstract

Humans constantly deal with multimodal information, that is, data from different modalities, such as texts and images. In order for machines to process information similarly to humans, they must be able to process multimodal data and understand the joint relationship between these modalities. This multimodal aspect of learning can also be helpful in multilingual applications, that is, applications that involve two or more languages. The goal of this project is to explore strategies that result in a more efficient pre-training in the context of multilingual and multimodal processing, such as analysis of the effect of visual region masking for the masked region classification task, as well as applying the knowledge and results of this research to complement the initial FAPESP project, which emerged as an attempt to tackle the problem of multimodal and multilingual learning involving the Portuguese language, by proposing the adaptation of the VTLM framework to different languages and to more challenging circumstances related to the image-text relationship. We believe that the results achieved will be interesting for the scientific community. Furthermore, this BEPE project aims to enrich the student's training in important areas of computing, deepening her knowledge in natural language processing and machine learning concepts, approaches and techniques. (AU)