Generating image descriptions on news websites

Abstract

The amount of content produced on the internet and on social networks increases the demand for automatic systems capable of processing and transforming them into information. The particular interest of this project are news from online newspapers, massively produced and available on the internet. For a better understanding of the news and its images, this project proposes description generation for specific regions of the image based on: (1) the explicitly defined alignment between the text and the image of the news and (2) the semantic information present in the text. To do so, it is intended to use a text-image aligner developed in (VELTRONI, 2018) and Abstract Meaning Representation (AMR) of (BANARESCU et al., 2013). This proposal is linked to the project MMeaning (Regular Aid FAPESP # 2016 / 13002-0) and to the proposal "RSMM: Region-specific multimodal models and their application in language tasks" submitted to Newton Mobility Grants 2018 Round 1, proposed in partnership with Profa. Dr. Lucia Specia of the University of Sheffield.