Meta-learning applied to search deep neural network architectures

Abstract

Deep learning has enabled several breakthroughs in complex tasks such as object detection, image segmentation, image classification, and ultra-realistic imaging. Such progress was possible due to the massive amount of data available nowadays and the hardware resources now capable of processing such data in an optimized way. Another important point was the feature extraction mechanisms in neural networks, in addition to the increasingly sophisticated neural architectures. Although manually designed architectures have been successful over the years, this practice has several limitations. One limitation is the time spent by experts in the process of adjusting hyperparameters, which could be better used in less manual and crucial tasks. Another limitation is the lack of an automated strategy that guides the search. Usually, architectures are defined in an empirical process where there are no mechanisms to assist decision making. Also, manual design limits the quality of neural architectures, which is mediated by expert knowledge. Motivated by these issues, research in Neural Architecture Search (NAS) has become increasingly popular, where the focus is on automating the engineering of neural network architectures. NAS methods have been proposed based on different search strategies, such as bayesian optimization, evolutionary algorithms, and reinforcement learning, which have achieved state-of-the-art results for many tasks. However, these methods still have a high computational cost. Moreover, a few works related to using previous knowledge obtained by NAS methods have been found, which could minimize the cost of the methods. One way of taking advantage of prior knowledge is through Meta-Learning (MtL), whose purpose is to learn from previous experiences, in other words, learning to learn. Considering these problems and the hypothesis that MtL is a potential tool to solve them, this project aims to investigate MtL approaches and its applicability in NAS, with the goal of propose methods for recommending neural architectures, which will be validated in image classification datasets. By the end of this project, it is expected that the proposed methods will generate new architectures with good predictive performances and reduced computational complexities regarding the state-of-the-art. (AU)