Tensor networks and deep learning for large scale machine learning and signal processing problems

Abstract

Deep learning neural networks have attracted the attention of the machine learning community because of their appealing data-driven framework and of their performance in a number of pattern recognition tasks. It is well known that the state-of-the-art Deep Neural Networks (DNNs) are highly redundant and contain hundreds of millions of parameters, using up all available memory of personal computers. However, attempts to decrease the width and depth of the neural network layers usually lead to considerable drop of performance. To tackle these issues, the present research project will be focused on using the methods of tensor decompositions and low-rank tensor networks to construct a compact representation of DNNs. It will allow to use a rich set of methods from the theory of tensor networks and to design architectures of the DNNs more efficiently, which will lead to accelerating the learning process and mathematical operations. A key feature of this project is the development of new fundamental approaches for training, testing and storing parameters of DNNs by using the formalism of tensor networks. These approaches will allow to reduce by several orders of magnitude computational complexity and required memory for the operation of the network, while maintaining a quality of prediction. Mathematical and algorithmic tools developed during the project could be used for a wide range of applied problems, such as image recognition, prediction and clustering. (AU)