An effective combination of loss gradients for multi-task learning applied on instance segmentation and depth estimation

Advanced driver assistance systems are responsible for assisting decision making and can play an important role in safety and traffic efficiency. Such systems require robust perception methods to handle complex urban scenes, and one way to achieve this is through instance segmentation. However, due to the difficulty in separating overlapping objects into different instances, this task becomes very challenging. For this, several authors proposed CNN-based methods and used depth information to enhance the instance segmentation performance. A promising way to explore this information is by adopting a multi-task learning approach, in which multiple tasks are learned simultaneously by sharing the same architecture. Usually, this combination is made by the weighted sum of loss functions, in which the weight of each task is defined manually. Nonetheless, when tasks have different natures with variation in the order of magnitude, performing this combination during training so that all tasks converge towards their optimal solution is not trivial. Aiming to get the best possible solution, we modeled the multi-task learning as a multiobjective optimization problem and, as the main contribution of this paper, we proposed a greedy approach to find the weighting coefficients for each task, performing a trade-off between tasks that allow the optimization of multiple loss functions. Experimental results showed that it is possible to enhance instance segmentation when depth information is properly explored. Moreover, not only did depth information help instance segmentation, but also did the instance segmentation help the depth estimations, achieving better performance compared to single-task models. (AU)