Determining empirical bounds of a dataset hardness embedding using the Instance Space Analysis framework

Abstract

Machine learning has undoubtedly transformed computer science research, promoting significant technological advances by enabling computers to learn from data, make decisions and recognise patterns autonomously. Although numerous ML algorithms are available, it has been shown that no single algorithm is superior across all possible datasets. Therefore, an effective algorithm selection process is essential, and meta-learning can be crucial for this. Meta-learning studies help to identify key characteristics of datasets, particularly for classification tasks, and relate them to information about the performance of ML algorithms. One framework supporting such a selection process is the Instance Space Analysis (ISA), developed at the University of Melbourne. ISA is unique in its ability to visualise algorithm performance in a 2D plane, providing a clearer understanding of how different algorithms perform. This visualisation provides valuable insights that can help improve algorithm performance and the representativeness of datasets. In previous collaborative work, the Brazilian and Australian groups instantiated the ISA to produce a hardness embedding of a classification dataset in ML. This proposal aims to highlight the benefits of the ISA methodology in this aim and the importance of fully implementing the ISA toolkit into a Python version, with the support of its developers at the University of Melbourne. More specifically, it will focus on implementing the Correlated Limits of the Instance Space's Theoretical or Experimental Regions (CLOISTER) module in the Python package PyISpace. This module allows delimiting feasible regions of an instance space based on empirical evidence.