Meta-learning for regularizing deep networks under small data regimes

Project summary

We hypothesize that deep learning can yield state-of-the-art performance also for small data sets by meta-learning how to regularize them to avoid overfitting and further improving them by ensembling. We aim to develop the fundamental methods for doing so, and to apply them to various data modalities. We will extend the advances made for rather large tabular datasets to “scale down” deep learning to be effective also in the regime of small datasets. Specifically, we will develop approaches to search for optimal combinations of regularization methods, based on a meta-learning approach across many small datasets and by ensembling different combinations of regularization methods. We will also tackle the more structured data modalities of longitudinal data and image data.

Our methods

  • Data-driven modeling
  • Neural networks
  • Meta-learning

Principal investigator 1

Doctoral researcher position 1

(supervised by PI Grabocka)


Work on meta-features for controlling the negative transfer of the meta-learning methods, adopting regularization cocktails for longitudinal datasets, and developing novel hyperparameter optimization techniques for stacking ensembles.


  • Master’s degree or equivalent in computer science with a specialization on machine learning or artificial intelligence
  • Knowledge of probabilistic machine learning and deep learning
  • Ideally, working knowledge of the transformer architecture, time-series mining, computer vision and analysis of tabular data
  • Strong programming skills, particularly in the PyTorch ecosystem

Principal investigator 2

Doctoral researcher position 2

(supervised by PI Hutter)


Meta-learning for regularizing models on tabular datasets, extending the paradigm towards image datasets, and training optimal ensembles of neural networks.


  • Master’s degree or equivalent in machine learning, data science, computer science, statistics, engineering, math, or similar
  • Knowledge of Python
  • Knowledge of machine learning and deep learning
  • Interest in deep learning for tabular data
  • Interest in AutoML

Administrative Manager

Marc Schumacher

Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center –
University of Freiburg