Preliminary Schedule Spring 2025
AITU is an open-for-all study group where interested people come together to share insights and discuss recent advencents in AI and general topics within the field of Machine Learning. One of our members always gives an introduction to the topic after which an open discussion follows. We meet in 2F13 (ITU DR Building, Kaj Munks Vej 11) each Wednesday at 6pm over snacks and pizza and finish around 8pm.
This semester we would like to strenghten our understaing about where the current revolution in AI started - Statistical Learning. Therefore, we will start by revisiting some topics in the area.
In the second half of the semester, we will take a turn back into the apple of (seemingly) everybody’s eye at the moment - Natural Langugage Processing. We will try to build our own GPT, and then explore other models based on the Transformer Architecture.
Statistical Learning Revisited
To start off, we will conduct a grand comparison of historical classification models, such as LDA/QDA, SVMs and Decision Trees as a Kahoot quiz. We will look at their performance in different dimensionality settings and discuss their respective strong and weak sides.
During our second meeting we will look at a paper that received some attention on Twitter (X) in 2022, a couple of months before ChatGPT was released. Does it still hold? Is there still space in the world dominated by deep neural networks for tree-based models?
We will continue the discussion with CatBoost, and later explore additional topics like GMMs and MCMC.
-
W8: A Comparison of Historical Classification Models
Introduction to Statistical Learning, Chapter 6.4: Considerations in High Dimensions -
W9: Can Statistical Modelling (Still) Beat Deep Learning?
-
W10: CatBoost
-
W11: Gaussian Mixture Models and the EM Algorithm
Pattern Recognition and Machine Learning, Chapter 9: Mixture Models and EM
Elements of Statistical Learning, Chapter 8.5: The EM Algorithm -
W12: Sampling Methods: Markov Chain Monte Carlo
Pattern Recognition and Machine Learning, Chapter 11.2-3: Sampling Methods
Elements of Statistical Learning, Chapter 8.6: MCMC for Sampling from the Posterior
Synthetic Data Generation
-
W13: General Comparison: GAN-based and Autoencoder-based Data Augmentation
Training our own GPT
After reading in detail about GPT-1, it will be time to get our hands dirty. We will make use of Andrej Karpathy’s open source implementation of GPT-2 to train our small language model. We will go through the whole setup from collecting and preparing the data to logging the training results.
- W14: GPT-1 / OpenAI / 2018: Generative Pre-Trained Transformer (Decoder-Only Architecture)
- W15: Live Coding: Training our own nanoGPT
- W16: Easter Break