// Stage 2 · Classical Machine Learning
Trees & Ensembles
Decision trees, random forests, gradient boosting, XGBoost/LightGBM — the kings of tabular data.
about 8 hours6 algorithms3 paperslesson text in progress
01Goals
- Understand information gain and Gini impurity
- Know the difference between bagging and boosting
- Tune the key parameters of XGBoost/LightGBM
Do these first: The ML Workflow: Data, Loss & Generalization
02Algorithms to master
Each card has the original paper, prerequisites and resources.
03Must-read papers
- Random Forests ↗2001 ✓ verified
- Greedy function approximation: A gradient boosting machine. ↗2001 ✓ verified
- XGBoost: A Scalable Tree Boosting System ↗2016 ✓ verified
04Courses & videos
- scikit-learn.org ↗Official docsEN
- xgboost.readthedocs.io ↗Official docsEN
- statquest.org ↗VideoEN
05Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Compare a single tree, a random forest and XGBoost on the same dataset, then explain the best model with SHAP.
- 1Kaggle Learn：Intermediate Machine Learning ↗Exercises
Use its XGBoost exercise to tune n_estimators and learning rate.
- 2scikit-learn.org ↗Notebooks
Run the official ensemble examples comparing random forests, gradient boosting and AdaBoost.
06Self-test
Answer in your head, then open to check.
Why is a random forest more stable than one tree?
Averaging many decorrelated trees reduces variance.
What does each round of gradient boosting fit?
The negative gradient of the current model’s loss (the residuals).
Do tree models need feature scaling?
Usually not — splits only depend on ordering.
On these routes: Domain Scientists