Gradient Boosting & AdaBoost

Boosting vs Bagging (Recap)

Bagging (Random Forest)

• Train N trees independently (random subsets)
• Predictions averaged
• Reduces variance

Boosting (AdaBoost, Gradient Boosting)

• Train trees sequentially (each corrects previous)
• Trees focused on hard-to-predict samples
• Reduces bias

AdaBoost (Adaptive Boosting)

How it works:

1. Train weak learner (shallow tree) on all data
2. Calculate error and increase weight on misclassified samples
3. Train next learner on reweighted data (hard samples matter more)
4. Repeat N times
5. Combine: weighted vote of all N learners

Why it works:

Early learners catch obvious patterns. Later learners focus on edge cases. Final model = consensus of experts, each specializing in different patterns.

Gradient Boosting

More general version of AdaBoost:

1. Train initial weak learner
2. Calculate residuals (errors)
3. Train next learner to predict residuals
4. Update predictions: pred = pred + learning_rate × residuals_pred
5. Repeat

Key difference from AdaBoost:

• AdaBoost: Reweight samples
• Gradient Boosting: Fit residuals

Gradient Boosting > AdaBoost in most cases!

Hyperparameters

Comparison: Boosting vs Bagging

Popular Boosting Libraries

• scikit-learn: GradientBoostingClassifier, AdaBoostClassifier
• XGBoost: Faster, handles missing data (Lesson 5, Module 6)
• LightGBM: Even faster, memory efficient
• CatBoost: Handles categorical features automatically