Decision Trees

The Intuition

A decision tree is a flowchart. It asks a series of True/False questions about features to isolate classes.

• Is Age > 30? -> If Yes, go Left. If No, go Right.

How does it pick the best question?

It uses Gini Impurity. Gini measures how "mixed" a group is.

• Gini = 0: Perfectly pure (All apples).
• Gini = 0.5: Maxially impure (50% apples, 50% oranges).

The algorithm tests every possible threshold for every feature, and picks the split that results in the lowest Gini Impurity (highest purity).

Information Gain

Gain = Gini(Parent) - (Weighted Average Gini of Children)