Section 2 · Lesson 5

Compare models, from baseline to boosting

Trace the path from default LightGBM to categorical regularization, ordinal models, and CatBoost.

Gradient boosting in one paragraph

A boosted model starts with a constant prediction. Then it repeats the following loop: measure the residuals (how far each prediction is from the truth), fit a small decision tree to those residuals, and add a fraction of that tree, set by the learning rate, to the prediction. Each round corrects what the earlier trees missed. The following demo runs the loop on a one-dimensional curve.

The following video runs the same idea on real data. It trains LightGBM on only geo_lat and geo_lon for the largest-spread metro, MTR-01, and shows the decision regions after each round. Holdout macro F1 climbs from 0.152 after one round to 0.421 after 300 rounds. With two features only, this is far below the full model, but the mechanism is the same.

The model ladder

The model experiments (experiments/models/RESULTS.md) use the basic features and the same holdout of 55,950 rows. The first row is the baseline.

Regularize the categorical splits

The biggest single-model gain (+0.0105 and a log loss drop from 0.628 to 0.603) came from changing four LightGBM parameters that control how it splits categorical columns. With about 42,000 operator levels, default categorical splitting memorizes operators. Dropping operator_ref or using integer codes was worse than regularizing it, which shows that the column carries signal. After this change, the sweeps of num_leaves, min_child_samples, feature_fraction, and lambda_l2 were all within noise.

The final notebook solved the same problem differently. It replaced the IDs with counts and target encodings (Lesson 4), which worked better than regularizing them.

Use the order of the tiers

An ordinal model respects the order 1 < 2 < 3 < 4. Instead of one four-class model, it trains three binary models, one each for P(tier > 1), P(tier > 2), and P(tier > 3). The notebook's cumulative_to_classes() turns them into class probabilities:

def cumulative_to_classes(p_above):
    p_above = np.minimum.accumulate(p_above, axis=1)   # keep P(>1) >= P(>2) >= P(>3)
    edges = np.column_stack([np.ones(len(p_above)), p_above, np.zeros(len(p_above))])
    return np.clip(edges[:, :-1] - edges[:, 1:], 1e-6, 1)

The probability of tier \(k\) is P(tier > \(k-1\)) minus P(tier > \(k\)). Three independent models can disagree and produce a negative result, so np.minimum.accumulate forces the sequence to be non-increasing.

The ordinal model beat multiclass LightGBM by about 0.003 macro F1 and trained in about 100 seconds. Its bigger value is diversity: it makes different errors from multiclass models, so it adds more to a blend than another multiclass model does.

CatBoost: strongest alone, left out of the notebook

CatBoost with native categoricals was the strongest single model (about 0.744 and log loss 0.589), but one fit took 33-37 minutes at four threads. On the full feature set it reached 0.753 and added only +0.0003 to the blend, at about 12 minutes per fold. The final notebook leaves it out to stay short and fast.

Dead ends worth knowing

• XGBoost with native categoricals overfit through operator_ref (best iteration 130, macro F1 0.714) and was slow. It would need operator_ref handled differently.
• Regression with tuned cut points scored 0.734, below multiclass with argmax.
• Balanced class weights during training were within noise.