Section 2 · Lesson 6

Validate with cross-validation and macro F1

Use stratified folds, read the confusion matrix, and tune per-class weights for macro F1.

The validation scheme

The notebook creates one set of five stratified folds at the start and reuses it everywhere:

SEED, FOLDS, K = 42, 5, 4
y = train["price_tier"].to_numpy() - 1
folds = list(StratifiedKFold(FOLDS, shuffle=True, random_state=SEED).split(train, y))

Stratified means that each fold keeps the same tier mix as the full data. Reusing the same folds keeps the out-of-fold target features (Lesson 4) and the model validation consistent. Every row is predicted exactly once by a model that never saw it, which gives 279,746 OOF predictions to score.

Fold scores and noise

The notebook prints the macro F1 of each fold and the pooled OOF score:

The folds agree within about 0.005, which suggests that the model is stable. The development experiments also found that rerunning the same configuration with a different seed moves the holdout score by about 0.002, so a gain below that is indistinguishable from luck. Multiclass log loss changes more smoothly, so the experiments report it alongside macro F1.

Read the confusion matrix

A confusion matrix counts how often each true tier gets each predicted tier. The following matrix comes from the fold-0 holdout predictions that the project saved (combo_*.npy): an equal blend of CatBoost, LightGBM, and ordinal LightGBM, with a macro F1 of 0.7604.

• Errors are mostly adjacent. Only 891 of the 13,286 mistakes (6.7%) are off by two tiers or more. This is why the ordinal model helps.
• Tier 3 is the weak class. The per-class F1 scores are 0.853, 0.732, 0.629, and 0.828 for tiers 1 to 4. Tier 3 is confused with tiers 2 and 4, which sit on both sides.
• Unseen operators cost about 0.03. Listings from operators that don't appear in train score about 0.03 lower, because the operator-level encodings carry no information for them.

Tune per-class weights for macro F1

Argmax picks the most probable tier. Macro F1 may do better if you nudge borderline rows toward a tier the model under-predicts. The notebook multiplies each class probability by a weight and takes the argmax of the product. tune_class_weights() searches for the weights that maximize macro F1 on the OOF predictions, one class at a time, for three rounds:

def tune_class_weights(p, rounds=3):
    w = np.ones(K)
    for _ in range(rounds):
        for k in range(1, K):
            grid = w[k] * np.exp(np.linspace(-0.3, 0.3, 31))
            scores = [macro_f1(p, np.r_[w[:k], g, w[k + 1:]]) for g in grid]
            w[k] = grid[int(np.argmax(scores))]
    return w

Tier 1's weight stays at 1, which fixes the scale. On the notebook's blend, the weights are [1, 0.98, 1.15, 0.852]. They raise tier 3, the weak class, and lower tier 4.

How much does tuning really help?

The notebook's weights raise the blend's OOF macro F1 from 0.7559 to 0.7577. But the weights are fit on the same predictions that report the score, so that gain is slightly optimistic. A cross-fitted check, which fits the weights on one stratified half and scores the other half, put the real gain at about +0.001. In the model experiments, class weights added +0.000 to +0.002, which is near noise. They helped single LightGBM models, where tier 2 was under-predicted, and added almost nothing to the CatBoost blend. If you use them, fit them on OOF predictions from full cross-validation.

Why the CV score is pessimistic

The cross-validated score understates the expected test score by about 0.004. The notebook builds the target features once, over all five folds. So each fold model's training rows carry encodings that include the validation fold's labels, a mismatch that test rows never have. A nested check on fold 0 rebuilt the training rows' encodings without the validation fold. It scored 0.7562 against 0.7523 for the multiclass model. The submission already follows the nested structure: training rows use OOF encodings, and test rows use encodings fit on all of train.