Section 2 · Lesson 8
Review lessons learned
Review what worked, what did not, and which limits the score still has, then check your understanding.
- Which choices moved the score the most, and which didn't beat noise.
- The known limits of the final score.
- A short checklist you can reuse on your next tabular project.
What moved the score
The following chart collects the holdout gains from the experiment logs. Each bar is the change in macro F1 against the baseline of its own track (experiments/fe, te, and models), so don't add the bars together. Move the slider to change the noise threshold, and see which gains stay above it.
Gains against noise
The project measured run-to-run noise of about 0.002. Bars inside the threshold are gray.
Values: experiments/fe/results.tsv, te/results.tsv, and models/RESULTS.md. The class-weight value is the cross-fitted estimate from NOTEBOOK_GUIDE.md.
What worked
- Treat high-cardinality IDs with care. Replacing raw IDs with counts and out-of-fold target encodings was the biggest lever.
- Encode what's actually informative. Residual encodings, which separate "premium operator" from "large space," were the most important features.
- Add model diversity. An ordinal model gave the blend a different error pattern.
- Check the cheap things first. Facility flags, a simple parsing step, gave +0.004.
- Judge changes against noise. The logs flag differences below about 0.002 as noise, and the final notebook avoids chasing them.
What didn't work
- XGBoost with native categoricals, regression with tuned cut points, facility TF-IDF with SVD, ratio features, date-difference features, and per-group aggregates did not beat noise or lowered the score.
- Balanced class weights during training were within noise. Weighting predicted probabilities afterward did slightly better.
- CatBoost was the best single model, but it cost far more time than it added to the blend.
Known limits of the final score
The class weights fit on the same OOF predictions that report the final score. The honest gain is about +0.001.
Target features are built once over five folds, so CV understates the test score by about 0.004.
Listings from operators that aren't in train score about 0.03 lower, and they're a third of the test set.
Ideas to try next (not tested in the project). Compute the encodings of unseen operators from other signals, such as the operator's profile fields. Or train a model specifically on rows that have no operator history, and see whether it narrows the 0.03 gap.
A reusable checklist
- Read the metric first and choose a validation scheme that matches it.
- Look at missingness before you fill it, and ask why each value is missing.
- Fit every learned step on training data only, and use out-of-fold encodings for target statistics.
- Change one thing at a time on a fixed holdout, and write down the noise level.
- Blend models that make different mistakes.
- Tune post-processing on out-of-fold predictions, and discount the gain you measure on the same data.
Recap quiz
Which statement describes macro F1?
- The share of listings predicted correctly.
- The F1 of the largest class.
- The unweighted average of the per-class F1 scores.
Each tier counts equally, so a weak class can't hide.
Why does the notebook turn operator_ref into counts and target encodings?
- Because LightGBM can't read strings.
- Because raw operator IDs overfit, and a third of test listings have operators that never appear in train.
- Because operators don't affect price.
Experiments showed that LightGBM overfits the raw IDs, and out-of-fold encodings carry the useful signal.
What stops a row's own label from leaking into its target encoding?
- Out-of-fold computation: a fold's encodings come from the other four folds.
- Dividing by the group size.
- Scaling the encoding to [0, 1].
Smoothing limits noise from small groups, but only out-of-fold computation removes the row's own label.
The ordinal model trains three binary models. What does each one predict?
- One of the four tiers.
- P(tier > 1), P(tier > 2), and P(tier > 3).
- The price in dollars.
cumulative_to_classes() subtracts neighboring cumulative probabilities to get the four class probabilities.
A change raises holdout macro F1 by 0.0015. What should you do?
- Keep it, because every gain counts.
- Reject it, because it's positive only by chance.
- Treat it as inconclusive, because the run-to-run noise is about 0.002, and test it with more seeds or a steadier metric such as log loss.
A difference smaller than the noise can't separate a real gain from luck.
Keep going
Review the whole project in the slide deck, or return to the section overview. To rebuild the figures and videos, see build/README.md.