Section 2 · Lesson 3

Preprocess without leaking

Turn raw columns into clean numbers, and fit every learned step on training data only.

Two pipelines for two jobs

The project has two preprocessing designs, and the difference teaches you something:

• The Milestone 2 pipeline builds a clean, scaled, fully numeric matrix of 363 columns for a logistic regression baseline.
• The final notebook feeds LightGBM, which doesn't need scaling or one-hot encoding. It uses a lighter set of steps and the target features from Lesson 4.

Both follow the same safety rule, covered later in this lesson.

Clean the data

Milestone 2 handled each data-quality issue with a stated reason:

The project doesn't drop duplicates from the test set, because every test listing needs a prediction.

Fill missing values

The method follows the reason for the gap (Lesson 2):

• borough gets a separate none level.
• Review fields, zones, utility_rooms, and workstations get the median plus a missing indicator, so the model can still see that the value was absent.
• operator_ref becomes unknown when the whole operator profile is missing.

Dropping rows would lose 20-77% of the data. Mean imputation gets pulled by the long right tails. KNN and iterative imputation are slow on 280,000 rows and add little for tree models.

Tame heavy tails

Count columns such as review_count have long right tails. For the linear baseline, the pipeline winsorizes at the 0.5th and 99.5th training percentiles, applies log1p, and then scales. The skew of review_count falls from 5.36 to 0.08. The clip bounds are learned in fit, so a custom transformer stores them:

class Clipper(BaseEstimator, TransformerMixin):
    def __init__(self, low=0.005, high=0.995):
        self.low, self.high = low, high
    def fit(self, X, y=None):
        self.lo_ = np.nanquantile(X, self.low, axis=0)
        self.hi_ = np.nanquantile(X, self.high, axis=0)
        return self
    def transform(self, X):
        return np.clip(X, self.lo_, self.hi_)

Encode categories by their cardinality

Milestone 2 chose an encoding for each column based on its number of levels:

• One-hot for low-cardinality columns: operator_class, access_type, compliance_status, metro, and borough. These add 83 columns, with handle_unknown="ignore".
• One-hot with rare levels grouped for space_class: 131 levels, 77 of them with fewer than 30 rows, collapse to 55 columns with min_frequency=30.
• Target encoding for operator_ref (41,903 values) and zone (1,380 values). One-hot encoding these would add more than 43,000 sparse columns.
• Multi-hot for facilities, and durations instead of raw dates.

Fit on training data only

Data leakage means information from outside the training data influences the model or its validation score. The most common cause is fitting a learned step, such as an imputer's median, a scaler's mean, or an encoder's category means, on data that includes the validation fold or the test set. The score looks better than reality, and the gap appears only on new data.

The project follows these rules:

• Fit every learned step with fit on training data only. Validation folds and the test set only go through transform.
• Put the whole pipeline inside cross_val_score, so it refits on each training fold.
• Use cross-fitting for target encoding, so a row never sees its own label (Lesson 4).
• In Milestone 2, drop exact duplicates before you split, so copies can't straddle a train and validation fold.

Does preprocessing matter? A logistic regression test

Milestone 2 tested the effect on a logistic regression, with 5-fold stratified cross-validation and macro F1. The minimal setup used only the numeric columns with median filling. The preprocessed setup used the full pipeline, refit inside each fold.

The minimal model scores far lower partly because unscaled columns such as max_term_days, in the hundreds of millions, make the optimization badly conditioned, so 300 iterations aren't enough. It also lacks every categorical, facility, and operator signal. For context, the final LightGBM notebook reaches about 0.758.