Section 1 · Lesson 6

Post-processing and thresholds

The network gives probabilities, not masks. Three tunable rules per class turn them into masks and lift the holdout score from 0.6420 to 0.6486.

From probabilities to masks

The model saves holdout probabilities as bytes from 0 to 255. The simplest rule is a global threshold \(\tau\): a pixel is defective when its probability is at least \(\tau\). The raw sweep over the whole 2,279-image holdout looks like this:

Raising the threshold removes weak, uncertain pixels. That helps empty accuracy (0.77 at tau 0.05, 0.92 at tau 0.95) but costs foreground Dice (0.52 down to 0.43). Naive mean Dice follows empty accuracy because 84% of its pairs are empty, so it prefers high thresholds. The official score weights foreground Dice at 70%, so it prefers low ones. This is the metric trade-off from lesson 3 playing out on real predictions.

Three rules for each class

A global threshold treats all classes alike, but they differ: defect_d blobs are big and low in contrast, while defect_a blobs are tiny. src/postproc.py therefore gives each class three parameters:

1. thr: the pixel probability threshold.
2. comp_min: remove connected components smaller than this many pixels.
3. area_min: if the surviving mask has fewer total pixels than this, predict nothing for the whole image and class.

def apply(prob, params):
    out = np.zeros(prob.shape, dtype=bool)
    for c, name in enumerate(CLASSES):
        p = params[name]
        b = (prob[c] >= round(p["thr"] * 255)).astype(np.uint8)
        n, labels, stats, _ = cv2.connectedComponentsWithStats(b, connectivity=8)
        keep = np.zeros(n, dtype=bool)
        keep[1:] = stats[1:, cv2.CC_STAT_AREA] >= p["comp_min"]
        m = keep[labels]
        if m.sum() >= p["area_min"]:
            out[c] = m
    return out

Tuning without fooling yourself

The tuner is a plain grid search over 7 thresholds (0.05 to 0.5), 6 values of comp_min (0 to 400), and 8 values of area_min (0 to 4,000) for each class. That is 336 combinations per class. Because the score is a sum over pairs divided by fixed pair counts, each class's contribution can be maximised independently, which keeps the search cheap.

Any parameter tuned on the holdout will look good on the holdout. To measure honestly, src/evaluate.py uses 2-fold cross-validation: it shuffles the holdout into two halves, tunes on one half, and scores on the other, then swaps and averages.

The gap between the cross-validated score (0.6486) and the in-sample score (0.6491) is only 0.0005, so the tuning isn't badly overfit to the holdout. The gain over the raw threshold is 0.0066. About 0.005 of it comes from empty accuracy rising by 0.05, and about 0.002 from foreground Dice.

The tuned parameters

The final parameters, tuned on the whole holdout, are saved in runs/r34_baseline/postproc.json:

Notice that all thresholds are low, consistent with a metric that rewards recall. defect_d uses the lowest threshold (0.05) paired with a component filter of 100 pixels. A plausible reading is that the model is less confident on this class, so the tuner accepts weak pixels but discards small specks. defect_b, the rare class, has the strictest component filter (200 pixels) and a 300-pixel total minimum, which fits a rare class where false alarms are a risk.

See it as a video

This clip sweeps the global threshold from 0.05 to 0.95 and back on three real holdout images. The chart below the images shows the whole-holdout curves from the figure earlier, with a marker that follows the threshold.