Section 1 · Lesson 3

The competition metric

The score combines Dice overlap, a false-positive penalty, and an empty-image bonus. Predicting nothing already earns 0.30, so the weights shape every decision that follows.

Dice overlap

Dice compares a predicted mask \(X\) with the true mask \(Y\):

It equals 1 when the masks match exactly and 0 when they share no pixel. The edge case is two empty masks: \(|X| = |Y| = 0\), so the fraction is 0/0. The project's code defines Dice(empty, empty) as 1, and a non-empty prediction against an empty truth as 0.

The score has three parts

A model predicts four masks for every image, so the metric works on image-class pairs: 45,572 of them in the training data. Of those, 7,201 pairs have a defect and 38,371 are empty. The score is:

• ForegroundDice averages Dice over pairs that have a defect. A missed defect scores 0.
• FalsePositiveScore is 1 minus the average fraction of the image that you wrongly mark on empty pairs.
• EmptyAccuracy is the share of empty pairs where you predicted nothing at all.

This is the function in src/common.py. It also returns the naive mean Dice, where every pair counts and empty-versus-empty is 1:

def competition_score(pred_area, gt_area, inter):
    pos = gt_area > 0
    dice = np.where((pred_area + gt_area) == 0, 1.0,
                    2 * inter / np.maximum(pred_area + gt_area, 1))
    fg = dice[pos].mean() if pos.any() else 1.0
    fp = 1 - (pred_area[~pos] / N_PIX).mean() if (~pos).any() else 1.0
    ea = (pred_area[~pos] == 0).mean() if (~pos).any() else 1.0
    return {"score": 0.7 * fg + 0.2 * fp + 0.1 * ea, "fg_dice": fg,
            "fp_score": fp, "empty_acc": ea, "naive_dice": dice.mean()}

The function only needs three numbers per pair: the predicted area, the true area, and their intersection. That is why the training code can score millions of pixels with a few array operations.

What the weights reward

Do the arithmetic for the model that never predicts a mask. ForegroundDice is 0, FalsePositiveScore is 1, and EmptyAccuracy is 1. The score is 0.2 + 0.1 = 0.3000. Its naive mean Dice is 0.8420, which is just the fraction of empty pairs. Naive Dice makes an empty submission look strong, which is why the project tracks the official score instead.

Why the metric rewards recall

One pair with a defect is worth up to \(0.7 / 7201 \approx 9.7 \times 10^{-5}\) of the score when you segment it perfectly. One empty pair where you wrongly predict something costs at least \(0.1 / 38371 \approx 2.6 \times 10^{-6}\), plus a small amount from the false-positive pixel penalty. A missed defect is therefore about 37 times more expensive than one false alarm.

If your Dice on a true defect is about 0.5, predicting a mask pays off once the chance that the pair has a defect exceeds roughly 5%. So the best decision threshold is low, and the tuned values in lesson 6 are low (0.05 to 0.2). Naive Dice pulls the other way. It keeps improving as you raise the threshold, which lesson 6 shows.

How the score is computed in the code

Two functions in src/train.py evaluate a set of predictions. score_at turns uint8 probabilities into a mask with a single threshold, and then counts areas and intersections:

def score_at(probs_u8, masks_u8, thr):
    pred = probs_u8 >= int(round(thr * 255))
    gt = masks_u8.astype(bool)
    pa = pred.sum((2, 3)).ravel()
    ga = gt.sum((2, 3)).ravel()
    it = np.logical_and(pred, gt).sum((2, 3)).ravel()
    return competition_score(pa, ga, it)

src/evaluate.py calls score_at at thresholds 0.3, 0.5, 0.7, and 0.9 to produce the raw sweep, then runs the post-processing tuner. Probabilities are stored as bytes from 0 to 255, so a threshold of 0.3 becomes 77.