Label less.
Two identical classifiers start from the same 10 labelled digits. One is handed random images; the other asks for the images it is least sure about. Press run and watch which one gets to 90% first.
How it works
The data is the UCI optical digits set (sklearn's load_digits): 1,797 handwritten digits at 8×8 pixels. I split it once into a 1,297-image unlabelled pool and a fixed 500-image test set that no learner ever sees.
Both learners are the same model: softmax regression (64 pixels + bias → 10 classes) trained from scratch with full-batch gradient descent, Nesterov momentum and a little L2. Both start from the same 10 random labels, chosen by the seed.
Every round each learner gets one more batch of labels. The random learner takes the next images from a shuffled pool. The active learner scores every unlabelled image with its current model and asks for the ones it is least sure about:
- margin — smallest gap between the top two class probabilities (default)
- entropy — highest entropy of the predicted distribution
- least-confident — lowest top probability
The "human" is simulated: it just reveals the true label. Both models are retrained from zero after each batch and scored on the test set. Everything — model, sampling, the chart — is plain JavaScript with no dependencies, in al.js and app.js.
Measured results
I ran bench() (in al.js, page bench.html) in headless Chrome 155: 5 seeds, batch of 10, up to 400 labels, same fixed 500-image test set. Raw output is in results/bench-chrome.json.
| metric (5 seeds) | active · margin | random |
|---|---|---|
| labels to reach 90% | 66 avg (60–80) | 126 avg (100–150) |
| accuracy at 100 labels | 92.9% | 88.6% |
| accuracy at 200 labels | 97.3% | 93.0% |
| labels to reach 95% | 110–160 | 230–400, one seed never |
| accuracy at 400 labels | 98.0% | 95.2% |
| strategy | labels to 90% | at 100 labels | at 200 labels |
|---|---|---|---|
| margin | 66 (60–80) | 92.9% | 97.3% |
| least-confident | 98 (90–110) | 90.7% | 96.8% |
| entropy | 116 (110–120) | 88.2% | 94.8% |
| random | 126 (100–150) | 88.6% | 93.0% |
Training on all 1,297 pool labels gives 97.2% (1,500 iterations; 97.4% at 300). So margin sampling reaches 90% with about half the labels random needs, and by ~200 labels it matches the all-labels model.
Numbers will differ slightly from the live demo if you change batch size or seed; the demo runs the exact same code.
What I learned
- Uncertainty sampling is not a free win. Entropy sampling was worse than random on two of five seeds (120 vs 100 labels to 90%) and on average at 100 labels. With 10 labels the model's probabilities are badly calibrated, and entropy over all ten classes rewards images that are vaguely confusing rather than ones near a real decision boundary.
- Margin was the most reliable, beating random on every seed, so it is the default. It only looks at the top two classes, which is the question that actually decides the prediction.
- Check the baseline before you brag. I expected the all-labels model to need many more iterations, and trained it to 1,500 to be sure. With Nesterov momentum it was already converged at 300 (97.4% vs 97.2%), so the comparison here is fair — but the only way to know was to measure.
- "Beats all labels" is mostly noise. Active learners at 400 labels scored 98.0% vs 97.2% for all 1,297 labels. On a 500-image test set that is 4 images. I report it, but I would not claim it as a result.
Limits
- Toy dataset. 8×8 digits are small, clean and balanced. Real labelling budgets fight class imbalance, noisy labels and duplicates.
- Linear model. Softmax regression is cheap enough to retrain every round in a browser. With a deep model, uncertainty estimates are worse calibrated and retraining is the expensive part.
- Simulated oracle. Labels come from ground truth instantly. A real annotator is slower and less accurate on exactly the ambiguous images uncertainty sampling picks.
- Five seeds. Enough to see the gap, not enough for tight error bars.