$ pytest -v test_llm_answer.py
press Run tests. The first run downloads two small models (~115 MB) into your browser cache; after that everything runs offline-fast on your machine.
grey text in an evidence card is the context the nli model also saw · "label" is my hand label for preset claims
Every claim gets three pieces of evidence and a three-way verdict.
- split
Sentences
The source is cut on. ! ?followed by a space and a capital letter, so “Python 3.0” and “$50.00” stay whole. - context
Premises
Each sentence is glued to the sentences its pronouns point back to, plus the document's first sentence, so “It was first released in 1991” knows what “it” is. - retrieve
Top-3 by cosine
all-MiniLM-L6-v2embeds premises and claims; the three closest premises become the claim's evidence. - judge
NLI cross-encoder
nli-deberta-v3-xsmall(int8 ONNX) scores each (premise, claim) pair. Any entailment ≥ 0.5 → PASS, else any contradiction ≥ 0.5 → FAIL, else SKIP.
Both models run through transformers.js 3.8.1 on onnxruntime-web (WebAssembly, one thread, since GitHub Pages can't send the cross-origin-isolation headers needed for more). The NLI label order is read from the model's own config.json (0 contradiction, 1 entailment, 2 neutral) rather than hard-coded. There's no server: the page is static files and the weights come from the Hugging Face hub.
7 of 8 on the set I tuned on, 12 of 14 on claims I wrote before tuning.
| claim | label | got | why |
|---|---|---|---|
| Python 3.0 is fully backward-compatible with Python 2 tuning | FAIL | SKIP | The right sentence is retrieved (“not completely backward-compatible”) but the model calls “fully” vs “not completely” neutral, 0.99. |
| Guido van Rossum worked at Google held-out | SKIP | FAIL | “created by Guido van Rossum at CWI” reads as a contradiction (1.00): the model treats “worked at X” as exclusive. |
| Northwind Audio offers a two-year warranty on all products held-out | SKIP | FAIL | A policy that lists what's allowed gets read as ruling everything else out (contradiction 0.89 from the 30-day returns sentence). |
Over all 22 labelled claims: 9/9 PASS, 8/9 FAIL and 2/4 SKIP were right. Both held-out misses are the same failure: a claim the source is silent about gets called false. Thresholds barely matter, because this model's probabilities are almost always above 0.9 or below 0.1; every passAt/failAt pair I tried between 0.3 and 0.9 scored the same on the tuning set. What moved the score was how much context each premise gets (below).
Timings are from headless Chrome 155 on my Windows desktop (16 logical cores, but the WASM backend uses one). The 8-claim figure is 5 repeats after a warm-up in each of 5 page loads; “cold” is 3 fresh browser profiles downloading ~115 MB on a fast connection. The first click in the UI also pays a one-time warm-up, so it took 2.2–3.7 s. Accuracy was identical on every repeat. You can re-run the whole eval in your own browser at eval.html.
The model was fine. The sentences I was feeding it weren't.
- Pronouns break sentence-level NLI, and retrieval too. With bare sentences the sample scored 4/8: in my first version “It was created by Guido van Rossum” didn't even make the top 3 for “Python was created by Guido van Rossum”, because the embedding doesn't know what “it” is.
- “Previous sentence” isn't enough. Adding just the previous sentence (6/8) fails on chains like “It was created… It was first released in 1991”. Walking back to the last sentence that names its subject, plus the document's first sentence for “The language…”, got 7/8.
- Context has a price. Gluing the first sentence everywhere means “The tower is 330 metres tall” can “prove” the tower is in Paris. The verdict is right, but the evidence card is less precise, so I show the context in grey instead of hiding it.
- Off-topic contradictions are common. “Python 3.0 was released in 2008” contradicts “Python was first released in 1991” with 0.95. I let any strong entailment win before looking at contradictions: with previous-sentence premises that scored 6/8 vs 5/8 for “take the larger of the two”; with the final premises both rules score 7/8.
A small model with a bias towards “false”.
- Silent ≠ false, but it often says false. 2 of 4 SKIP claims came back FAIL. Treat a FAIL as “go check”, not as proof.
- 22 labelled claims is a small eval. It shows the failure modes; it can't give a precise accuracy.
- No real numeric reasoning. Mismatched years worked here (“completed in 1899” vs “1887 to 1889”), but ranges, units and arithmetic are not checked.
- One claim per line, judged alone. Long, compound claims (“A and B”) should be split; the splitter also breaks on abbreviations like “Dr. Smith”.
- The LLM answers are hand-written. I wrote the claims to look like typical model output; they aren't sampled from a real model.