Eval
draft-automated, single-annotator v0. "predicted" = live src/core/segmenter.ts output at report-generation time. "reviewed" (this report's reference for these numbers) = the machine-drafted v0-set label, confirmed-or-corrected once by the builder during eval/v0-set/ construction (eval/v0-set/reviewed-overrides.json holds only the divergent items). This is NOT the single-annotator human self-test-retest protocol docs/batch2-linguistic-spec.md Section 4 defines, and no such pass has run yet (docs/DEVIATIONS.md). These numbers measure whether the CURRENT code reproduces its own reviewed v0-set labels — a regression/consistency check, not an accuracy claim against independent gold, and not inter-annotator agreement. Single-annotator v0, seeking co-annotators.
Source
| Texts | 16 (original compositions, no scraped content — eval/v0-set/texts.json) |
| Tokens (leaves) | 438 |
| Reviewed overrides applied | 1 (see reviewed-overrides.json) |
Boundary-F1 (primary metric)
High variance at small n (docs/batch2-linguistic-spec.md Section 5.2) — a single missed or extra boundary swings this number by several points at this sample size. Does not say WHICH label was wrong on either side of a correctly-placed boundary.
| Precision | Recall | F1 | n (gold switch points) | TP | FP | FN |
|---|---|---|---|---|---|---|
| 100.0% | 97.9% | 98.9% | 95 | 93 | 0 | 2 |
Per-label token accuracy (secondary — never pooled)
Per-label, never pooled (Section 5.1) — a trivial always-predict-majority-label baseline would score high pooled accuracy while being useless at the rare classes (MIXED, AMBIGUOUS, NE) this project exists to surface.
| Tag | Support (n) | Accuracy |
|---|---|---|
| TAG | 317 | 100.0% |
| ENG | 56 | 98.2% |
| MIXED | 3 | 100.0% |
| NE | 5 | 100.0% |
| OTHER | 56 | 100.0% |
| AMBIGUOUS | 1 | 100.0% |
Confusion matrix (rows = reviewed/gold, columns = predicted)
| gold \ pred | TAG | ENG | MIXED | NE | OTHER | AMBIGUOUS |
|---|---|---|---|---|---|---|
| TAG | 317 | 0 | 0 | 0 | 0 | 0 |
| ENG | 0 | 55 | 0 | 0 | 0 | 1 |
| MIXED | 0 | 0 | 3 | 0 | 0 | 0 |
| NE | 0 | 0 | 0 | 5 | 0 | 0 |
| OTHER | 0 | 0 | 0 | 0 | 56 | 0 |
| AMBIGUOUS | 0 | 0 | 0 | 0 | 0 | 1 |
Self-test-retest kappa
Not available. Self-test-retest kappa requires two independent annotation passes over blind-shuffled items (docs/batch2-linguistic-spec.md Section 4.1) — that is what the annotation workbench (/annotate) is for, and no human pass has run yet. No kappa number is fabricated or estimated here.
Reproduce this page
pnpm gen:eval regenerates eval/results/v0-eval-report.json from the live segmenter — deterministic, no timestamps, no randomness, verified byte-identical across repeated runs, so pnpm ci:eval-check (run in CI) fails on any drift between the committed file and what the current code actually produces.