# Proof-of-concept evaluation report

## Status and scope

Phase 1.5 stabilization is complete. The gap-range model now reuses full
`sourceDate` endpoints, the operational prompt contains explicit examples for
the principal PoC failures, and a generated static owner-review bundle covers all
14 pages. The owner subsequently compared all 14 pages with the authoritative
scans, approved 12 diagnostics as-is, and supplied corrections for pages 181 and
232. No additional diary page or model was run during that stabilization.

This report covers only the 14 deliberately selected pages in
`evaluation/selection.json`. The page PNGs were treated as authoritative, raw
OCR as supporting evidence, and every generated or agent-reviewed JSON file as
diagnostic only. Owner-approved derivatives are now stored separately under
`edition/owner-approved/`. The full 295-page corpus was not submitted to a model.

The subsequent 20-page consecutive production pilot remains a separate,
unapproved corpus under `pilot/`; it is summarized in
`docs/production-pilot.md` and must not be treated as additional goldens.

The evaluation used the ChatGPT-authenticated Codex CLI backend with model
`gpt-5.6-sol`, `xhigh` reasoning, JSON Schema constrained output, one permitted
structural repair attempt, limited adjacent OCR context, and deterministic
canonical validation. Exact run details and raw responses are under
`artifacts/runs/20260808T180301Z-eval/`.

The initial batch produced 12 valid candidates and preserved two canonical
failures (pages 128 and 181). Targeted supplemental runs produced valid
candidates for those two pages without rerunning successful pages. All 14
image-checked diagnostic fixtures validate, remain `agent_reviewed`, and retain
`authority: diagnostic_only`. `evaluation/candidate-index.json` records the
successful candidate run for every selected page.

Golden-style comparison found reading-text changes on four pages (6, 127, 181,
and 232) and no top-level block-type changes. That comparison is only a diff aid,
not an accuracy score: consequential review changes on pages 110, 116, 128, and
288 affected date, boundary, continuity, or review metadata without necessarily
changing ordinary rendered text.

## What worked

- The image/OCR/context evidence hierarchy was followed well on ordinary prose.
  The model regularly corrected obvious OCR corruption without modernizing the
  visible source.
- Logical diary entries, letters, prose, ledgers, parallel supply lists, blank
  pages, book dividers, annotations, and the explicit skipped-book notice fit in
  a compact block-first model. Physical lines remained nested evidence.
- Explicit inline nodes were usable for handwriting, uncertainty, deletion, and
  correction. Qualitative confidence allowed the model to decline certainty.
- The provider-neutral backend boundary kept Codex CLI event handling, raw
  response capture, and constrained generation outside the editorial model.
- Provenance, content hashes, cache keys, immutable run artifacts, and strict
  source path checks made individual page runs reproducible and inspectable.
- Adjacent-page reciprocal links were not guessed during page generation. The
  deterministic post-pass paired only compatible declarations once both selected
  neighboring pages existed and left a one-sided page 288 declaration unresolved
  for review rather than fabricating a reciprocal target.
- Best-effort exact-token TSV alignment often produced a useful coarse bounding
  box without becoming a blocking dependency.

## What failed or required review correction

- Page 6 silently lost the visibly intentional internal hyphen in
  `nice-pleasant`. The raw candidate is preserved; only the diagnostic review
  copy was corrected.
- Page 127 repeatedly read the visible acronym `A.P.C.M.` as `A.F.C.M.` and added
  a nonexistent line-end hyphen after `cases`. These were corrected only in the
  diagnostic review copy.
- Page 116 classified the end of an incomplete supply-list item as definitely
  closed. Review changed the page boundary to `uncertain`; the next scanned page
  begins unrelated diary prose.
- Page 110 exposed a schema defect when the candidate inferred `1897-11-28` from
  the abbreviated source endpoint `Nov. 28.`. Schema 1.1 now represents both
  endpoints as full source dates: the literal endpoint stays `Nov. 28.`, while
  `1897-11-28` is explicitly `inferred` with confidence and reason.
- Page 181's agent diagnostic incorrectly changed the visible heading to 1896
  and treated 1898 as an inference. Owner review of the scan confirmed the source
  plainly reads `Monday July 11 - 1898`; the owner-approved edition restores 1898
  as faithful text with ordinary normalization and removes the erroneous
  source-error note. The raw model candidate and mistaken diagnostic remain
  preserved as separate historical stages.
- Page 288 incorrectly declared that a completed Thursday entry continued onto
  page 289. The cancellation operation spans the boundary, but page 289 begins a
  newly dated (and deleted) Friday entry. Diagnostic review corrected the block
  boundary without inventing a reciprocal link.
- Page 232 supplied a confident handwritten motto reading. Agent review made it
  low-confidence but chose `Keep` as the initial word. Owner review confirms the
  initial word is `Sleep`; the approved edition retains low confidence only for
  the unresolved final `sleep`/`asleep` wording.
- Pages 128 and 181 initially passed the provider schema but failed stricter
  canonical date-state invariants because a `normalized` date lacked a required
  reason. Their raw failures were retained, and the pipeline was changed so the
  one configured repair attempt also covers canonical invariant failures.
- Provider-schema definition order initially depended on Python set iteration,
  so equivalent schemas could hash differently between processes and miss the
  content-addressed cache. Definition traversal and final ordering are now
  deterministic and are verified under different `PYTHONHASHSEED` values.
- The first evaluation runs referenced the generated provider schema by its
  project path and hash but did not snapshot the schema bytes inside each run;
  later regeneration could replace that path. Future runs now store exact
  `input-schema.json` and `input-transcription-specification.md` snapshots at the
  run root. The historical hashes and raw responses remain preserved, but this is
  a real reproducibility limitation of the initial run artifacts.
- Some blurred ledger names on page 38 remained genuinely uncertain. This is an
  appropriate unresolved reading, not a reason to force a clean string.
- Dense lists and shipment strings preserve text well but expose subjective
  choices about when visual columns are independent lists, table cells, or
  continuations within one item.

## Common OCR problems observed

- The supplied summary reports mean page confidence 92.19 and correctly flags
  difficult evaluation pages including 38 (79.95), 127 (81.21), 116 (83.14),
  288 (89.51), and 128 (89.64). These figures are useful triage hints, not textual
  authority.
- Literal quote characters in Tesseract TSV text were apparently parsed with CSV
  quote semantics when the supplied manifest metrics were produced. Literal-tab
  reparsing finds 50 word-count mismatches, including 21 differences of at least
  20 words; page 6 is reported as 17 words in the manifest but contains 407
  literal TSV word rows, and page 278 is reported as 35 rather than 419.
- Fractions, superscripts, ditto marks, ampersands, abbreviations, and tightly
  packed punctuation are unstable in raw OCR.
- Two-column lists and ledgers are frequently linearized in the wrong reading
  order or lose the relationship between amounts and descriptions.
- Handwriting, diagrams, strikeouts, overtyping, faint carbon-copy characters,
  and page-edge text predictably cause omissions or substitutions.
- Line-end word breaks are generally visible to OCR but do not indicate whether a
  hyphen is merely typographic or lexically intentional.

## Common model risks

- A fluent reading can silently remove small but real source marks, especially
  internal hyphens, repeated punctuation, ditto marks, and irregular spacing.
- The model may be overconfident about document-boundary state when an item ends
  incomplete but the next page changes genre.
- It may make a historically reasonable date inference but label it as mechanical
  normalization or leak it into faithful source text.
- It can let that inferred date leak back into `date_source` and physical lines;
  downstream date consistency checks cannot detect this without image review.
- It sometimes marks a legible but blurred word as unclear, or chooses one
  plausible segmentation of irregular list material without explaining the
  alternative.
- Non-textual layout such as a box around an abbreviation or diagram strokes is
  only partly represented by the small PASS 1 model.

## Remaining schema ambiguities

- `ends_mid_block` and `continuation.to_next` need a documented combination for a
  visibly incomplete block that has no surviving adjacent continuation. The
  Phase 1.5 prompt now gives a conservative rule; owner review will test it.
- The distinction between a table, independent parallel lists, and a ledger is
  necessarily editorial for some pages; examples should be added to the prompt.
- A compact optional layout-feature note may be useful for boxes, rules, and
  diagrams. It should not grow into full page-layout modeling in PASS 1.
- Coarse block-level TSV evidence works for the PoC. Inline-node or line-level
  alignment should remain optional until review needs demonstrate its value.
- A cross-page cancellation region is not necessarily a cross-page logical
  entry. The current model intentionally links blocks, not editorial operations;
  the Phase 1.5 prompt now makes this distinction explicit.

## Owner-review stabilization result

`evaluation/owner-review/index.html` is the 14-page checklist and per-page review
bundle. Each page exposes the authoritative image, OCR, raw candidate,
agent-reviewed diagnostic, deterministic reading and diplomatic text, JSON and
reading diffs, editorial notes, and extracted uncertainties. Its JSON index also
records the selection reason, model run status, repair use, diagnostic-change
categories, and owner state.

Explicit approval writes to `edition/owner-approved/`; corrected submissions and
event state remain separately preserved. Current state is 14 approved pages: 12
as-is and two owner-corrected. The summary command reports approved, corrected,
needs-work/deferred, and unapproved counts plus owner change categories and pages
with textual changes. It deliberately calculates no “accuracy” against agent
diagnostics.

## Recommendations before all 295 pages

1. Use these 14 owner-approved pages as the genuine regression set, while keeping
   candidate and diagnostic stages visible in every comparison.
2. Compare at least one additional multimodal provider/model on this same fixed
   set. Evaluate image-checked diffs, not self-reported confidence.
3. Keep page runs resumable and cached, but schedule adjacent pages close together
   so deterministic reconciliation and reviewer context are immediately
   available.
4. Expand only in owner-reviewed batches. Monitor schema failure, repair,
   uncertainty, alignment, and manual-change rates before increasing batch size.
5. Keep TSV alignment optional and bounded; do not invest in fuzzy region
   alignment or a large review UI unless owner-review experience demonstrates a
   concrete need.

The production pilot has now satisfied the operational prerequisite in item 3,
including resumability and reciprocal reconciliation. The corpus-wide decision
remains deferred until owner review of the pilot text.
