# Architecture and editorial data model

## Purpose

This stabilized pipeline establishes a trustworthy PASS 1 intermediate format for
the J. S. Crowley diary. It transcribes what is visible on each physical page and
models local document structure. It deliberately excludes corpus-wide identity
resolution, historical geography, relationships, and other PASS 2 work.

The supplied page PNG is the authority. Raw Tesseract text and TSV are immutable
supporting evidence. A Codex response is a candidate interpretation. An
agent-reviewed fixture is a diagnostic reference, not independent ground truth.

## Pipeline

1. `source verify` checks the immutable package, source metadata, file layout,
   image dimensions, and literal Tesseract TSV statistics.
2. The input builder attaches the current page image and supplies full current
   OCR plus at most 1,500 characters of context on either side. Neighbor context
   is explicitly marked non-authoritative.
3. `CodexExecBackend` invokes the ChatGPT-authenticated `codex exec` command in an
   empty, read-only, ephemeral workspace with a constrained JSON output schema.
   No API key, API SDK, or per-token API request is used.
4. The raw response, CLI events, command profile, exact current OCR, exact
   neighbor-context payload, prompt, errors, and usage fields (when reported) are
   preserved under an immutable run directory. New runs also
   snapshot the exact provider schema and transcription specification at the run
   root so later global schema regeneration cannot erase an input.
5. Deterministic code assigns stable IDs, injects trusted file provenance,
   validates schema and invariants, and performs limited exact-token TSV
   alignment. Alignment failure is only a warning.
6. After both adjacent pages exist, reconciliation creates reciprocal
   continuation links. Independent generation never predicts a future block ID.
   The same deterministic pass revalidates every page and checks for suspicious
   exact suffix/prefix duplication across page boundaries.
7. Image-checked agent revisions may be staged as `agent_reviewed` diagnostic
   fixtures. A static bundle exposes the source image, OCR, candidate, diagnostic,
   two deterministic renderings, diffs, notes, and uncertainties one page at a
   time.
8. Only an explicit owner action can create `owner_approved` status. Approval
   writes a separate edition file and decision event; viewing, bundle generation,
   schema validation, and agent staging cannot authorize edition text.
9. A batch can resume under the same run ID. A previously valid, source-verified
   candidate is reused without another model call; incomplete and failed pages
   remain eligible for the normal attempt and at most one explicit repair.
   Profile changes are appended to run history and remain visible per page.

## Canonical page model

`PageEdition` is the canonical container. Its principal semantic units are
ordered logical blocks: headings, text, diary entries, letters, lists, tables,
annotations, gap notices, and separators. Diary entries and letters own ordered
content parts. Physical source lines are nested within those semantic units for
diplomatic evidence; they are not the corpus's primary semantic structure.

Each source line contains explicit inline nodes for plain text, unclear readings,
illegibility, additions, deletions, and corrections. A line-ending vocabulary
distinguishes word-wrap hyphens from lexical hyphens. Deterministic diplomatic and
reading renderers can therefore be generated without custom inline markup.

Dates retain literal source headings. ISO normalization is separate metadata with
an explicit status, qualitative confidence, and reason. A normalized date never
changes faithful source text.

Schema 1.1 removes the exceptional four-field gap-range shape. A gap notice now
has nullable `start_date` and `end_date` values, and each available endpoint is a
complete `sourceDate`. Thus an endpoint printed as `Nov. 28.` can retain that
literal string while a context-supplied `1897-11-28` is explicitly marked
`inferred` with confidence and reason. Deterministic validation rejects a
year-bearing gap normalization labeled `normalized` when no year appears in its
`date_source`.

Schema 1.2 adds one deliberately narrow `graphical_feature` block for
non-textual source marks. It carries medium, placement, a neutral description,
an optional image-space bounding box, and related block IDs. It contains no
invented transcription and does not participate in textual continuation.
Schema 1.1 pages remain valid; newly generated pages declare 1.2.

## Provenance and review authority

Canonical page envelopes contain source paths and SHA-256 hashes, image
dimensions, OCR engine metadata, model/run identifiers, prompt and schema hashes,
context page numbers, and ordered provenance steps. The raw candidate remains in
its run directory after any review.

Review status is enforced:

- `model_candidate` + `diagnostic_only`
- `agent_reviewed` + `diagnostic_only`
- `owner_approved` + `owner_approved_edition`

The agent review command cannot create or modify owner-approved status. Likewise,
`owner-review build`, `validate`, and `owner-review summary` have no approval side
effects. The explicit transitions are:

- `owner-review approve-as-is`: derives an approved page from the diagnostic;
- `owner-review approve-corrected`: preserves the owner-supplied JSON under
  `evaluation/owner-review/submissions/`, then derives the approved page;
- `owner-review needs-work`: records a reason but produces no approved edition.

Both approval forms append an `owner_review` provenance step, retain the input
SHA-256, and store event lineage under `evaluation/owner-review/states/`. Raw
model candidates and agent diagnostics are never overwritten. The approved
edition is a separate file under `edition/owner-approved/`.

## Source and generated layout

The OCR package remains at an operator-supplied `--source-root`; it is neither
copied nor rearranged. `source/package-lock.json` records immutable expectations
without storing a machine-specific source path.

- `schema/`: canonical and constrained-output schemas
- `prompts/`: direct model transcription specification
- `edition/pages/`: agent-reviewed, diagnostic-only canonical pages
- `edition/owner-approved/`: explicit owner editions only; absent until approval
- `evaluation/goldens/`: diagnostic fixtures, never independent ground truth
- `evaluation/owner-review/`: static checklist/bundle, submissions, and decision
  state
- `artifacts/runs/`: reproducible raw model runs and candidates
- `artifacts/cache/`: content-addressed per-page cache
- `artifacts/reports/`: source audit and evaluation reports
- `pilot/selection.json`: fixed consecutive pages 41–60 selection
- `pilot/runs/`: production-pilot inputs, raw attempts, candidates, deterministic
  renderings, reconciliation, validation, and review report

## Failure and retry behavior

Malformed responses and validation failures are preserved. At most one repair
attempt is allowed by the evaluation profile. Repeated failure remains visible
and prevents creation of a candidate page. Batch processing defaults to one page
at a time and refuses more than 20 pages without an explicit override.

An unreadable addition or deletion may retain its intervention type with
`confidence: unreadable`, a null reading, and no alternatives. The canonical
schema already permitted this; the production pilot exposed an over-strict
validator that forced page 43 to repair an unreadable deletion into a generic
illegible span. The historical attempts and repaired candidate remain unchanged,
while the validator now preserves this distinction in future runs.

## Consecutive production pilot

The fixed pilot is physical pages 41–60, the first 20 content pages of Book 2.
It does not overlap the 14-page evaluation set and does not skip inconvenient
pages. Run `20260808T201223Z-pilot` produced 20 valid diagnostic candidates: 19
first-pass successes, one repaired success, and no failures. Reconciliation made
13 reciprocal links with none unresolved and found no duplicated boundary text.

The owner stopped the initial Sol/xhigh profile for latency after pages 41–43 and
requested Luna/high. The same run resumed without reprocessing those valid pages;
pages 44–60 use Luna/high. The interrupted page-44 Sol attempt is explicitly
archived and produced no candidate. The run manifest and each page retain the
resolved profile and Codex CLI version.

The operational pipeline is ready for owner inspection, but the pilot pages are
not approved and are not goldens. Corpus-wide processing should wait for owner
review of this package, especially the uncertainties on pages 43, 48, and 59 and
the policy question raised by page 56's non-textual handwritten marks.

## Corpus-wide bounded parallel execution

Corpus processing partitions physical pages into consecutive shards of at most
20 pages. A bounded local thread pool schedules shards, while every worker walks
its assigned shard sequentially. The next page can therefore use a newly
validated structured tail from the same shard. At shard boundaries, an existing
canonical predecessor is used when already available; otherwise the established
bounded OCR-tail fallback applies without waiting on another worker.

Page locks prevent duplicate execution. Page attempts, cache entries, and shard
completion reports use independent atomic files; workers never append to a
shared manifest or report. The coordinator writes the run manifest before and
after the pool, then stops model execution before deterministic aggregation.
Whole-corpus reconciliation treats shard boundaries exactly like ordinary
adjacent pages.

Backend/process, authentication, service, timeout, and throttling failures have
their own finite retry/backoff loop and artifacts. They do not consume the one
transcription-repair attempt, which remains reserved for parse, constrained
schema, or canonical invariant failures. Repeated throttling can reduce the
effective shard concurrency without changing the Luna/high production profile.

Final aggregation creates exactly one current state per physical page: a valid
canonical PageEdition or an explicit quarantine record. It also produces source
digest comparison, reciprocal-link and boundary-duplication checks, a targeted
anomaly queue, a seeded 5% sample of otherwise-clean new pages, corpus reports,
and a lightweight static review bundle. None of these deterministic operations
changes review authority.

## Known source-package issue

Literal TSV inspection found manifest metric discrepancies. Tesseract TSV text
may contain literal quote characters; parsing TSV with CSV quote semantics can
collapse many physical records. This project parses each row by literal tab
separation with quoting disabled, reports derived statistics separately, and
never modifies the supplied manifest.
