# J. S. Crowley diary Pass 1 pipeline

This project builds a faithful, page-scoped structured edition from the scanned
page image, raw OCR, and tightly limited neighboring context. The page image is
authoritative; OCR and model output are evidence and interpretation, not ground
truth.

The owner-approved regression set is deliberately limited to the 14 pages in
`evaluation/selection.json`. A separate, unapproved production pilot covers the
20 consecutive physical pages 41–60. Both use a ChatGPT-authenticated Codex
subscription through `codex exec`; the project contains no API key, API SDK, or
per-token API request.

## Quick start

```sh
uv sync --dev
uv run diary source verify --source-root /path/to/diary_ocr_package
uv run diary transcribe batch \
  --source-root /path/to/diary_ocr_package \
  --selection evaluation/selection.json
uv run diary validate edition/pages/page-*.json \
  --source-root /path/to/diary_ocr_package
uv run diary evaluate indexed
uv run diary owner-review build \
  --source-root /path/to/diary_ocr_package
uv run diary pilot package \
  --run-id 20260808T201223Z-pilot \
  --source-root /path/to/diary_ocr_package
uv run pytest
```

Corpus-wide PASS 1 uses bounded parallelism between consecutive physical-page
shards while remaining sequential inside each shard:

```sh
uv run diary corpus transcribe \
  --source-root /path/to/diary_ocr_package \
  --workers 4 \
  --shard-size 20 \
  --profile-path config/pilot-luna-high.toml \
  --run-id STABLE-RUN-ID
```

Resume the same run with `--resume --run-id STABLE-RUN-ID`. Infrastructure
retries are finite and recorded separately from the single transcription-repair
attempt. Workers write only independent page and shard artifacts; deterministic
aggregation, whole-corpus reconciliation, QA sampling, reports, and the static
review index run after model execution stops.

See `docs/architecture.md` and `docs/editorial-specification.md` before treating
any transcription as edition text. Agent-reviewed evaluation fixtures remain
diagnostic-only historical stages. The owner-approved regression derivatives are
stored separately under `edition/owner-approved/`.

The successful raw-candidate paths are listed in
`evaluation/candidate-index.json`. The corresponding image-checked diagnostic
copies are under `evaluation/goldens/` and `edition/pages/`; both locations
deliberately retain `authority: "diagnostic_only"`. Use `diary reconcile` to run
the deterministic reciprocal-link step on a supplemental adjacent pair.

The Phase 1.5 inspection entry point is
`evaluation/owner-review/index.html`. Building or viewing it cannot approve a
page. The only approval transitions are the explicit commands
`owner-review approve-as-is` and `owner-review approve-corrected`; approved pages
are written separately under `edition/owner-approved/`. Use
`owner-review needs-work` to leave a page unapproved with a recorded reason and
`owner-review summary` for current decision counts and change categories. See
`docs/owner-review.md` for the complete workflow.

The production pilot is under `pilot/runs/20260808T201223Z-pilot/`; open its
generated page-by-page inspection bundle at `review/index.html`. Each review
page keeps the authoritative scan beside the OCR, candidate, deterministic
renderings, uncertainties, notes, validation, and preserved run artifacts. The
narrative report remains at `review/report.md`. All 20 pages remain
`model_candidate` / `diagnostic_only`. The run began with Sol/xhigh for pages
41–43, then resumed with Luna/high for pages 44–60 after the owner requested a
faster profile. See `docs/production-pilot.md` for results and the stop decision.

Corpus runs are stored under `corpus/runs/`. Each completed run contains
`canonical/pages/`, `quarantine/`, `shards/`, `reconciliation.json`,
`report/corpus-report.{json,md}`, `report/qa-queue.json`, and
`review/index.html`. The review index is an engineering/editorial QA surface,
not the public diary website and not an approval action.
