# Verification Loop V2

## Metadata

- Experiment: verification-loop-v2
- Date: 2026-07-01
- Owner: 我 / Codex
- Status: active trajectory-primary viewer + completed L3 ablation
- Primary source archive: `../.cache/tmp-code/VDiff-image2code-v2.2.zip`
- V-harnesses under review: `v-harness-atomic-actions-v2.2` and
  `v-harness-native-claude-v2.3`
- Related current line: `../20260627-pdf-image-to-html-core-verification-loop-v1/`
- Related foundation line: `../20260617-vdiff-pdf-to-html-data-generation-loop/`

## v-harness Version Relationship

- `external_v_harness_atomic_actions_v2_2` is `executed`: the completed V2 L3
  results use `image2code-v2.2@e224610753529d5eae9a9589bd7485363f5ff1b6`
  and the verified v2.2 archive source.
- `external_v_harness_native_claude_v2_3` is `prepared_only` in V2: source and
  condition records exist, but V2 does not claim a confirmed completed v2.3
  experiment. The confirmed v2.3 execution belongs to Verification Loop V3.
- Canonical cross-project mapping:
  `../20260714-vharness-upstream-manager/VERSION_PROJECT_MATRIX.md`.

## Goal

Use controlled `v-harness-*` code roots for Verification Loop V2: a smaller,
clearer experiment line for testing whether process-level multimodal verification
behavior changes repair behavior and final VDiff-rubric quality.

Compared with V1, this branch is not trying to broaden every verifier condition at
once. V2 starts from the 13-case v-harness slice, fixes the harness and final
rubric, and changes only selected verification signals exposed to the agent loop.

## Current Scope

This project currently does six things:

- records where the v-harness sources live;
- summarizes what the codebase appears to do;
- builds immutable-source provenance manifests for the 13 runnable VDiff v2.2 cases;
- runs a shared active comparison matrix with normalized result rows;
- defines Verification Loop V2 ablations over gate feedback and self-check tools;
- reserves comparison slots for future harness experiments.

It intentionally does not yet:

- copy or vendor the 164MB zip into git;
- couple the harness into Core Verification Loop V1;
- treat native v-loop completion and final visual quality as the same metric.

The extracted v-loop cache is patched only under `.cache/` with reproducible runtime
guards so long Claude CLI epochs are bounded. The canonical source archive remains
the zip SHA recorded in `manifests/source-archives.json`.

## Source Lineage Names

This project separates provider/source names from experiment condition names:

- `mentor_pdf_image_to_html_dgl_v1` - mentor-provided PDF-image-to-HTML
  `data_generation_loop` workflow and its historical `output-test` artifacts.
- `mentor_pdf_to_html_no_image_vTBD` - reserved slot for the second
  mentor-provided workflow mentioned by the user; source not imported yet.
- `external_v_harness_atomic_actions_v2_2` - external collaborator/student
  v2.2 v-harness with explicit MCP doing actions, used by existing L3
  conditions.
- `external_v_harness_native_claude_v2_3` - external collaborator/student v2.3
  v-harness with native Claude `Bash`/`Write`/`Edit` doing tools.

The L3 condition name (`L3_vloop_base`) is an experiment condition, not a source
name. Older post-atomic L3-v1 prototypes are historical implementation notes, not
the current V2 condition naming. See `sources/README.md` for the naming policy.

## Working Hypothesis

The `v-harness-*` entries are not just another PDF-to-HTML pipeline. They are
loop-design candidates:

```text
target image
  -> Claude CLI agent with v-harness-specific tool policy
  -> write relative/flow HTML
  -> render
  -> hard gate: canvas + grounded text geometry + visual assertions
  -> failure feedback
  -> next epoch, up to a fixed cap
```

This is useful as a V2 comparison point against V1 because its control boundary is
different:

- V1 controls rounds and verifier conditions inside our own experiment runner.
- the v-harness controls epochs outside Claude CLI, while the inner code-writing
  loop is delegated to `claude -p`;
- v2.2 uses explicit MCP doing actions; v2.3 keeps Claude native doing tools.
- The acceptance oracle is explicit and external, but its ground truth for images is
  partly model-derived rather than PDF-derived.

## Directory Layout

```text
20260701-vdiff-image2code-v2-harness-comparison/
  README.md
  configs/
    conditions.json
    condition-aliases.json
    conditions-v2-design.json
    models-v2-verification-loop.json
    models.json
  methods/
    README.md
    registry.json
    L1_direct_api/
    L2_claude_sdk/
    L3_v_harness/
  docs/
    source-audit-vdiff-image2code-v2.2.md
  experiments/
    README.md
    verification-loop-v2-spec.md
    mentor13-verification-loop-v2-plan.md
    runbook-l3-v1-v2-ablation.md
    mentor13-v2-verification-loop-v2-l3-ablation.json
    mentor13-v2.2-claude48.json
    runbook-mentor13-v2.2-claude48.md
    results-l3-vloop-base-13case-final-scores-20260704.md
  harnesses/
    README.md
    registry.md
    v-harness-atomic-actions-v2.2/
      README.md
      source.json
      SOURCE-SNAPSHOT.md
      source/
      variants/
    v-harness-native-claude-v2.3/
      README.md
      source.json
      source/
      variants/
  sources/
    README.md
    registry.json
    mentor_pdf_image_to_html_dgl_v1/
    mentor_pdf_to_html_no_image_vTBD/
  manifests/
    source-archives.json
    seeds/
    historical/
    rubrics/
  rubrics/
    README.md
    final/
  runs/
    README.md
  scripts/
    backfill_vloop_final_metrics.py
    patch_vloop_verification_ablation.py
    prepare_assets.py
    patch_vloop_post_atomic.py
    run_experiment.py
```

## Source Notes

The source archive remains in cache:

```text
0-Project/.cache/tmp-code/VDiff-image2code-v2.2.zip
```

The tracked v-harness code roots live in:

```text
harnesses/v-harness-atomic-actions-v2.2/source/
harnesses/v-harness-native-claude-v2.3/source/
```

Those code roots intentionally exclude zip/git provenance, `devset/`, generated
outputs, and logs. Experiments provide `DEVSET_DIR` from the generated cache
mirrors recorded in `manifests/source-archives.json`.

Important archive paths:

- `VDiff-image2code-v2.2/image2html_v2.2/README.md`
- `VDiff-image2code-v2.2/image2html_v2.2/harness/epoch_runner.py`
- `VDiff-image2code-v2.2/image2html_v2.2/mcp_server.py`
- `VDiff-image2code-v2.2/image2html_v2.2/gate.py`
- `VDiff-image2code-v2.2/image2html_v2.2/skills/image2html.md`
- `VDiff-image2code-v2.2/image2html_v2.2/devset/`
- `ChengJade/VDiff:image2code-v2.3` at
  `e9c04d4b382beaa18dc96f1e1fded91eb942dd76`

## Experiment Design

The active small comparison uses the VDiff v2.2 devset cases `case1` through
`case13`, matching the previous mentor-output-test reference count. Historical
mentor artifacts are preserved as reference-only provenance; runnable inputs come
from the v2.2 archive devset.

Existing broad comparison conditions:

- `L1_direct_api`
- `L2_native_claude_sdk`
- `L2_v1_agent_verify`
- `L2_v2_verify_everywhere`
- `L3_vloop_base`

Verification Loop V2 first-priority conditions:

- `L3_vloop_base_max6` - native v-harness reference: gate feedback exposed,
  `diff/align` exposed.
- `L3_v1_hidden_gate_feedback_max6` - gate still runs and logs detail, but
  detailed gate failure feedback is hidden from the next epoch prompt.
- `L3_v2_no_self_check_tools_max6` - gate feedback remains exposed, but the
  agent cannot call `diff` or `align`.
- `L3_v_harness_native_claude_v2_3_max6` - v2.3 native-Claude v-harness with
  `Bash`/`Write`/`Edit` retained and no MCP `write/render/crop` actions.

Reserved follow-up:

- `L3_v3_low_verification_baseline` - hidden gate detail plus disabled
  `diff/align`; intentionally not part of the first interface-ready step.

Each result row records three model roles separately: main agent, verifier, and
final evaluator. The current model for all roles is `claude-opus-4.8`, displayed
as `Claude Opus 4.8 [1m]`, through the local proxy recorded in
`configs/models.json`.

L3 rows intentionally distinguish:

- native v-loop/gate status;
- final offline quality evaluator score;
- timeout or harness failure category;
- legacy condition aliases for interpreting old run artifacts.

## Initial Read

See `docs/source-audit-vdiff-image2code-v2.2.md` for the first pass codebase read.

## Current Focus

The active V2 entry is now trajectory-primary. It opens the v-harness visual
trajectory viewer on `V2/L3 · native v-harness · max6 · 13/13`, while the comparison bundle is
kept as the secondary summary/table view:

```text
Compare V2/L3 native v-harness max6 against V2/L3-v1 hidden gate feedback and
V2/L3-v2 no agent-facing diff/align self-check tools on the 13-case mentor slice.
```

Primary trajectory viewer:

```text
../20260704-vdiff-vharness-visual-trajectory-viewer/public/?dataset=this-run-v3-l3base-max6-20260705-01&case=case1
```

Secondary comparison viewer:

```text
viewer-bundles/verification-loop-v2-l3-ablation-20260705/index.html
```

## Verification Loop V2 Viewer

Run set:

```text
manifests/run-sets/verification-loop-v2-l3-ablation-20260705.json
```

Trajectory datasets:

| Dataset | Meaning | Cases |
|---|---|---:|
| `this-run-v3-l3base-max6-20260705-01` | `V2/L3 · native v-harness · max6 · 13/13` | 13 |
| `verification-loop-v2-l3-v1-hidden-gate-max6-20260705` | `V2/L3-v1 · hidden gate feedback · max6 · 13/13` | 13 |
| `verification-loop-v2-l3-v2-no-self-check-tools-max6-20260705` | `V2/L3-v2 · no diff/align tools · max6 · 13/13` | 13 |
| `this-run-v3-l3base-max10-20260705-01` | `Ref · V2/L3 · native v-harness · max10 · 13/13` | 13 |

Comparison viewer bundle:

```text
viewer-bundles/verification-loop-v2-l3-ablation-20260705/index.html
```

Included viewer conditions:

| Viewer condition | Run id | Cases | Status | Notes |
|---|---|---:|---|---|
| `L3_native_vharness_max6` | `mentor13-v2.2-claude48-v3-l3base-max6-20260705-01` | 13 | complete reference | Native v-harness max6; gate feedback exposed; `diff/align` exposed |
| `L3_v1_hidden_gate_feedback_max6` | `mentor13-v2-loop-v2-l3-ablation-max6-20260705-01` | 13 | complete | Gate details logged but hidden from the next epoch prompt |
| `L3_v2_no_self_check_tools_max6` | `mentor13-v2-loop-v2-l3-ablation-max6-20260705-01` | 13 | complete corrected | Agent-facing MCP server does not register `diff/align`; gate keeps internal diff backend |

Aggregate VDiff rubric scores:

| Viewer condition | Native gate pass | Avg rubric | Min | Max |
|---|---:|---:|---:|---:|
| `L3_native_vharness_max6` | 2 | 3.4962 | 0.00 | 4.72 |
| `L3_v1_hidden_gate_feedback_max6` | 0 | 3.4046 | 0.00 | 4.81 |
| `L3_v2_no_self_check_tools_max6` | 3 | 3.4208 | 0.33 | 4.91 |

Version note:

```text
runs/mentor13-v2-loop-v2-l3-ablation-max6-20260705-01-invalid-l3v2-tool-leak/
```

is archived as an invalid first L3-v2 attempt. It is intentionally excluded from
the primary V2 viewer because deferred `ToolSearch` could still materialize
`diff/align` in that first attempt.

Local viewer:

```bash
cd 0-Project/20260701-vdiff-image2code-v2-harness-comparison
python -m http.server 8768
```

Open:

```text
http://127.0.0.1:8768/viewer-bundles/verification-loop-v2-l3-ablation-20260705/index.html
```

## L3 V-loop Epoch Sweep Result Record

Run set:

```text
manifests/run-sets/l3-vloop-base-epoch-sweep-20260705.json
```

Completed viewer conditions:

| Viewer condition | Run id | Cases | Status | Notes |
|---|---|---:|---|---|
| `L3_vloop_base_max3` | `mentor13-v2.2-claude48-v3-l3base-20260704-01` | 13 | complete reference | Existing run, not rerun in the sweep |
| `L3_vloop_base_max6` | `mentor13-v2.2-claude48-v3-l3base-max6-20260705-01` | 13 | complete | Reran with `workers=12`; all rows success |
| `L3_vloop_base_max10` | `mentor13-v2.2-claude48-v3-l3base-max10-20260705-01` | 13 | complete | Reran with `workers=12`; all rows success |
| `L3_vloop_base_max20` | `mentor13-v2.2-claude48-v3-l3base-max20-20260705-01` | 13 | complete | Reran with `workers=12`; all rows success |

Acceptance checks passed for max6, max10, and max20:

- `run_status=success` for all 13 cases.
- No outer subprocess timeout.
- `artifact_status=valid_html` and `render_status=render_ok` for all rows.
- Both final metrics are present: `offline_quality_v1` and `vdiff_rubric_v1`.
- Every native `result.json.epochs` count matches the number of epoch HTML, PNG,
  and JSON artifacts.
- Full `events.jsonl`, `stdout.log`, and `stderr.log` are present for every row.

Max20 final summary:

| Case | Epochs | Native stop | Holistic | VDiff rubric | Inner epoch timeouts |
|---|---:|---|---:|---:|---:|
| `case1` | 20 | `max_epochs_exhausted` | 4 | 4.35 | 4 |
| `case2` | 20 | `max_epochs_exhausted` | 4 | 4.51 | 4 |
| `case3` | 4 | `gate_passed` | 4 | 4.48 | 0 |
| `case4` | 20 | `max_epochs_exhausted` | 4 | 4.76 | 6 |
| `case5` | 20 | `max_epochs_exhausted` | 3 | 3.78 | 0 |
| `case6` | 20 | `max_epochs_exhausted` | 5 | 4.91 | 2 |
| `case7` | 20 | `max_epochs_exhausted` | 3 | 3.65 | 0 |
| `case8` | 20 | `max_epochs_exhausted` | 3 | 2.61 | 0 |
| `case9` | 20 | `max_epochs_exhausted` | 4 | 4.62 | 5 |
| `case10` | 20 | `max_epochs_exhausted` | 3 | 3.15 | 5 |
| `case11` | 20 | `max_epochs_exhausted` | 2 | 2.46 | 3 |
| `case12` | 1 | `gate_passed` | 0 | 0.00 | 0 |
| `case13` | 20 | `max_epochs_exhausted` | 4 | 0.11 | 5 |

Max10 final summary:

| Case | Epochs | Native stop | Holistic | VDiff rubric | Inner epoch timeouts |
|---|---:|---|---:|---:|---:|
| `case1` | 10 | `max_epochs_exhausted` | 4 | 4.35 | 4 |
| `case2` | 4 | `gate_passed` | 4 | 4.44 | 0 |
| `case3` | 10 | `max_epochs_exhausted` | 4 | 4.48 | 0 |
| `case4` | 10 | `max_epochs_exhausted` | 4 | 4.81 | 1 |
| `case5` | 10 | `max_epochs_exhausted` | 3 | 3.33 | 4 |
| `case6` | 10 | `max_epochs_exhausted` | 5 | 4.83 | 1 |
| `case7` | 10 | `max_epochs_exhausted` | 4 | 4.04 | 0 |
| `case8` | 10 | `max_epochs_exhausted` | 4 | 3.43 | 1 |
| `case9` | 10 | `max_epochs_exhausted` | 5 | 5.00 | 0 |
| `case10` | 10 | `max_epochs_exhausted` | 3 | 3.31 | 2 |
| `case11` | 10 | `max_epochs_exhausted` | 3 | 2.88 | 3 |
| `case12` | 3 | `gate_passed` | 2 | 3.55 | 0 |
| `case13` | 10 | `max_epochs_exhausted` | 4 | 0.33 | 3 |

Two follow-up inspection cases:

- `case12`: native gate passes early at max10, while final metrics remain low.
  In max20 it passes even earlier at epoch 1 but final metrics are `0/0.00`.
  This is a useful gate-vs-final-eval mismatch case.
- `case13`: holistic quality and VDiff rubric strongly disagree. The render is
  plausible to the holistic evaluator but fails most rubric checks, and this
  persists through max20.
- `case6`: max20 ends with many native gate mismatches but still receives high
  final scores (`5` holistic, `4.91` rubric), showing that native gate mismatch
  counts and final quality are not interchangeable.
- `case11`: native gate feedback improves near the end of max20, but final
  scores drop to `2` holistic and `2.46` rubric, showing deeper loops can repair
  one signal while hurting another.

Local viewer:

```bash
cd 0-Project/20260701-vdiff-image2code-v2-harness-comparison
python -m http.server 8769
```

Open:

```text
http://127.0.0.1:8769/viewer-bundles/l3-vloop-base-epoch-sweep-20260705/index.html
```

An HTTP `200 OK` for `index.html` is expected. If the browser shows a directory
listing or cannot load the page, the local server is not running from this
subproject root.

## Next

- Keep this subproject as the single home for Verification Loop V2.
- Inspect the max3/max6/max10/max20 viewer traces, especially `case6`,
  `case11`, `case12`, and `case13`.
- Treat any future epoch extension as a new condition and run id, not an
  overwrite of the completed max3/max6/max10/max20 runs.
- Keep L3-v1/L3-v2 ablations as separate follow-up conditions, not mixed into
  the base epoch sweep.