# Verification Loop V3

## Metadata

- Experiment: `verification-loop-v3-extra-hard10-gate-score`
- Date: 2026-07-09
- Status: formal 70-cell run complete; Gate Score v2 reevaluation complete
- Trajectory evaluation: 180 L3 epochs audited; 176 scored and 4 artifact-unavailable
- Seed: `vdiff-image2code-v2.3-extra-hard-case-h1-h10`
- Harness source: `external_v_harness_native_claude_v2_3`
- Primary evaluator: `vharness_gate_score_v1` with contract revision `vharness_gate_score_v2_shared_grounding_semantic_bank_computed_dom`

## v-harness Version Relationship

This project `executed` Native Claude v2.3 pinned at
`image2code-v2.3@e9c04d4b382beaa18dc96f1e1fded91eb942dd76`.
The canonical cross-project mapping is
`../20260714-vharness-upstream-manager/VERSION_PROJECT_MATRIX.md`.

## Goal

V3 restarts the image-to-code comparison with a cleaner evaluation contract.
V1 and V2 relied heavily on visual-only or cached rubric artifacts, which makes
them vulnerable to HTML that visually matches by absolute coordinate pinning,
hidden text, screenshot backgrounds, or other non-live structure hacks.

V3 keeps the L1/L2/L3 ladder but changes the headline score:

- L1: direct image-to-HTML baseline.
- L2: native Claude SDK image-to-HTML baseline.
- L3: `v-harness-native-claude-v2.3`, retaining Claude native `Bash`,
  `Write`, `Edit`, and `Read` while exposing MCP perception tools
  `grep/inspect/align/diff/inpaint`.

The initial cases are `case-h1` through `case-h10` from the v2.3 harness
`devset`. Each case records SHA-256, dimensions, and any matching flat PNG from
`extra-hard-case/`.

## Evaluation Contract

The headline metric is `vharness_gate_score_v1`, a 0-5 weighted pass-rate:

| Subscore | Weight | Purpose |
| --- | ---: | --- |
| `canvas` | 10% | Render exists and canvas size matches the source. |
| `text_geometry` | 40% | Playwright computed DOM text boxes compared with one independent evaluator-owned grounding bank per source case. |
| `fresh_visual_bank` | 30% | Up to eight source-only semantic questions and cached target answers generated once per case and shared by all seven cells. |
| `structure_audit` | 20% | Source and computed-DOM anti-hack checks for absolute positioning, hidden text, and large screenshot/image backgrounds. |

The formula is:

```text
score = 5 * weighted_pass_rate
pass_at_4 = score >= 4
```

`current_html_diagnostics_v1` is stored for debugging and viewer display, but it
is not a cross-condition headline score because it can be candidate-adaptive.
`holistic_quality_v1` is retained only as a secondary continuity metric and is
explicitly visual-only.

Unavailable evidence is never converted to a passing item. A missing grounding,
browser measurement, semantic candidate answer, or equivalence judgment is
reported as `unavailable`; a partial score records its effective reweighted
components instead of silently assigning 5/5.

## Formal Result

The canonical run is
`runs/real-v3-extra-hard10-gate-score-70-20260709-03/`: 70/70 generation cells
completed after removing the outer L3 timeout and preserving the six-epoch cap.
The corrected evaluator was promoted from
`gate-score-v2-independent-grounding-20260711-02`.

| Level | Cells | Mean Gate | Gate >= 4 | Semantic visual | Computed geometry | Structure |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| L1 direct | 10 | 2.254 | 0/10 | 3.876 | 0.292 | 3.124 |
| L2 native Claude SDK | 30 | 2.248 | 0/30 | 3.536 | 0.236 | 3.165 |
| L3 v-harness v2.3 max6 | 30 | 2.934 | 1/30 | 4.439 | 1.029 | 3.498 |

L3 is better than L1/L2 under the corrected common evaluator, especially on
semantic visual checks, but only one L3 cell reaches the strict 4/5 threshold.
Visible absolute/fixed text is still detected in 8/10 L1, 27/30 L2, and 12/30
L3 cells, so the remaining anti-hack findings must stay visible beside scores.

## Resource-To-Quality Result

The L3 harness launches six independent Claude CLI epochs per cell. Although
the experiment config exported `AGENT_MAX_TURNS=40`, the upstream v2.3 epoch
runner did not consume that value, so 40 turns was not an effective guard.
Observed L3 consumption was 9-108 turns per epoch and 87-358 turns per cell.

The posthoc trajectory evaluation reuses the same Gate Score v2 grounding and
semantic bank for every epoch. Of 180 epoch records, 176 contain scoreable HTML
and render artifacts; four first epochs are explicitly artifact-unavailable.

| Signal | Result |
| --- | ---: |
| First available Gate, mean | 2.717 |
| Best observed Gate, mean | 3.134 |
| Final Gate, mean | 2.934 |
| Cells whose best result is epoch 6 | 4/30 |
| Mean epochs executed after the best result | 3.03 |
| Natural Claude epoch completions | 93/180 |
| 900-second epoch timeouts | 87/180 |
| Mean processed input per L3 cell | 8.93M tokens |

Processed input includes uncached input, cache creation, and cache reads; it is
not a fresh-token or uniform-price billing total. Timeout output token and cost
fields are marked partial or unavailable. Resource/quality correlations are
reported as descriptive evidence, not causal effects.

Detailed result slices:

- `comparisons/v3-gate-score-v2-seven-schemes-and-cases.md`: seven-scheme
  aggregate followed by ten per-case tables, each with all seven cells.
- `comparisons/v3-gate-score-v2-case-by-seven-long.csv`: 70-row long-form data
  with all scores, anti-hack counts, epochs, duration, and viewer deep links.
- `comparisons/v3-formal70-cell-matrix.csv`: compact 10 x 7 Gate matrix.

## Directory Layout

```text
20260709-vdiff-image2code-v3-gate-score-evaluation/
  configs/
    conditions.json
    models.json
    prompts/
  experiments/
    verification-loop-v3-extra-hard10-gate-score.json
  harnesses/
    v-harness-native-claude-v2.3/
      README.md
      source.json
      source/
      variants/
  manifests/
    source-archives.json
    seeds/
  methods/
    L1_direct_api/
    L2_claude_sdk/
    L3_v_harness/
  runs/
  scripts/
    prepare_assets.py
    analyze_epoch_resource_quality.py
    reevaluate_v3_run.py
    run_experiment.py
    v3_eval_runtime.py
    v3_metrics.py
  sources/
    registry.json
  viewer-bundles/
```

The tracked harness root is code-only:

```text
harnesses/v-harness-native-claude-v2.3/source/
```

The large/runnable input mirror stays in cache:

```text
.cache/extracted/github-ChengJade-VDiff-image2code-v2.3-e9c04d4b382b/
```

Tracked source excludes `devset/`, `extra-hard-case/`, `output/`, logs, `.git/`,
`__pycache__/`, and generated files. The runner supplies `DEVSET_DIR` from the
cache mirror.

## First Implementation Target

The concrete V3 matrix is 70 cells: 10 L1 cells, 30 L2 replicate cells, and 30
L3 replicate cells. The first done state is:

- V3 subproject metadata is coherent and points to v2.3.
- The seed manifest contains exactly `case-h1..case-h10`.
- L1, L2, and L3 dry-runs produce normalized rows.
- `final-scores.csv/json`, `matrix.json`, eval banks, and diagnostics exist.
- Structure-audit fixture tests catch absolute-position text, hidden text, and
  screenshot/image-background hacks.
- A static viewer bundle can open the dry-run outputs.

## Commands

Prepare source/cache metadata:

```bash
python3 scripts/prepare_assets.py
```

Run one-case dry-runs:

```bash
python3 scripts/run_experiment.py --dry-run --force --run-id dry-v3-onecase-L1 --condition L1_direct_api --case case-h1
python3 scripts/run_experiment.py --dry-run --force --run-id dry-v3-onecase-L2 --condition L2_native_claude_sdk --case case-h1
python3 scripts/run_experiment.py --dry-run --force --run-id dry-v3-onecase-L3 --condition L3_v_harness_native_claude_v2_3_max6 --case case-h1
```

Run the full scaffold matrix without real model calls:

```bash
python3 scripts/run_experiment.py --dry-run --force --run-id dry-v3-scaffold-all70
```

Rebuild the resource-to-quality trajectory evaluation without regenerating any
HTML cells:

```bash
python3 scripts/analyze_epoch_resource_quality.py --workers 18
```