# Source Audit: VDiff-image2code-v2.2

## Snapshot

- Archive: `0-Project/.cache/tmp-code/VDiff-image2code-v2.2.zip`
- Archive size on disk: about 164MB
- Uncompressed listed payload: about 256MB
- Top-level source folder: `VDiff-image2code-v2.2/`
- Main new harness folder: `VDiff-image2code-v2.2/image2html_v2.2/`
- Curated devset in new harness: 24 cases, each with `page_snapshot.png` and `src.pdf`
- Status of this note: source-audit only; no code was run or changed for this audit

## What Is In The Archive

The archive contains several layers:

| Path | Role | First read |
| --- | --- | --- |
| `image2html_v2.2/` | New standalone harness | Main object of this project |
| `data_generation_loop/` | Older PDF-to-HTML 5-step VDiff pipeline | Foundation / prior workflow |
| `rubrics-diff/` | Rubric generation and verification utilities | Reusable final evaluator idea |
| `viewer/` | Bbox/layout viewers | Useful debugging UI reference |
| `testcase/` | Older small case set with PDFs and snapshots | Historical sample inputs |
| `diff_results.json` | Existing comparison output | Possible historical baseline |

The old `data_generation_loop` is still the large majority of the zip. The new
`image2html_v2.2` folder is comparatively small in code, but includes a 24-case
devset plus some generated logs/artifacts.

## New Harness: image2html_v2.2

The README describes the new harness as a screenshot-to-editable-HTML testbed. Its
core idea is a "multimodal bash": a fixed set of perception interfaces plus a few
atomic actions that a coding agent can use while building HTML.

The intended loop is:

```text
target.png
  -> Claude CLI agent
  -> MCP tools: grep / inspect / align / diff + render / write / crop / snapshot / revert
  -> code.html
  -> deterministic hard gate
  -> pass or failure feedback
  -> next epoch
```

The key distinction from the older VDiff pipeline is that this harness starts from
an image and asks for relative/flow HTML. It is not primarily a PDF text extraction
and spatial binding pipeline.

## Main Components

| File / folder | Purpose |
| --- | --- |
| `README.md` | Explains the harness idea, setup, run modes, and hard gate |
| `config.py` | Central env-overridable config for proxy, models, paths, and gate tolerances |
| `run_one.py` | One-case CLI driver that streams harness events |
| `harness/epoch_runner.py` | Outer loop around `claude -p`; runs epochs and the hard gate |
| `mcp_server.py` | MCP server exposing the 4 interfaces and 5 atomic actions |
| `gate.py` | External acceptance policy: canvas, geometry, visual checks |
| `backends/` | LLM calls, perception, diff, browser render, image-derived truth |
| `skills/image2html.md` | Agent instructions: relative HTML, no absolute text pinning, data-gt binding |
| `web/` | FastAPI UI for choosing/uploading cases and watching events |
| `devset/` | 24 curated hard cases with screenshots and PDFs |

## Control Model

`harness/epoch_runner.py` is the outer harness. Per epoch it:

- launches `claude -p`;
- appends `skills/image2html.md` as the system prompt;
- gives Claude a strict MCP config exposing only the atomic server;
- disallows broad built-in tools such as Bash/Edit/Write/Grep/WebSearch;
- streams Claude's tool calls and results as structured events;
- runs `gate.run_gate()` after the CLI returns;
- writes per-epoch HTML, render, and gate summaries;
- if rejected, injects the hard gate feedback into the next epoch.

This means the harness controls the epoch boundary, while Claude CLI owns the inner
agent loop and compaction.

## Tool Surface

`mcp_server.py` exposes four perception interfaces:

| Tool | Meaning |
| --- | --- |
| `grep(query, on)` | Locate elements matching a text or visual query in target/render |
| `inspect(format, bbox, on)` | Transcribe or describe a region; includes sampled dominant color |
| `align(text, bbox, on)` | Check whether text matches a region |
| `diff(mode)` | Compare target and current render; `spatial` or `visual` |

It also exposes five atomic actions:

| Tool | Meaning |
| --- | --- |
| `render()` | Render `code.html` to `render.png` and return the image |
| `write(...)` | The only mutation path for `code.html` |
| `crop(...)` | Crop target/render regions into local assets |
| `snapshot(...)` | Save current HTML checkpoint |
| `revert(...)` | Restore checkpoint |

This is important for later comparison: the harness is not merely prompting harder;
it constrains the available operations and turns visual perception into named tools.

## Atomic Capability Surface

The code uses the word "atomic" in two nested senses.

Narrow sense: `mcp_server.py` labels five tools as atomic actions. These are
deterministic operations that change or preserve workspace state:

| Atomic action | What it does | Backend type |
| --- | --- | --- |
| `render()` | Render current `code.html` to `render.png` and return the image to the agent | Playwright/browser |
| `write(...)` | The only mutation path for `code.html`; full document write or element-level edits | File + BeautifulSoup |
| `crop(...)` | Crop a target/render region into `assets/<name>.png` | Pillow |
| `snapshot(...)` | Save current HTML as a checkpoint | File copy |
| `revert(...)` | Restore a previous checkpoint | File copy |

Broad sense: the harness's actual "atomic capability surface" has nine tools:

| Capability | Harness role | Is it deterministic? |
| --- | --- | --- |
| `grep(query, on)` | Find every matching visual/text element | No, MLLM grounding |
| `inspect(format, bbox, on)` | Read/describe a region and sample dominant color | Mixed: MLLM + deterministic color sampling |
| `align(text, bbox, on)` | Check text/region match | No, MLLM judgment |
| `diff(mode)` | Find target-vs-render differences | No, MLLM for spatial/visual difference listing |
| `render()` | Execute candidate HTML into pixels | Yes |
| `write(...)` | Mutate candidate HTML through a constrained API | Yes |
| `crop(...)` | Materialize non-text visual assets | Yes |
| `snapshot(...)` | Checkpoint current candidate | Yes |
| `revert(...)` | Restore candidate checkpoint | Yes |

So the important abstraction is not that every atomic capability is deterministic.
It is that each capability is a small, named, inspectable operation with a typed-ish
input/output contract. The agent is not given a general shell/editor; it composes
these operations inside the task loop.

## Does The Harness Verify These Capabilities Broadly?

Current answer from source inspection: not yet in a strong, benchmark sense.

What exists:

- `devset/` includes 24 curated cases, so the harness has a place to test broadness.
- `web/server.py` and `web/static/app.js` expose case gallery, completed-sample DB,
  bbox debug, per-epoch timeline, and batch-style UI affordances.
- `_build_compare_deck.py` suggests a reporting path for many cases.
- `_run_case1.log` is a concrete successful historical run with many tool calls and a
  final hard-gate pass.
- The hard gate validates candidate outputs after each epoch, not the tools in
  isolation.

What I did not find:

- no unit tests for `grep`, `inspect`, `align`, `diff`, `render`, `write`, `crop`,
  `snapshot`, or `revert`;
- no tool-level calibration report showing precision/recall or failure rates across
  the 24-case devset;
- no ablation that disables one atomic capability at a time;
- no aggregate table proving that the nine-tool surface is sufficient across many
  document/UI families;
- no repeated-run stability check for MLLM-backed tools.

Therefore the harness argues for broadness by design and by having a 24-case devset,
but the archive does not yet prove broad existence of the behavior. The strongest
existing evidence is case-level: if a case reaches hard-gate acceptance while using
only the restricted tool surface, then that run demonstrates sufficiency for that
case. It does not by itself establish that the atomic operations are broadly reliable
or generally sufficient.

For our comparison project, a fair next evaluation should separate two questions:

| Question | Needed evidence |
| --- | --- |
| Are the atomic tools individually reliable? | Tool-level probes with expected outputs or human-reviewed labels |
| Is the tool set sufficient for full tasks? | Case-level runs with hard-gate outcomes, epochs, tool traces, and failure categories |

Suggested first audit metrics:

| Metric | Unit | Notes |
| --- | --- | --- |
| `tool_call_count_by_type` | per case / per epoch | Shows which capabilities are load-bearing |
| `tool_error_rate_by_type` | per tool | Captures schema/runtime failures |
| `grounding_nonempty_rate` | per case | Key for `grep` and gate geometry |
| `diff_actionability_rate` | per failed epoch | Whether `diff` feedback leads to next-epoch improvement |
| `accepted_rate` | per case set | Task-level sufficiency, not tool-level reliability |
| `repeat_stability` | repeated runs per case | Especially important for MLLM-backed tools |

## Hard Gate

`gate.py` makes the admission decision. It rejects on any issue from:

- canvas mismatch: rendered page must match source canvas within `CANVAS_TOL_PX`;
- geometry mismatch: grounded target text blocks must bind to rendered DOM text,
  preferably by `data-gt="block_N"`, and land within `ACCEPT_POS_PX`;
- visual mismatch: falsifiable assertions are seeded from the target, discovered
  over epochs, cached, and all must verify against the render.

The agent cannot declare success by itself. The gate renders the HTML fresh, inspects
DOM bboxes, and owns pass/fail.

## Ground Truth Strategy

For PDF-native VDiff, text/object truth can come from PyMuPDF or document layout.
For `image2html_v2.2`, the target is an image, so `backends/image_truth.py` asks a
vision model to ground text blocks and assets. The code comments explicitly call
these coordinates estimated, and the gate tolerance is looser as a result.

Important implication for research framing:

- This harness has a strong external gate.
- But part of the "truth" is itself model-derived.
- For apples-to-apples comparison with V1, we should record whether a condition uses
  PDF-derived truth, image-derived MLLM truth, human rubric, or visual-final evaluator.

## Inputs And Outputs

Expected case input:

```text
devset/<case>/page_snapshot.png
```

Optional PDF exists in devset cases, but the new harness path is image-first.

Per run output goes under:

```text
output/<case>/
  target.png
  code.html
  render.png
  grounding_text.json
  visual_questions.json
  epochs/
  result.json
```

`result.json` includes at least:

- `case`
- `accepted`
- `epochs`
- `acceptance`
- `final_render_path`

## Runtime Requirements

From README/config:

- Python 3.11+;
- Playwright + Chromium;
- FastAPI/Uvicorn for web UI;
- Pillow, BeautifulSoup, python-pptx, `mcp[cli]`;
- Claude Code CLI on PATH;
- model proxy at `http://localhost:4141`;
- `ANTHROPIC_BASE_URL=http://localhost:4141` for Claude Code;
- orchestrator model default: `claude-opus-4-8`;
- grounding model default: `gemini-3.1-pro-preview`;
- perception model defaults to the grounding model.

The config comments warn that the Gemini pro grounding model can return empty content
if `max_tokens` is too low relative to thinking budget. This is directly relevant to
previous V-harness local reproduction where empty OCR/grounding was observed.

## Existing Evidence In The Zip

`image2html_v2.2/_run_case1.log` appears to be a historical run log. It reports:

- case: `case1`
- accepted: `true`
- epochs: `1`
- final gate summary: spatial issues `0`, visual issues `0`
- total time: about `1227s`

That is useful as a "claimed working example", but it is not yet a local reproduction
in this project.

## Relationship To Existing Project Lines

| Existing line | Relationship |
| --- | --- |
| `20260617-vdiff-pdf-to-html-data-generation-loop` | Older VDiff workflow; PDF-to-HTML, spatial binding, calibration, validate, CSS polish |
| `20260622-minimal-multimodal-verification-loop` | Earlier agent harness comparison with direct/basic/in-loop/sidecar conditions |
| `20260627-pdf-image-to-html-core-verification-loop-v1` | Current V1 ladder with L0/L1/L2/L3/L8/L9 verification/control conditions |
| `20260701-vdiff-image2code-v2-harness-comparison` | New neutral comparison space for harness-level experiments |

## Open Questions

- Can `image2html_v2.2` run unchanged in the current Linux workspace, or does it carry
  path/model assumptions from its original environment?
- Does `claude-opus-4-8` map cleanly to the current local proxy model IDs?
- How stable is the model-derived grounding across repeated runs?
- How often does the hard gate reject due to grounding error rather than candidate error?
- Can the hard gate output be normalized into the V1 trajectory schema without losing
  epoch-level detail?
- Should V1 call this harness as an external condition, or should only its tool/gate
  abstractions be borrowed?

## Suggested Comparison Axes

For future multi-harness comparison, record each harness on these axes:

| Axis | Why it matters |
| --- | --- |
| Input modality | PDF, image, DOM, screenshot, or mixed |
| Ground truth source | PDF parser, image MLLM grounding, human annotation, model rubric |
| Agent controller | Our runner, Claude CLI, direct API, sidecar controller, external pipeline |
| Tool surface | Raw filesystem/browser tools vs named perception/actions |
| Verification timing | End-only, per round, per epoch, agent-invoked, controller-invoked |
| Acceptance oracle | Hard deterministic checks, MLLM visual judge, VDiff rubrics, human review |
| Artifact schema | Whether outputs can be placed into a common viewer/result table |
| Cost/time envelope | Per case latency, model calls, retries, manual dependencies |

## Recommended Next Step

Before any V1 coupling, do a standalone smoke run in a throwaway source mirror:

```bash
# future, not yet run in this audit
cd <unpacked>/VDiff-image2code-v2.2/image2html_v2.2
curl http://localhost:4141/v1/models
python run_one.py case1
```

Then compare the resulting `output/case1/result.json`, `render.png`, `epochs/`, and
gate failures against the historical `_run_case1.log`.