# VeriHarness v0.5 — Round 1 Results (baselines + P0)

Run date: 2026-07-23. Two generator models, run serially via the local 4151
proxy (opus native; gpt-5.6 via the 4153 Anthropic→responses adapter).
Each model: 3 conditions × 8 cases × 2 replicates = 48 cells.
Offline scorer: gemini-3.1-pro-preview, frozen gold+discovery rubric bank,
per-case normalized, score = pass_rate × 5. No feedback to generation.

Runs: `runs/v05-r1-opus`, `runs/v05-r1-gpt`
(offline: `offline-evaluations/offline-gemini-v1/summary.json`).

## Mean score by condition (0–5)

| condition | opus mean | opus pass% | gpt-5.6 mean | gpt-5.6 pass% |
| --- | --- | --- | --- | --- |
| A-raw (no tool/MCP)          | 4.016 | 80.3% | 3.932 | 78.6% |
| B-base (tool + CallMLLM)     | 4.276 | 85.5% | 4.374 | 87.5% |
| P0 (B-base + "you may verify")| 4.280 | 85.6% | 4.444 | 88.9% |

## Deltas vs A-raw (score points)

| step | opus | gpt-5.6 |
| --- | --- | --- |
| B-base − A-raw | +0.259 | +0.442 |
| P0 − A-raw     | +0.264 | +0.512 |
| P0 − B-base    | +0.004 | +0.070 |

## Observations (Round 1, within-set only)

- **The biggest jump is A-raw → B-base**, consistent across both models
  (+0.26 opus, +0.44 gpt). Giving the agent the CallMLLM tool + tool prompt
  (i.e. access to an external multimodal verifier) lifts fidelity the most.
- **P0 (the minimal "you may verify" reminder) adds little on top of B-base**:
  +0.004 opus, +0.070 gpt. The permissive one-liner barely moves the needle —
  the tool's *availability* (B-base) already captures most of the effect; a bare
  reminder to use it is near-nil for opus and small for gpt.
- **Direction is consistent across models** (A-raw < B-base ≤ P0 for both), which
  is what this dual-model design was meant to check. Magnitudes differ: gpt-5.6
  responds more to both the tool and the reminder than opus does.
- Caveat: this is within-set on 8 cases × 2 replicates, no significance testing
  yet, single round. Not a cross-case generalization claim.

## What this suggests for the next verification step (P1+)

P0's near-zero increment over B-base is itself the key finding: a bare "you may
verify" is too weak to change behavior beyond just having the tool. The next
conditions should test whether *stronger/more specific* verification prompting
(e.g. telling it what to check, or to actually run a compare-and-fix loop)
produces a real increment over B-base — that's the open question P1 should target.

## Run infrastructure (Round 1) and next-round plan

Round 1 ran opus and gpt-5.6 SERIALLY, both via the local 4151 proxy:
- opus: native Anthropic transport to 4151.
- gpt-5.6: local `scripts/anthropic_to_responses_proxy.py` on :4153, translating
  Anthropic `/v1/messages` → 4151 `/v1/responses`. (Used only because the TRAPI
  device-code login failed that night; the 4151 side saw ~25×429 rate-limit
  retries, all auto-retried.)

**TRAPI is actually available** (verified 2026-07-24). The blocker was only that
the default az identity is a no-permission systemAssignedIdentity. The az cache
already holds the SC user account `sc-vax904199@microsoft.com`; selecting a
`user.type=='user'` subscription (no interactive login) yields a working
`api://trapi/.default` token. See the ready skill `4-Skills/use-trapi-gpt56/`.
Confirmed: `gcr/shared` deployments `gpt-5.6-{sol,luna,terra}_2026-07-09`
(rpm=850, tpm=850000), chat needs `max_completion_tokens` (reasoning model).

**Next round can run opus+gpt in TRUE PARALLEL** (physically separate backends):
gpt → TRAPI `gcr/shared` (ample rpm), opus+gemini → local 4151 (workers 16).
This halves wall-clock vs the Round-1 serial-local approach.