在关键操作附近主动取得证据
Agent 可以在 OCR、视觉定位、局部比较、render、交互状态变化或其他 consequential intermediate operation 后，调用隔离模型检查刚得到的结果。重点不在“调用次数”，而在是否选择了正确的时机、输入、问题和证据方法。
VeriHarness V0.1
frozen baseline general multimodal verification research harness项目定位
verification behavior first · image-to-code is the first task packageVeriHarness 是一个独立的 Claude Agent SDK 实验 harness，用来研究 agent 如何在工作过程中获得、判断和使用多模态证据。它的目标不是只为 image-to-code 增加一个终点检查器，而是把 verification 作为一种可观察、可消融、可训练的行为。Image-to-code 是 V0.1 的第一个可执行任务包，不是 harness 的能力边界。
在完成边界执行外部验收
Agent 提议完成后，harness-owned hard gate 检查当前 artifact。通过则 accept；存在 grounded mismatch 则返回修复反馈并进入下一 epoch；required evidence 不可用时按 infrastructure failure 处理。
V0.1 已冻结：本页保留实验基础设施、首个 task package、case1 管线审计与可审计 trajectory 的历史基线。新的研究变化与 harness 架构将在 VeriHarness V0.2 中重新讨论，不覆盖这里的分数和解释边界。
v-harness 来源关系
Native Claude v2.3 · vendored, not runtime-linked固定来源
V0.1 从 image2code-v2.3@e9c04d4b… 复制 gate、config、backends 和 13 个 seed cases，并在项目内保存独立 lineage。
关系边界 ·
vendored运行时不读取外部 VDiff repo 或旧 comparison project。vendored 表示有固定来源的本地派生，不表示整个 v2.3 harness 被直接嵌入。
四个实验条件
2×2 treatment matrix · H = harness gate · P = prompt policyHarness enforcement 与 Verification Policy Prompt 是两个独立实验轴。四组共享 runtime、任务、agent-visible tools、model profiles 和输入案例，因此这里比较的是 verification 行为如何被引入，而不是谁拥有更多工具。
H0 / H1完成边界是否执行 harness-owned hard gate
P0 / P1system prompt 是否要求模型主动验证 consequential claims
自然基线组
没有 hard gate，也没有主动 verification policy。模型仍看得到通用 invoke_model，可以自然选择是否使用。
- Harness gate
- off
- Prompt policy
- off
- Completion
- agent finish · 500-turn infrastructure guard
模型自发验证组
不强制终点 gate；system prompt 要求模型在高后果 claim 上主动选择 verification action。
- Harness gate
- off
- Prompt policy
- on
- Research signal
- spontaneous tool-driven verification
Harness 强制验证组
模型没有主动验证要求；完成提议必须经过 score-free hard gate，失败反馈驱动下一 epoch。
- Harness gate
- on
- Prompt policy
- off
- Budget
- 6 epochs · no semantic turn cap
组合验证组
模型可在 loop 内主动取证，同时完成边界仍由 harness gate 验收，用于研究两种行为是否互补或干扰。
- Harness gate
- on
- Prompt policy
- on
- Research signal
- self-verification + forced acceptance
共同控制：P0 中的 invoke_model 仍然可用，所以 P 轴测量的是使用 policy，而不是 tool availability。H0 的 max_turns=500 只是防失控 guard；H1 只用 max_epochs=6 控制外层循环。Online gate 向 agent 返回 grounded repair feedback，不输出 score。
Offline Evaluator · 四项结果测量
outcome measurement · runs after every agent stops下面四项不是另外四个实验组，而是所有条件共享的独立最终评估。Evaluator 在 agent 全部停止后运行，不控制 gate、不触发 repair、也不向任何 condition 暴露反馈，因此不属于 H1 treatment。
Canvas
检查最终 render 是否存在，以及 canvas 尺寸是否与 source 完全一致。
Text Geometry
把 source-grounded text 绑定到 candidate live DOM text，检查位置与尺寸。
Fresh Visual Bank
从 source 单独生成并冻结问题与 target answer，所有条件共享，用于检查语义、外观和视觉关系。
Structure Audit
审计 live text、absolute/fixed text、隐藏文本、截图替代和极端 inline style 等 anti-hack 属性。
Headline score
5 × weighted pass rate范围 0–5；严格通过线为 4.0。Unavailable dimension 会被明确记录，不能被默认为通过。Case1 · 冻结管线审计
historical run · not a general verification-effect result先看 validity，再看分数
这是单个 case1 的首轮管线审计。冻结 run 中，agent-facing router 把历史 kind:image descriptor 当成文本，P1 调用没有收到 image bytes。因此 H0P1 与 H1P1 不能用于判断 tool-driven multimodal verification。
| condition | Total | Canvas | Text Geometry | Fresh Visual | Structure | valid reading |
|---|---|---|---|---|---|---|
自然基线组H0P0 | 2.44 | 5.00 | 0.00 | 4.38 | 3.12 | pipeline baseline |
模型自发验证组H0P1 | 2.62 | 5.00 | 0.00 | 5.00 | 3.12 | P1 multimodal evidence invalid |
Harness 强制验证组H1P0 | 4.08 | 5.00 | 3.64 | 5.00 | 3.12 | positive evidence on case1 only |
组合验证组H1P1 | 3.26 | 5.00 | 1.59 | 5.00 | 3.12 | P1 multimodal evidence invalid |
两条 transport 必须分开解释。H1 private gate 使用另一条 path-to-data-URI transport，没有受到上述图片缺失 bug 影响；但 gate 仍可能产生语义错误或不一致判断。
Text Geometry 有 matcher caveat。当前 greedy one-to-one text binding 可能把标点、空格、换行或重复文本的绑定失败报告成 missing，不能把每个 item failure 直接解释成可见文字真的缺失。
当前最强结论仍然很窄。可以说 harness-only 在这个 case1 artifact 上得到正向证据；不能说 verification 已被证明普遍提升最终质量。
System Prompt · 模块与组合
source snapshot · English preserved verbatim
H 轴不会改变 agent system prompt；只有 P 轴决定是否插入 Verification Policy。下面展示四个版本化 source modules，不把几乎相同的 prompt 复制成四份。
P0 = Runtime Base
+ Model Invocation Policy
+ Task Profile
P1 = Runtime Base
+ Model Invocation Policy
+ Verification Policy
+ Task ProfileRuntime model profiles
vision_primary, vision_independentRuntime recipe catalog
background_inpaint_v1, grounded_ocr_v1, visual_align_v1, visual_compare_appearance_v1, visual_compare_spatial_v1, visual_locate_v1Model Invocation Policy 中的 {{MODEL_PROFILES}} 与 {{RECIPES}} 是 source template 占位符；runtime assembly 会用上面的 catalog 替换后再传给 agent。
Runtime Base · v1shared · prompts/runtime/base_v1.md
Complete source
# Runtime
You are an agent operating on artifacts in an isolated workspace.
- Use the available tools according to their schemas and preserve the user's requested output contract.
- Treat tool failures and invalid structured outputs as explicit observations. Do not invent missing results.
- Model calls receive only the context explicitly supplied to them.
- When the requested artifact is ready, call `propose_finish` with the current artifact references. If you end the turn without calling it, the runtime may treat the end-turn as an implicit finish proposal.
- Do not claim that an artifact was rendered, executed, tested, or observed unless that action actually occurred.Model Invocation Policy · v1shared · prompts/runtime/model_invocation_v1.md
Complete source template
# Model Invocation Policy
`invoke_model` calls a fresh isolated model. It receives only the instruction, declared output contract, and explicit inputs in that call. It does not inherit this transcript. Direct inspection by the main agent is not an isolated model call.
Available model profiles: {{MODEL_PROFILES}}.
Available recipes: {{RECIPES}}.
## Capability Selection
- `visual_locate`: enumerate every matching object or text region and return locations.
- `visual_inspect`: directly transcribe text or describe one specified object/region. OCR uses `visual_inspect` plus an OCR recipe; `ocr` is not a capability.
- `visual_align`: test whether one proposed text/object claim matches one localized region.
- `visual_compare`: compare explicitly labelled reference and candidate observations. Use only after a fresh candidate render exists.
- `image_generate` / `image_edit`: create an artifact; these are generative actions, not verification evidence.
- `custom`: use only when no standard capability expresses the requested observation.
Choose the narrowest capability that can answer the claim. Locate broadly, inspect or align a disputed region, and compare only when both observations are supplied.
## Input Contract
Image inputs use canonical `type` and a workspace-relative path:
```json
{"type":"image","path":"crop_terms.png","label":"source crop"}
```
Never pass an image as a path mentioned only in instruction text. Never use `/tmp` or a path outside the workspace; copy a crop into the workspace first. A comparison supplies both labelled images:
```json
[
{"type":"image","path":"source.png","label":"reference"},
{"type":"image","path":"render.png","label":"candidate"}
]
```
## Instruction Construction
Construct each verifier instruction as a small evidence contract:
1. **Claim and object**: name the exact object, region, behavior, or relation being checked.
2. **Observation and location**: say which labelled input to inspect and give a bbox/crop/landmark when useful.
3. **One observable question**: ask one falsifiable property per item. Avoid broad requests such as "is the page good?".
4. **Method boundary**: say what must be read, measured, localized, aligned, or compared; state what may be ignored.
5. **Direct result**: require the observation/verdict now, never a plan for future inspection.
6. **Uncertainty rule**: if the evidence is unreadable, absent, ambiguous, or unavailable, return `insufficient` and say what crop/view would resolve it. Do not guess.
7. **Output contract**: choose `json` for machine-used evidence and provide a precise JSON Schema; use `text` only for genuinely free-form observations.
For source/candidate comparison, ask a neutral property and keep polarity explicit: identify `expected` from the reference, `observed` in the candidate, and whether they match. Do not infer the reference answer from the candidate. Separate spatial, text/content, and appearance questions when their evidence methods differ.
For OCR, request verbatim text, reading order, line breaks, punctuation, line/region bboxes, and explicit ambiguous characters. Prefer a focused high-resolution crop over whole-page OCR when exact text matters.
Use this reusable meta-template; fill every bracket that matters and delete irrelevant lines:
```text
Task: [locate | inspect | align | compare] one observable claim.
Inputs: [label -> role], including the exact crop/bbox/landmark when available.
Object: [unique object name and page region].
Question: [one neutral, falsifiable property].
Method: [what to transcribe/localize/measure/compare] and [what to ignore].
Return now: [direct observation plus expected/observed/match when comparing].
Uncertainty: return insufficient with a reason and the next required view; never guess.
```
The system prompt governs how the agent frames a model call. The harness independently
enforces that declared images are attached, the output contract is appended, paths stay
inside the workspace, and JSON satisfies its schema. Neither layer substitutes for the
other.
## Result Admission
A successful tool return is not automatically evidence. Before using it, check:
- the model actually observed every required labelled input;
- the response answers the requested question rather than promising to inspect later;
- the result is relevant to the named object and location;
- uncertainty is represented rather than invented detail;
- structured output satisfies the declared schema;
- consequential claims have enough independence or corroboration for their risk.
Treat placeholder complaints, unrelated OCR, plan-only responses, missing verdicts, and schema-valid but irrelevant content as `insufficient`. Refine the crop/instruction or choose a different evidence method; do not adopt them as artifact facts.
## Observation Identity And Freshness
Each successful call returns a harness-generated `call_id`. The runtime privately records
the exact input hashes and the artifact version that existed at call time. You do not need
to cite or manage evidence IDs.
A model result describes only the inputs it actually observed. After changing an artifact,
do not treat an earlier candidate render or candidate crop as proof of the new version;
create a fresh observation when that claim matters. Source/reference observations remain
reusable only while their recorded inputs are unchanged.
Call fields are `capability`, `recipe`, `instruction`, `inputs`, `output_kind`, `output_schema`, and `model_profile`.Verification Policy · v1P1 treatment only · prompts/verification/policy_v1.md
Complete source
# Verification Policy
Treat every consequential belief about an artifact as a claim that may be wrong. Before relying on it, seek fresh, relevant, and sufficiently independent evidence.
## Philosophy
- Evidence over confidence: certainty is not evidence.
- Verify by consequence: spend verification effort where an error would matter.
- Observe the artifact: test the rendered or executed result, not your implementation intent.
- Preserve uncertainty: conflicting or missing evidence remains uncertain.
- Completion requires evidence: writing the artifact does not prove the goal is satisfied.
## Decision Protocol
At an evidence-use, observable-checkpoint, or acceptance boundary:
1. Identify the claim and the consequence of being wrong.
2. Choose `check`, `discover`, `defer`, or `skip`.
3. Select a capability, recipe, explicit inputs, and a precise instruction.
4. Use normalized feedback to repair, gather discriminating evidence, or continue.
5. Propose finish only when the current artifact has adequate fresh evidence.
Do not verify mechanically. Defer when an artifact is still intentionally incomplete. Skip when existing evidence is fresh and sufficient. Use focused multi-view questions for consequential claims; seek an independent source when correlated model errors remain plausible.
## Isolated Verifier Action
Direct inspection by the main agent, deterministic measurements, and an isolated model call are different evidence sources. Do not describe the first two as independent model verification.
When a task contains a consequential perceptual or semantic claim that a configured model can check, make at least one explicit `invoke_model` call before the first finish proposal. Choose the timing, claim, capability, recipe, inputs, and instruction according to expected information gain. This is a minimum opportunity to exercise isolated verification, not a requirement to call a model after every action.
For visual artifact work, useful choices include grounded source extraction before relying on uncertain text, a focused source/render comparison after a consequential render, or an independent check of a disputed region. Prefer a precise question over a generic request to judge the whole artifact.
## Examples
- A grounded OCR result will determine the page title and layout: check before adopting it.
- Several edits are incomplete and no new render exists: defer.
- Only the title region changed: check title-related claims rather than repeating a full-page audit.
- Fresh independent evidence already agrees: skip another redundant call.
- A click should change UI state: observe the before/after state and check the behavior claim.Image-to-Code Task Profile · v1shared task package · prompts/tasks/image_to_code_v1.md
Complete source
# Image To Code
Create a faithful, self-contained, live, and editable web artifact from the provided reference image.
- Reproduce visible content, presence, geometry, appearance, and structure.
- Build primary layout with normal flow, flex, grid, padding, margins, and gaps.
- Do not pin readable content to screenshot coordinates with widespread absolute positioning. Use absolute positioning only for elements whose real semantics are overlays or deliberate overlap.
- Separate base background, visual assets, and live content. Readable text and controls remain live content; do not bake them into crops or generated images.
- Do not use the reference image as the page or a large background substitute.
- Keep every dependency inside the workspace. Do not load remote images, fonts, scripts, or styles.
- The entrypoint is `answer.html`, which must open directly in the configured browser environment.设计理念
why these boundaries existVerification 是行为，而不只是终点 gate
验证可以发生在采用 OCR 之前、一次 consequential render 之后、争议区域附近、交互状态变化之后或最终验收时。研究对象是模型何时以及如何取得证据，而不是机械地增加一次 evaluator call。
Prompt policy 与 harness enforcement 正交
Prompt 教模型识别 consequential claim、选择证据方法并构造精确 instruction。Harness 负责 prompt 无法保证的事实：隔离上下文、真实图像字节、workspace path、output contract、schema、retry 与 terminal failure。
Evidence admission 先于答案质量
一个输出即使 JSON 合法、文字流畅，若 required input 没有成功附加并被观察，或者输出只承诺稍后检查、回答了无关 OCR、缺少 verdict，它就不能支持 artifact claim。
Online feedback 与 offline scoring 分离
Online verifier 面向修复，只给 observation 和 mismatch；offline evaluator 面向实验比较，只在 agent 停止后衡量最终 artifact。这样 outcome metric 不会悄悄变成 H1 loop 内的隐藏优化目标。
Required verification 必须 fail closed
H1 的 required component unavailable 时，harness 在原 gate attempt 层重试。重试耗尽后以 infrastructure failure 结束，既不能 accept，也不能伪装成 artifact mismatch 去错误地 reopen agent。
Freshness 是 provenance 属性
每个 model observation 都关联 exact input hash 与调用时 artifact version。Source 没变时 observation 可以复用；artifact 一旦修改，旧 candidate render 就不能继续证明新版本。
Transcript 与结构化 event trace 分工
Transcript 保存模型交互；event trace 保存 attempt、normalized observation、artifact version、gate decision 和 provenance。二者分开，才能离线分析 verification episode，而不要求 agent 自报 bookkeeping。
Accumulating gate 必须可审计
H1 以最新 accumulated claim bank 作为当前验收标准，但每次 attempt 的 artifact、render、完整 bank、新增 claims 与 feedback 都不可覆盖地保存，避免事后把最终 contract 投射到早期 epoch。
通用 core 与任务行为分开
模型调用、隔离、evidence contract、event 和失败语义属于 generic core。HTML、DOM、Playwright、text geometry 与 absolute-position 规则属于 image-to-code task package，不能定义未来所有任务。
Harness trajectory 是能力内化的起点
H1 可以产生显式 verification-and-repair trajectory，P1 用来观察模型能否在没有强制 gate 时选择相似行为。未来可筛选高质量 verification episode 做 cold start 与 RL；V0.1 只建立数据和因果边界，不宣称已经完成内化。