Coverage

The agentic coverage harness turns “review this repo for X” from an unauditable one-shot into a persistent, accumulating, comparable record of what an agent examined, for which concerns, and what it concluded.

What coverage is

When a model performs a survey task — “review this repo for vulnerabilities”, “find every place we touch the database”, “audit for accessibility” — findings vary run-to-run, and you can’t tell “no issues found” apart from “the model never looked.” The coverage harness fixes that by making coverage explicit and durable:

• Coverage is explicit. For every (file × concern) the agent assesses, it records a verdict — clean / finding / examined / not_applicable — with evidence. File reads, greps, and edits are tracked automatically.
• Runs accumulate. A second pass (same model or a different one) picks up where the first left off; cross-model verdicts are attributed and merge-able.
• Industry-anchored catalogs. Ship OWASP Top 10, CWE Top 25, the full CWE weakness list, STRIDE, secrets, and more; include or extend them.
• Surface-uniform. Works the same whether the agent loop is driven by rupu run, a workflow, an autoflow cycle, or an interactive session.

On top of that foundation, the harness also lets you measure and reproduce variance: diff two runs, and replay a run to compare it against the original.

The model

Coverage data for a target lives under <workspace>/.rupu/coverage/<target_id>/ as append-only JSONL plus a catalog snapshot. The <target_id> is derived deterministically from (workspace, scope_name), so the same agent against the same repo accumulates into one target across runs, while different workspaces stay distinct.

The ledger files under each target directory:

One file sits a level up, in <workspace>/.rupu/coverage/ itself, because it spans every target: finding_tags.jsonl, the workspace’s log of finding tag add/remove events (see tagging findings).

That per-row run_id + model + surface attribution is what makes cross-run and cross-model analysis possible.

Declaring coverage

An agent activates the harness by declaring a concerns: block in its frontmatter. The block is a list of catalog includes (with optional overrides, filters, and render mode). When present, the runtime flattens the catalog, snapshots it, injects the catalog into the system prompt, and registers the coverage tools.

---
name: security-assessor
permissionMode: readonly
concerns:
  - include: owasp-top10-2021
    mode: full
  - include: cwe-top25-2023
    mode: full
  - include: secrets-in-source
    mode: full
  - include: stride
    mode: full
  - include: cwe-software-development   # ~399 CWEs
    mode: index                          # one-line table, searched on demand
---
You are a security assessor. For each (file × concern) you assess, call
coverage_mark; for each issue, call report_finding…

Render modes. full inlines each concern’s body into the prompt; index renders a compact one-line-per-concern table that the agent searches on demand (use it for large catalogs like the full CWE list so the prompt stays small); auto (default) picks based on catalog size.

Coverage across a workflow

A workflow can declare the same concerns: block at its top level. It then overrides every step agent’s own block: each agent step runs the workflow’s catalog, and the harness is active on every step even when an agent declares no concerns of its own. Without a workflow-level block, each step uses its agent’s block, if any.

# .rupu/workflows/api-hardening.yaml
name: api-hardening
concerns:
  - include: owasp-api-top10-2023
steps:
  - id: sweep
    agent: route-sweeper
    actions: []
    prompt: "Assess every handler under api/."
  - id: verify
    agent: finding-verifier
    actions: []
    prompt: "Re-check each finding sweep reported."

Every step of a workflow writes to one target, derived from the workflow’s name rather than each step’s agent, so the ledger accumulates across steps and across runs of that workflow (an agentiflow-launched workflow unit uses the agentiflow’s target instead). File touches from workflow steps are attributed to the workflow surface. Use the workflow-level block whenever more than one step does coverage work, so the shared target has one catalog.

Bundled catalogs

rupu coverage templates list prints them all:

owasp-top10-2021        owasp-api-top10-2023
cwe-top25-2023          cwe-software-development   cwe-research
stride                  secrets-in-source          code-smells
web-security-default    api-security-default

User catalogs may live in .rupu/concerns/ (project) or ~/.rupu/concerns/ (global) and are discovered by name; project overrides global overrides builtin.

Coverage tools the agent gets

When concerns: is set, these tools are injected automatically — you do not list them in the agent’s tools::

Two more tools are not injected; grant them explicitly in the agent’s tools:: query_findings (list the workspace’s findings matching a findings query) and tag_findings (add or remove tags). Under an active engagement profile, asset_mark is offered as well. The findings.* MCP tools are never offered to an agent, even under tools: ["*"]; they serve workflow action: steps.

Running & auditing

All inspection commands take the global --format table|json|csv flag (table is the default; structured commands support table/json, tabular ones also csv).

rupu coverage list                          List targets under .rupu/coverage/
rupu coverage templates {list, show}        List bundled catalogs / print one's concerns
rupu coverage catalog <target>              Print the effective catalog snapshot
rupu coverage show <target>                 Derived view: files touched + assertions + findings
rupu coverage audit <target>                Full report: per-concern coverage, gaps, cross-model, findings
rupu coverage gap <target>                  Just the gaps (in-scope files lacking an assertion)
rupu coverage runs <target>                 List the runs recorded against a target
rupu coverage diff <target> [base compare]  What changed between two runs (defaults: previous latest)
rupu coverage rerun <target> <run_id>       Replay an agent run, appending a new run to the same target

Find a target id with rupu coverage list, then inspect:

rupu coverage audit a1b2c3d4e5f6          # human report
rupu --format json coverage audit a1b2c3d4e5f6   # machine-readable
rupu --format csv  coverage runs  a1b2c3d4e5f6

audit surfaces per-concern coverage, gaps, cross-model attribution, and serendipitous findings; the underlying findings and verdicts come straight from the target’s ledger. Because coverage is surface-uniform, runs dispatched through the control plane accumulate into the same target and show up in the same audit. Findings themselves have their own command, rupu findings (list, query, tag, export, import) — see Findings.

Remote units. Every rupu run also streams its coverage (runs, file touches, assertions, findings, assets, tags) to runs/<run_id>/coverage.jsonl. When a workflow unit runs on a remote host (host: / distribute:), that stream is delivered to the coordinator when the unit ends — on every transport — and merged under the coordinator’s own targets, so remote work shows up in the same audit. A stream that didn’t arrive or may be incomplete raises a step warning rather than failing the unit.

The rerun → diff loop

Model output varies run-to-run — sometimes usefully (different angles surface different bugs), but you could never evaluate the combination for completeness. Now you can:

rupu run security-assessor "assess this repo"   # run 1
rupu coverage runs <target>                      # grab run 1's id
rupu coverage rerun <target> <run_id>            # replay it → run 2, same target
rupu coverage diff <target> <run_id> latest      # what did run 2 do differently?

diff reports four dimensions: cell-coverage delta (newly / no-longer asserted), verdict flips (with clean → finding flagged), findings appeared / disappeared, and file-touch delta.

Determinism

Everything the harness controls about what the model sees — concern ordering, the catalog snapshot, and the live file list — is byte-stable and independent of the order catalog inputs are declared in. That makes the model the only source of run-to-run variance, which is exactly what diff measures. The harness does not claim byte-identical model output: prompt construction is deterministic; sampling is not.

Engagement profiles

Engagement profiles type a run’s findings by asset domain (code, binary, network, web, …): each finding is routed to the profile that owns its asset kind and gated on that profile’s completeness checks, and assets and their depth are recorded in assets.jsonl. They have their own page: Engagement profiles.