Netflow

rupu records where it reaches out to, not just what an agent does — every outbound HTTP call rupu itself makes becomes a flow, written to an append-only ledger, enriched with ASN and org, and rendered as a topology in the Control Plane.

What it is

rupu already has deep observability for what an agent does — tool calls, command runs, tool audits, coverage ledgers, findings, usage accounting. Netflow closes the other half: where an agent reaches. Every request rupu's own HTTP clients send — model provider APIs and SCM / issue connectors, plus the control plane's own fleet traffic and its auth / system calls — is captured as a FlowRecord: method, host, port, query-stripped path, peer IP, status, byte counts and timings, tagged with the run, step and agent that produced it.

Those records land in an append-only JSONL ledger next to the run store, and — for flows observed during a run — as Event::NetFlow lines in that run's transcript. At read time the control plane resolves each peer IP to an ASN and org and paints a bipartite topology graph, a sortable flow table, and a per-host rollup.

The whole subsystem exists to let a viewer tell "we could not observe this" apart from "it was zero". That distinction is enforced end to end: an unknown byte count is never summed as 0, an unmeasured percentile is never rendered as 0 ms, a dropped record is never silently absorbed, and every row carries a fidelity badge saying how much of it is actually known.

The choke point

Capture only works if there is exactly one door. Before netflow there wasn't one — roughly thirty reqwest::Client::new() / Client::builder() sites were scattered across six crates. Netflow created the choke point and then nailed it shut: a clippy.toml lint denies reqwest::Client::new, Client::default, ClientBuilder::new, ClientBuilder::default and — the actual bypass point — ClientBuilder::build everywhere except inside the netflow crate itself, backed by a repo-level test. A regression is a build failure, not a review miss.

Attribution is bound when a client is constructed, not discovered at request time. Task-locals and tracing spans both lose context across a tokio::spawn, which would produce flows with silently missing attribution — worse than none. Provider and SCM clients are already built per run, so binding the context there costs no new plumbing. Process-global clients (the control plane's host registry, the ASN refresh) simply carry no run id — that is the honest shape of the data, and precisely why the ledger exists alongside the transcript events rather than being a convenience index.

What a flow records

One FlowRecord is one outbound request. Fields typed as optional are genuinely unknown when absent — they are never defaulted to zero:

Read back through the control plane, each flow gains one more field — asn — resolved at render time rather than stamped at write time (see ASN enrichment below).

Deliberate omissions

• Query strings and headers are never stored. They routinely carry tokens. The path is stored query-stripped, and there is no opt-out.
• No TLS version or cipher. The HTTP layer doesn't expose it — a field that could only ever be empty doesn't belong in the schema.
• No ASN on the record. Stamping it at write time would freeze history against whatever dataset happened to be on disk that day.

Fidelity — the honesty field

Every flow declares how much of itself is real, and every CP view renders the badge. The subsystem never claims coverage it does not have.

Streaming bodies

Every provider chat path is a streamed response, where Content-Length doesn't exist. Netflow does not estimate. The middleware emits the record at response-header time with an unknown bytes_in and body_complete: false; the stream-consuming loop — which already counts bytes — later calls complete(flow_id, bytes_in, duration_ms) to finalize it. The caller minted the flow id and attached it to the request, so it knows exactly which record to close: explicit, no ambient magic.

The ledger

Flows are written to an append-only JSONL ledger. There are two, and which one a flow lands in depends on who produced it:

The ledger is strictly append-only — a record written at header time is never rewritten. Each line is one of three envelopes:

Reads tolerate damage: a missing file is an empty ledger, not an error, and a malformed line is skipped so a torn write at the tail can't lose the whole history.

How flows tie back to runs

For flows observed while a run is active, netflow also appends an Event::NetFlow line to that run's transcript JSONL — so they appear live wherever transcripts are already tailed, with no new plumbing. That's not redundancy with the ledger; it's what makes run scope correct. Only provider flows carry a run id: SCM, auth, system and update flows are run-less by design, so filtering the ledger by run id alone would silently drop every connector call the process made while the run was active. The run's Network tab therefore merges both sources.

Flows are transcript events, not workflow-executor events — they do not appear in events.jsonl, which is a step-level lifecycle log where a per-request record has no business.

In the Control Plane

Netflow follows the same placement rule as findings: a tab on a run, a tab on a project, and a global page — never on a workflow definition. A flow belongs to a run, not to a workflow.

Behind those surfaces sit four read-only endpoints:

# run, project, and global scopes
GET /api/runs/:id/netflow
GET /api/projects/:id/netflow
GET /api/netflow

# precomputed topology; scope is run:<id>, project:<id>, or absent for global
GET /api/netflow/graph?scope=run:01J9Z4W7Q0X8Y6V5K3M2N1P0R8

The topology graph

The primary visualization is a bipartite topology graph: sources on the left — one node per run, plus a system node for unattributed process-global egress — and endpoints on the right, one per host:port. Edges connect who reached what, thickness scaled by call count, and an edge that carried any failure is drawn in the same red the run graph uses to mark a problem.

The flow table

A sortable, per-flow list — time, origin, host, path, network (ASN and org), outcome, bytes and timings — with a fidelity badge on every row, so a coarse row never looks as complete as an instrumented one. Three states are called out rather than glossed:

• Dropped records get a loud banner — "N flows dropped — the capture buffer overflowed, so this list is incomplete." It renders even when the table is empty, because a scope where everything was dropped is exactly where silent incompleteness does the most damage.
• Unavailable ASN enrichment gets its own note, so a blank Network column reads as "we couldn't look it up" rather than "this peer has no ASN."
• An empty table restates netflow's scope limit, so "no rows" never implies "no network activity happened."

The summary

A per-endpoint rollup — one row per host:port with calls, errors, bytes in / out, p50 and p95 latency — computed server-side so the percentile and unknown-bytes rules have exactly one implementation. Both rules are strict: if any contributing flow had an unobservable byte count, that total collapses to unknown rather than under-reporting (and in/out are tracked independently, so an in-flight stream doesn't blank a known request total). If no flow had a known duration, the percentile is unknown — because a displayed 0 ms can't be told apart from a genuine sub-millisecond p50.

The fidelity badge and the scope disclosure

Two small components carry most of netflow's integrity. The fidelity badge is rendered next to every flow and explains itself on hover — coarse means host, outcome and timing are real but bytes and peer IP were not observable for that connector. The scope disclosure is a single authored sentence, rendered on all three Network surfaces, naming exactly what netflow covers at that scope and what it cannot see. It is scope-aware on purpose: CP fleet traffic is only claimed on the global page, because that's the only place it can actually appear. Every scope-limit sentence in the CP — including the table's and the graph's empty states — draws from that one source, so the wording can't drift apart into claims the code doesn't back.

ASN enrichment

A peer IP on its own is nearly unreadable. Netflow resolves it to an autonomous system number and org — so a flow reads AS13335 Cloudflare rather than a bare address — and that is the enrichment the flow table's Network column shows.

The operator never runs a command for this. The dataset is acquired and refreshed automatically from a combined IPv4+IPv6 prefix→ASN table, compacted on ingest into a sorted binary range table at ~/.rupu/netflow/asn.db (v4 and v6 held separately, binary-searched; unrouted rows skipped). It isn't bundled with the binary — compacted it's meaningful bloat on a binary that already embeds the CP's web assets. Two triggers keep it current:

• The rupu cp serve sweep loop gains an ASN freshness tick, alongside the gate sweep and the cron tick.
• Any netflow read that finds the table missing or stale spawns a detached background fetch — so operators who never run cp serve still get enrichment. A burst of requests collapses to a single download via a process-wide single-flight guard.

Enrichment is resolved at read time, never stamped onto the record. Three consequences fall out of that, and they're the reason for the design:

• A dataset that arrives late automatically improves every historical record. No backfill job exists, or is needed.
• The write path never blocks on a file that may not be there.
• Offline and air-gapped installs degrade to "ASN data not loaded" while every other view works unchanged.

Refresh is best-effort and fails safe: on any failure the existing table is left untouched, and a download that parses to an empty table (a gzipped error page, say) is refused rather than allowed to destroy working enrichment. The parsed table is cached in-process and keyed on the file's mtime, so three Network tabs opening at once don't reparse hundreds of thousands of ranges three times. And the refresh request is itself recorded as a system flow — the subsystem stays honest about its own egress.

Configuration

Netflow capture itself needs no configuration — it's on wherever a sink is installed. The [netflow] section governs one thing: automatic acquisition and refresh of the ASN table.

Unknown keys in this section are rejected, and any key you leave out takes its default:

# ~/.rupu/config.toml — every [netflow] key, at its default
[netflow]
asn_auto_refresh = true
asn_refresh_interval_days = 7
asn_source_url = "https://iptoasn.com/data/ip2asn-combined.tsv.gz"

# air-gapped: keep capture, skip the download entirely
[netflow]
asn_auto_refresh = false

Telemetry only

Netflow is pure telemetry. A flow carries no severity, no triage state, and no coupling to the findings subsystem. Anomalous egress does not become a finding, and nothing in netflow decides what "anomalous" means.

That's a deliberate decision, not an omission. Judgement baked into a recording layer is judgement you can't revise without rewriting history — and a record that carries a verdict stops being evidence. Keeping the data inert means the flows stay exactly as trustworthy as their fidelity field claims, and nothing more. Should anomaly surfacing be wanted later, it lands as an additive detector sitting on top of this data, reading the same ledger and emitting findings of its own — not as a change to what a flow is.

The same restraint applies to enforcement: netflow observes egress, it does not block it. Deny-by-default egress policy needs the isolated-runtime substrate that the full-fidelity backend would bring, and is out of scope until then.

Still coming

Everything above ships today. What follows genuinely does not, and the page says so rather than implying otherwise: