Netflow

rupu records where it reaches out to, not just what an agent does — every call rupu makes to a model provider or SCM, and every connection an agent's bash commands open, becomes a flow, written to a per-run ledger, enriched with ASN and org, and explored in the Control Plane.

What it is

rupu already has deep observability for what an agent does — tool calls, command runs, tool audits, coverage ledgers, findings, usage accounting. Netflow closes the other half: where an agent reaches. It has two sources. Every request rupu's own HTTP clients send on an agent's behalf — model provider APIs and SCM / issue connectors — is captured as a FlowRecord: method, host, port, query-stripped path, peer IP, status, byte counts and timings. And every TCP or UDP connection opened by a process the agent started through its bash tool — a curl, a git fetch, a pip install, a scanner — is captured from the operating system's own socket table, with the process that owned it. Both are tagged with the run and agent that produced them.

Those records land in an append-only JSONL ledger per run. At read time the control plane resolves each peer IP to an ASN and org and renders a Network explorer — activity over time, a topology from workflows to origins to the orgs they reached, per-endpoint timelines, and a sortable flow table — at run, project and global scope. A unit that ran on a remote host sends its ledgers back when it ends, so they are read at every scope too.

The whole subsystem exists to let a viewer tell "we could not observe this" apart from "it was zero". That distinction is enforced end to end: an unknown byte count is never summed as 0, an unmeasured percentile is never rendered as 0 ms, a dropped record is never silently absorbed, and every row carries a fidelity badge saying how much of it is actually known.

The choke point

Capture only works if there is exactly one door. Before netflow there wasn't one — roughly thirty reqwest::Client::new() / Client::builder() sites were scattered across six crates. Netflow created the choke point and then nailed it shut: a clippy.toml lint denies reqwest::Client::new, Client::default, ClientBuilder::new, ClientBuilder::default and — the actual bypass point — ClientBuilder::build everywhere except inside the netflow crate itself, backed by a repo-level test. A regression is a build failure, not a review miss.

Attribution is bound when a client is constructed, not discovered at request time. Task-locals and tracing spans both lose context across a tokio::spawn, which would produce flows with silently missing attribution — worse than none. Provider and SCM clients are already built per run, so binding the context there costs no new plumbing. Process-global clients with no run behind them (the control plane's host registry, the ASN refresh, the updater, login) are wired to a null sink and record nothing: a flow with no run has no ledger to land in, and pretending otherwise would scatter unattributable rows across every view.

Bash subprocess connections

An agent's bash tool spawns ordinary processes, and those don't go through rupu's HTTP client. Rather than put a proxy in their path or require root, netflow reads the socket accounting the kernel already keeps — the same data nettop and ss show — and attributes each socket to the tool call whose processes opened it. Nothing sits in the command's network path: its environment, privileges and performance are unchanged, and raw-socket tools keep working.

Each connection becomes one flow at socket fidelity: transport, remote IP and port, local address, the process name and pid, open and close times, and bytes in and out where the OS reports them. It carries the tool-call id of the bash call that caused it, so the CP's flow table links straight to that call in the run's transcript. There is no URL, method or status — those live inside the encrypted stream, and the views show a dash rather than inventing them.

What a flow records

One FlowRecord is one outbound request. Fields typed as optional are genuinely unknown when absent — they are never defaulted to zero:

Read back through the control plane, each flow gains one more field — asn — resolved at render time rather than stamped at write time (see ASN enrichment below).

Deliberate omissions

• Query strings and headers are never stored. They routinely carry tokens. The path is stored query-stripped, and there is no opt-out.
• No TLS version or cipher. The HTTP layer doesn't expose it — a field that could only ever be empty doesn't belong in the schema.
• No ASN on the record. Stamping it at write time would freeze history against whatever dataset happened to be on disk that day.

Fidelity — the honesty field

Every flow declares how much of itself is real, and every CP view renders the badge. The subsystem never claims coverage it does not have.

Streaming bodies

Every provider chat path is a streamed response, where Content-Length doesn't exist. Netflow does not estimate. The middleware emits the record at response-header time with an unknown bytes_in and body_complete: false; the stream-consuming loop — which already counts bytes — later calls complete(flow_id, bytes_in, duration_ms) to finalize it. The caller minted the flow id and attached it to the request, so it knows exactly which record to close: explicit, no ambient magic.

The ledger

Every run, workflow step, fan-out unit, sub-agent and session turn gets its own append-only ledger:

The ledger is strictly append-only — a record written at header time is never rewritten. Each line is one of six envelopes:

Reads tolerate damage: a missing file is an empty ledger, not an error, and a malformed line is skipped so a torn write at the tail can't lose the whole history. Flows observed during a run are also written into that run's transcript as net_flow events, which the CLI's transcript views render inline.

When a run ends, every flow its transcript holds and its ledger lacks (the writer could not open the ledger, or dropped the line) is appended to the ledger, followed by any complete / socket_complete line the ledger already had for it and a recovered line. A run that crashed or was killed never reaches its end: its own view, and its agentiflow's, still merge the transcripts of the run and of every run it launched, while project and global views show only what reached the ledger.

Runs on remote hosts

A unit placed on a remote host (host:, distribute:, the fallback-host retry) and a run placed there whole write their ledgers on that host. The host also tees every ledger line of the run, and of every sub-agent and child beneath it, into one stream in the run's directory on the host (runs/<id>/netflow.jsonl), which starts with a begin line even when the run made no network calls — so "no flows" is never confused with "not collected". When the run ends, the coordinator collects the stream and lands it: split per ledger id into its own ~/.rupu/netflow/<id>.jsonl, byte for byte, with a landing record, ~/.rupu/runs/<id>/netflow.landing.json, that says what arrived: complete, partial (with the reason) or unavailable (a host too old to stream, or a stream that failed or was lost). A complete record is never downgraded by a later landing.

Nothing lands while the unit is still running: until it ends, its flows show only through the transcript merge above. Once landed, the ledgers are read like any other at run, project, global and agentiflow scope, and a copy that is partial, unavailable or not collected at all is listed as incomplete at every scope, naming the host and the reason; the workflow run also gets a step warning for it. A missing or short copy never fails the unit and is never shown as whole. rupu netflow prune treats landed ledgers like any other ledger.

CLI

rupu netflow show run_01J...                     # one run's flows
rupu --format json netflow show run_01J...       # full records, plus dropped_total
rupu netflow prune --older-than 14d --dry-run    # preview
rupu netflow prune --older-than 14d              # delete

rupu netflow show <run-id> prints the run's flows (time, method, host, path, status), merging its ledger with the net_flow events in its transcript, and reports any dropped records. It finds the run in the global run store or in any registered project.

Ledgers are kept until you prune them; nothing deletes them automatically. rupu netflow prune deletes ledgers older than --older-than (default 30d) from the current project's .rupu/netflow/ and the global ~/.rupu/netflow/. It goes by file modification time, never touches a file modified in the last hour, and rejects a zero or negative cutoff. A ledger whose run is still live is never deleted, however long the run has been idle: an unfinished workflow run (paused and awaiting-approval runs included, unless it is running/pending with a dead recorded runner pid) keeps its own, its steps', its fan-out units' and its sub-agents' ledgers, and a standalone rupu run whose process is alive keeps its own and its sub-agents'. Those rows report skipped_live. --dry-run previews, and it supports --format json|csv.

In the Control Plane

Netflow follows the same placement rule as findings: a tab on a run, a tab on a project, and a global page — never on a workflow definition. A flow belongs to a run, not to a workflow. All three render the same Network explorer; the scope only narrows the data.

Ledgers landed from a remote host (Runs on remote hosts above) are read at every scope like any other; a copy that is partial or missing is listed as incomplete, with the host and the reason.

The explorer

• Query line. The one place to narrow the view. since: and until: set the window (since:6h, since:2026-10-01, an RFC 3339 time), window:all reads every recorded flow, and workflow:, origin: (provider:anthropic, scm:github, subprocess:curl), org: (as15169, unknown) and host: (api.github.com:443) keep flows; a - in front leaves them out (-origin:provider:anthropic hides the model traffic). With no since: or until: the view opens small, on the window the table above names, and a line under the query says which window that is.
• Activity strip. Across the whole retained range; drag across it to zoom, which writes the window into the query (Zoom out undoes it).
• KPIs. Flows, endpoints, networks (distinct orgs), error rate, bytes and p95 latency. Byte totals are marked partial when any flow in view had an unknown count; a p95 with no measured duration is a dash, never 0 ms.
• Topology. Workflows → origins (a provider, an SCM, or a subprocess such as curl) → the ASN orgs they reached, weighted by call count.
• Timeline. One lane per endpoint, grouped by org.
• Click to filter. Click a workflow, origin, org or host to add it to the query; click it again to take it out.
• Flow table. Sortable, with a detail panel per flow. Socket rows show the process (curl (pid 4412)), the transport and peer (tcp → 203.0.113.7:443), a dash for path and status, and a link to the bash call in the run's transcript.

Honesty on demand

A coverage popover on every Network surface carries the one authored sentence that says what netflow covers at that scope and what it cannot see, the fidelity legend, and the dropped-record accounting — and, at run scope, whether subprocess capture was active or why it wasn't. Every scope-limit sentence in the CP, including the empty states, draws from that one source, so the wording can't drift into claims the code doesn't back. Three states are called out rather than glossed: dropped records ("N flows dropped — the capture buffer overflowed", shown even on an empty table, where silent incompleteness does the most damage), unavailable ASN enrichment (so a blank Network column reads as "couldn't look it up", not "no ASN"), and an empty table, which restates the scope limit so "no rows" never implies "no network activity".

Loading large histories

rupu cp serve keeps a netflow index of every ledger, warmed when the server starts. It notices changed files with a cheap stat instead of re-opening them, reads only the lines appended since last time, and keeps a small summary of every file (time span, counts, origins) in memory always, plus compact flow rows up to a memory budget. Over the budget, the rows of the files with the oldest flows are dropped and re-read from disk when a view needs them — a wide window gets slower, but the answer never changes. Each section of a Network page loads on its own, so the KPIs and topology appear before a long flow list finishes, and long tables are virtualized.

Behind those surfaces sit these read-only endpoints:

# flows at run, project and global scope (?from= / ?to= to window the run's)
GET /api/runs/:id/netflow
GET /api/projects/:id/netflow
GET /api/netflow

# the explorer's aggregates; scope is run:<id>, project:<id>, or absent for global
GET /api/netflow/explorer?scope=project:ws_01JB7…

# the topology graph
GET /api/netflow/graph

# the index: its size, budget and evictions
GET /api/netflow/index

# the latest run with flows (no scope = global)
GET /api/netflow/latest?scope=project:ws_01JB7…

Every read but the graph takes workflow, origin, org and host (keys to keep) and not_workflow, not_origin, not_org and not_host (keys to leave out), each a comma-separated list.

ASN enrichment

A peer IP on its own is nearly unreadable. Netflow resolves it to an autonomous system number and org — so a flow reads AS13335 Cloudflare rather than a bare address — and that is the enrichment the flow table's Network column shows.

The operator never runs a command for this. The dataset is acquired and refreshed automatically from a combined IPv4+IPv6 prefix→ASN table, compacted on ingest into a sorted binary range table at ~/.rupu/netflow/asn.db (v4 and v6 held separately, binary-searched; unrouted rows skipped). It isn't bundled with the binary — compacted it's meaningful bloat on a binary that already embeds the CP's web assets. Two triggers keep it current:

• The rupu cp serve sweep loop gains an ASN freshness tick, alongside the gate sweep and the cron tick.
• Any netflow read that finds the table missing or stale spawns a detached background fetch — so operators who never run cp serve still get enrichment. A burst of requests collapses to a single download via a process-wide single-flight guard.

Enrichment is resolved at read time, never stamped onto the record. Three consequences fall out of that, and they're the reason for the design:

• A dataset that arrives late automatically improves every historical record. No backfill job exists, or is needed.
• The write path never blocks on a file that may not be there.
• Offline and air-gapped installs degrade to "ASN data not loaded" while every other view works unchanged.

Refresh is best-effort and fails safe: on any failure the existing table is left untouched, and a download that parses to an empty table (a gzipped error page, say) is refused rather than allowed to destroy working enrichment. The parsed table is cached in-process and keyed on the file's mtime, so three Network tabs opening at once don't reparse hundreds of thousands of ranges three times. The download itself has no run behind it, so — like the rest of rupu's housekeeping — it is not recorded as a flow.

Configuration

Netflow capture needs no configuration — HTTP capture is on wherever an agent runs, and subprocess capture is on wherever the OS allows it. The [netflow] section tunes subprocess capture, the control plane's index, and the ASN table.

Unknown keys in this section are rejected, and any key you leave out takes its default:

# ~/.rupu/config.toml — every [netflow] key, at its default
[netflow]
subprocess_capture = true
subprocess_poll_ms = 50
subprocess_linger_ms = 3000
cp_index_budget_mb = 256
asn_auto_refresh = true
asn_refresh_interval_days = 7
asn_source_url = "https://iptoasn.com/data/ip2asn-combined.tsv.gz"

# air-gapped: keep capture, skip the download entirely
[netflow]
asn_auto_refresh = false

Telemetry only

Netflow is pure telemetry. A flow carries no severity, no triage state, and no coupling to the findings subsystem. Anomalous egress does not become a finding, and nothing in netflow decides what "anomalous" means.

That's a deliberate decision, not an omission. Judgement baked into a recording layer is judgement you can't revise without rewriting history — and a record that carries a verdict stops being evidence. Keeping the data inert means the flows stay exactly as trustworthy as their fidelity field claims, and nothing more. Should anomaly surfacing be wanted later, it lands as an additive detector sitting on top of this data, reading the same ledger and emitting findings of its own — not as a change to what a flow is.

The same restraint applies to enforcement: netflow observes egress, it does not block it. Deny-by-default egress policy needs the isolated-runtime substrate that the full-fidelity backend would bring, and is out of scope until then.

Still coming

Everything above ships today. What follows genuinely does not, and the page says so rather than implying otherwise: