Multi-host

Control rupu agents and workflows running on other machines from one central Control Plane — register remote hosts, launch work on the host you choose, and watch it stream back, all in a single browser tab.

What it is

Multi-host is a hub model. One central Control Plane federates out to a fleet of remote hosts, each running its own control plane (rupu cp serve). A Host is a top-level peer in rupu — it sits alongside your Projects, not underneath them. The machine the central CP runs on is itself just a host: host[0], "this host", always present and always first.

The remote is always the source of truth — the central CP keeps no authoritative copy of a host's run state, and tags every run with the host it came from. How it reaches that state depends on the transport: directly-reachable hosts (HTTP, SSH) are live-queried on demand — open a run list and the CP fans out to each host, merges the results, and proxies straight through to a run's detail / event stream / transcript; hosts that can't be reached inbound (tunnel, bucket) mirror their run artifacts back to the central CP as they change. Either way you get one merged fleet view, and no host's state is duplicated as a second source of truth.

Connection types

Every host is reached through a single transport port (HostConnector), so the Fleet surface, host attribution, launcher, and live streams work the same no matter how a host is wired up. Five transports ship today, covering hosts you can reach directly, hosts behind a NAT or firewall, and hosts you can't reach at all:

Local — host[0]

The machine running the central CP is itself a host: the built-in local entry, always present and always first. It needs no registration and can't be removed. Single-host setups never touch any of the transports below — everything just runs on local.

HTTP federation

Federation has two sides. On the remote machine, run the control plane so it's reachable from the center and guarded by a token. Bind to an address other than loopback, and set --token so /api/* requires Authorization: Bearer <token>:

# on the REMOTE machine — serve the control plane, bound + token-guarded
$ rupu cp serve --bind 0.0.0.0:7878 --token "$RUPU_CP_TOKEN" --no-open

Put a TLS-terminating reverse proxy in front for a real https:// URL — remote base URLs are expected to be HTTPS so the token is protected in transit. Then, on the central machine, register that host with a display name, the remote's base --url, and the --token:

# on the CENTRAL machine — register the remote host
$ rupu host add prod --url https://host.example.com --token "$REMOTE_TOKEN"
host_01J9Z4W7Q0X8Y6V5K3M2N1P0R8

To keep the token out of your shell history and process list, read it from stdin instead with --token-stdin:

# pipe the token in on stdin (mutually exclusive with --token)
$ pass show rupu/prod | rupu host add prod --url https://host.example.com --token-stdin

SSH

The SSH transport targets a host that runs a full rupu install but exposes no rupu cp serve and runs no dial-home agent — yet you can already ssh to it. The central CP connects out over SSH: it dispatches a run with ssh host rupu workflow run … (detached on the remote so it survives the SSH session), then keeps a long-lived ssh … tail -f pump that mirrors the run's artifact files back into the central run store — so the run shows up in the same host-aware lists, detail, and live events as any other. SSH hosts are fully at parity with the other transports: they report the remote rupu version and list sessions, agent + autoflow runs, and events just like HTTP hosts do. Register it with the --ssh destination:

# register an SSH host — destination is user@host or a ~/.ssh/config alias
$ rupu host add build-box --ssh deploy@build.example.com --port 22 --identity ~/.ssh/id_ed25519
host_01J9ZB3KQ8M2T0V4W6X8Y1A3C5

Transcripts stream back on demand. Every step, fan-out unit and agent run that executes on an SSH host — a whole workflow launched there, or a single host: / distribute: unit of a local workflow — shows its transcript in the central CP, live while it runs and complete afterwards. The mirror is lazy: no SSH session is opened for a transcript nobody is looking at. Opening one starts a single shared tail -F for that file, however many viewers have it open, and it is stopped when the last one leaves. When the run finishes, one batched pull copies every transcript (and the run's usage, coverage and netflow streams) to the coordinator, so they stay readable and counted in usage even after the host goes offline or is removed. A transcript the coordinator couldn't finish collecting is labelled partial, never silently cut short. Launches on the remote are detached with setsid where it exists, else nohup, so they survive the SSH session on Linux and macOS alike.

Tunnel (dial-home)

When a host is behind a NAT or firewall and can't expose any inbound port, invert the direction: a lightweight rupu node agent on the box dials out to the CP over a persistent WebSocket, executes dispatched runs locally, and streams their artifacts back. The CP reaches the node through its node registry — no inbound ports, no reachable server. Start on the central machine by enrolling the node, which mints a one-time token and prints the exact command to run on the box:

# on the CENTRAL machine — mint a node + one-time token
$ rupu node enroll build-box-01 --cp-url wss://cp.example.com
enrolled: build-box-01 (host_01J9ZC4M0N1P2Q3R4S5T6U7V8W)

⚠  token shown ONCE — copy it to the node now:

  rupu node --cp-url wss://cp.example.com --token <token> --node-id host_01J9ZC4M0N1P2Q3R4S5T6U7V8W

Copy the token to the remote box and run the node agent there, reading the token from stdin so it never lands in shell history. The agent authenticates, stays connected, and reconnects with backoff if the link drops:

# on the REMOTE box — dial home and stay connected
$ rupu node --cp-url wss://cp.example.com --token-stdin < token.txt

Bucket (pull / dead-drop)

The most decoupled transport: for a host the CP can't reach at all and which can't reach the CP either. Both sides independently talk to a shared object-store bucket that acts as the gateway. The CP writes dispatched work into the bucket; the worker polls it, atomically claims a job, runs it locally, and writes results back; a CP-side poller reads those results and mirrors them in. Register the host on the central machine with a bucket URL and optional prefix:

# on the CENTRAL machine — register a bucket (dead-drop) host
$ rupu host add nightly --bucket s3://my-bucket --prefix rupu/host-1
host_01J9ZD5N1P2Q3R4S5T6U7V8W9X

On the worker box, run the pull agent against the same bucket and prefix. It claims jobs, runs them locally, and writes results back — no inbound or outbound connectivity to the CP required. By default it loops forever, polling on an interval; pass --once to drain whatever is queued a single time and exit (handy for cron / batch hosts):

# on the WORKER box — poll the bucket, claim + run jobs, write results back
$ rupu node pull --bucket s3://my-bucket --prefix rupu/host-1

# or drain once and exit (cron / batch)
$ rupu node pull --bucket s3://my-bucket --prefix rupu/host-1 --once

Manage hosts

rupu host list shows every configured host. The built-in local host is always first and can't be removed — it's host[0], "this host." Each row is the host id, name, and a transport label (the base URL for HTTP, or ssh:… / tunnel:… / bucket:… for the others):

$ rupu host list
local                              local         local
host_01J9Z4W7Q0X8Y6V5K3M2N1P0R8   prod          https://host.example.com
host_01J9ZB3KQ8M2T0V4W6X8Y1A3C5   build-box     ssh:deploy@build.example.com:22
host_01J9ZC4M0N1P2Q3R4S5T6U7V8W   build-box-01  tunnel:host_01J9ZC4M0N1P2Q3R4S5T6U7V8W
host_01J9ZD5N1P2Q3R4S5T6U7V8W9X   nightly       bucket:s3://my-bucket

Remove a remote host by its id with rupu host remove. This deletes the TOML record and any keychain token; removing local is refused:

$ rupu host remove host_01J9Z4W7Q0X8Y6V5K3M2N1P0R8
removed host_01J9Z4W7Q0X8Y6V5K3M2N1P0R8

The same registry is exposed in the browser — adding and removing hosts in the Control Plane writes the exact same TOML records and keychain entries, so the CLI and the CP are always looking at one shared fleet. Removing a tunnel host from the control plane also closes its live node connection at once; the node's reconnect then fails enrollment.

In the Control Plane

With hosts registered, the central Control plane becomes fleet-aware. Everything you can do on the local host you can now do on a remote one — whichever transport reaches it.

• Fleet → Hosts. A new top-level surface listing every host (name, transport, status, version, active runs, last seen), with local pinned first and an Add Host form that registers HTTP, SSH, and bucket hosts right from the browser (tunnel nodes enroll via the CLI). Each host's status is probed live; a host's detail view scopes the run list to that one host plus its connection and health info.
• Host attribution + filtering. Run lists gain a Host column and filter so you can see at a glance where each run is executing. Runs are addressed by (host_id, run_id), so deep-links stay stable; run detail shows the owning host. The default view stays "this host," so single-host setups are completely unchanged.
• Host selector in the launcher. The launcher gains a host picker (defaulting to local). Launch a workflow run, an agent run, or a session on whichever host you choose — and send session turns to it — all proxied to that host's launch endpoints. Skills on a remote launch need a rupu there that advertises agent.skills (a run) or session.skills (a session); an older host is refused before anything is sent. A remote host gets no / skill menu in the composer, since this control plane can't list its skills, but a typed /name still reaches a rupu there that reads it.
• Observe a host's runs. List, detail, the live event stream, and the transcript of any remote run all surface in the central CP — proxied straight through for HTTP hosts, mirrored into the central run store for tunnel / SSH / bucket hosts. The same live view works whether the run is local or three hosts away.
• Control remote runs. Approve, reject, and cancel a paused or in-flight run on the host that owns it — over whichever transport that host uses (queued in the bucket for dead-drop hosts).
• Resume a paused run. Resume works on HTTP hosts (the host's own resume worker runs it) and on SSH hosts, where one SSH command checks the remote's features and runs rupu workflow resume --if-unfinished — so a run that finished since you asked is refused instead of retried. The CP only asks a host to resume a run it sees as paused. Tunnel and bucket hosts have no resume path yet.
• Loaded host by host. The Activity tables, the Usage headline and the ⌘K palette ask each host independently and merge the answers as they arrive, so one slow SSH host never holds the page. SSH listings share a short single-flight cache (cleared by any change you make), which keeps an open, polling table from hammering your hosts.
• Graceful degradation. When a host is unreachable, the run-list fan-out tolerates it per-host: that host shows an offline marker (or unavailable when it's reachable but runs a rupu too old to answer) and the rest of the page stays intact. Launch or control against an offline host returns a clear error (a 502; a run the host doesn't have is a 404, a transition it refuses a 409, and an operation its transport or rupu version can't do a 501). Live streams auto-reconnect and pick up after the last event they saw, rather than replaying the run (an HTTP host on an older rupu still replays).

Distribute work across the fleet

Beyond running a whole workflow on one host, a single workflow run can place individual workflow steps — or spread one step's units — across the fleet. Three placement controls ship today, and they all work over every transport (HTTP / SSH / tunnel / bucket): fan-out units, per-step placement, and workspace sync so a remotely-placed step can read and write your real project files.

Fan-out units across hosts

A for_each step's units can be spread across the fleet. Add a distribute: block naming the hosts; rupu assigns the units round-robin, dispatches each to its host, attributes the unit's run to that host, and aggregates the results back into the step — partial failures honored per continue_on_error.

  - id: review_each
    agent: code-reviewer
    for_each: "{{ inputs.files }}"
    distribute:
      hosts: [gpu-box, build-box]
    prompt: "Review {{ item }}."

Per-step host placement

A single linear step (one with an agent and a prompt) can run on a named fleet host with a host: field. The whole step's agent runs on that host instead of locally, and its output feeds downstream steps exactly as a local step would. host: is linear-only — it's rejected on for_each, parallel, and panel steps (use distribute: to spread a for_each), and it must name a non-empty host. Absent, the step runs locally, exactly as before.

  - id: train
    agent: model-trainer
    host: gpu-box
    prompt: "Fine-tune on {{ inputs.dataset }}."

A placed unit (host: or a distribute: unit, including a fallback-host retry and a resumed run) runs under the run's engagement profile: the ids travel to the host as rupu run --engagement-profile and resolve against the host's own profiles, so install a custom profile there too. A host that can't honour it is refused, never run on the code path. A placed step's skills: / preload_skills: travel with the unit too — see Skills on placed units below.

Workspace sync for remote steps

By default a remotely-placed step is self-contained — it sees only its rendered prompt plus prior steps' string outputs, not your files. Add workspace: sync to a remote step (one with host: or distribute:) and the coordinator ships its file workspace to the host: the step's agent reads and writes the real project files, and the changed files are propagated back so downstream steps see them. This is how file-mutating steps can now run remotely.

The mechanism is auto git-or-tar: if the workspace is a git repo it syncs via git (patch / 3-way merge on the return path); otherwise it packs a gitignore-aware tar and returns a changed-files delta. Detection is automatic — no extra config, and no git/tar tooling needs to be installed on the host (rupu carries a vendored libgit2 and tar). It's opt-in and only valid on a remote step; on a purely-local step workspace: sync is rejected as a no-op.

  - id: refactor
    agent: refactorer
    host: gpu-box
    workspace: sync
    prompt: "Apply the migration across the crate."

See Workflows for the full step format.

Local concurrency and host memory

A wide fan-out or a split: graph can ask for far more concurrent agents than a machine can hold — each one keeps a conversation in memory and spawns tool subprocesses. Every rupu process therefore runs local agent jobs through an admission gate with two layers, whatever a workflow's max_parallel: or max_concurrency: says:

A throttled step gets a run warning, so a slow fan-out explains itself. Remote units are not counted against the coordinator's gate: they spend the remote host's memory, and that host's own gate bounds them.

What comes back from a placed unit

A placed unit runs as a standalone rupu run on its host, so what it records lands there first. Three streams bring it home when the unit ends: its coverage and findings, its netflow ledgers, and (on first view) the artifacts its findings reference. None of them can fail the unit; what could not be collected is reported, never silently missing.

Coverage and findings

A remote unit's coverage and findings come home through the run's coverage stream — every rupu run streams its runs, file touches, concern assertions, findings and engagement assets to $RUPU_HOME/runs/<run_id>/coverage.jsonl, which the host's connector delivers to the coordinator when the unit ends, success or failure, on every transport. The coordinator merges it under its own workspace's targets (re-keyed to the run's scope, de-duplicated, the unit's tag events appended to the workspace tag log by event id) before the unit's workspace: sync delta applies, and records the unit's artifacts as stored: external with the host's id. A unit placed on the coordinator's own local host shares its store, so its artifacts stay copied.

A finding's artifacts (proof-of-concept files, evidence-block files such as images) stay on the host that recorded them until someone opens them. The first preview or download of one (GET /api/findings/:id/artifacts/:sha256, the web finding page's artifact browser) pulls the blob from its host, capped at the recorded size and checked against the recorded size and SHA-256 before it enters the coordinator's store. Concurrent first downloads share one pull, and a recorded size over the coordinator's own [findings].artifact_max_bytes is refused without contacting the host. Anything that can't be had — the host unreachable or not registered, the node offline, the blob not in the host's store, the host too old to serve it, a failed size or hash check — is a 404 with {"unavailable": "<reason>"}; nothing is stored for a failed pull, and the next view tries again. How each transport delivers the blob is in the per-transport summary below.

Netflow

A placed unit's netflow ledgers, and those of every sub-agent and child beneath it, are teed on the host into one stream in the run's directory (runs/<id>/netflow.jsonl), which starts with a begin line even when the run made no network calls. When the unit ends the coordinator collects the stream and lands it: split per ledger id into its own ~/.rupu/netflow/<id>.jsonl, byte for byte, with a landing record (~/.rupu/runs/<id>/netflow.landing.json) that says what arrived — complete, partial with the reason, or unavailable for a host too old to stream or a stream that failed or was lost. A complete record is never downgraded. A run placed on a host whole lands the same way at its end.

Nothing lands while the unit is still running. Once landed, the ledgers are read at run, project, global and agentiflow scope like any other, and a copy that is partial, unavailable or not collected at all is listed as incomplete at every scope, naming the host and the reason; the workflow run also gets a step warning. A short or missing copy is never shown as whole.

Skills on placed units

A placed unit does not need its skills installed on the host. The coordinator resolves them the way a local run does — the skills: / preloadSkills: of its own copy of the agent plus the step's skills: / preload_skills: (* means the coordinator's global layer) — and ships them. A step skill not installed on the coordinator fails the unit before dispatch, as it would a local run. If the coordinator has no copy of the agent, or its file does not load, only the step's skills ship, with a warning naming the file.

Each skill is packed as a tar named by its content hash; the coordinator asks the host which hashes it lacks and sends only those. The host checks every tar before keeping it (no absolute or .. paths, no symlink that leaves the skill, only files, directories and symlinks, and the unpacked tree must hash to its name) and keeps it in <RUPU_HOME>/cache/skills/<hash>/<name>/, which is never evicted. A shipped skill replaces the host's installed skill of the same layer and name, so the coordinator's version wins; a grant nobody shipped still resolves from what the host has installed, and fails closed if it has nothing. The unit's skill_index marks shipped skills (shipped), and rupu run --continue on the host runs with the same shipped skills.

Messaging on placed units

Placed units are not in the coordinator's message space: each runs as a standalone rupu run on its host, with a space of its own there and the default messaging tools. A workflow's messaging: key still reaches them — false is sent as rupu run --no-messaging, and true or { tools: [...] } as one rupu run --messaging-tool per tool, so the unit gets the same list on its host. A placed unit of a workflow launched with messaging off has it off too. A host whose rupu predates the flags — it doesn't advertise agent.no_messaging or agent.messaging_tools — is refused the unit rather than run it with messaging on.

Operator messages to a remote run go through the same GET/POST /api/{runs,sessions,agentiflows}/:id/messages routes as a local one. HTTP and SSH hosts answer when their rupu advertises run.messages; tunnel and bucket hosts answer 501.

Per-transport summary

What each transport carries for a placed unit, from the sections above. A dash means the transport has no path for it today.

Feature advertisement

Hosts in a fleet don't all run the same rupu. Rather than guess from version numbers, each host advertises the features it supports, and the coordinator only asks for what's advertised: HTTP hosts list them in /api/host/info, tunnel nodes send them in their hello, bucket workers write a marker file in the bucket, and an SSH host answers the hidden rupu __features (an older one prints nothing, which reads as no features).

The rule is never to drop a request silently: a feature the peer didn't advertise is refused with an explanation, or the request falls back to the older form and says so — in the log, or as a warning on the run. Over SSH the launch command itself checks the remote's rupu __features for what the launch needs, so a remote that can't honour it prints a marker instead of launching.

Choosing a transport

All five transports satisfy the same port, so pick by how the host can be reached on the network — not by what features you want:

• The CP can reach the host directly. If the host can run a reachable, TLS-fronted server, use HTTP. If it can't run a server but you can already ssh to it, use SSH — no daemon to deploy, auth comes from your SSH config.
• The host is behind a NAT or firewall. It can't accept an inbound connection but can dial out — use the Tunnel: run rupu node on the box and it connects home.
• Neither side can reach the other. Air-gapped, locked down, or batch / cron hosts that only share an object store — use a Bucket dead-drop and run rupu node pull on the worker.

Still coming

All five transports above are shipped today, as are the three placement controls in Distribute work across the fleet (fan-out units, per-step host:, and workspace: sync). The architecture was built around one transport port (HostConnector) so the remaining work slots in without disturbing the surfaces above. These are genuinely not done yet: