Recovery & reliability

Long agent runs meet refusals, cut-off replies, rate limits, crashed terminals and killed processes. rupu classifies every reply into a typed outcome, works down a fixed recovery ladder before it gives up, and picks interrupted work up where it stopped instead of starting it over.

Response outcomes

A model's reply does not always give the agent what it needs: it can be cut off at the output limit, refused, blocked by a safety filter, empty, or stopped on a tool call that cannot be parsed. The provider can also fail the request outright. rupu classifies every reply and every provider error. A normal reply (an end_turn, tool_use or stop_sequence stop with usable text or a tool call) has no outcome; anything else gets a typed class with a severity:

Severity decides what the run does next. A warning outcome never fails a run on its own: the reply is kept, the outcome is recorded so the odd stop is visible, and the run carries on. The one info outcome, pause_turn, is handled in place. An error outcome enters the ladder.

Each outcome is one outcome line in the transcript, written ahead of the turn's content, and every recovery step taken for it is one recovery line pointing back at it by id (oc_1, oc_2, … are run-local), so a transcript reads as a timeline. The final run_complete line of a run that failed on an outcome carries that outcome too. The field-level reference is on the Transcript schema page.

The transcript viewers (rupu transcript show, rupu session, rupu watch), the live run view, the workflow printer and rupu autoflow serve print each event as one line, with a glyph for the outcome severity (✗ error, ! warning, ● info) and a ↺ for a recovery:

✗ outcome   truncated · output limit
↺ recovery  rung 0 · continued 1/3
✗ outcome   refused · cyber
↺ recovery  rung 1 · fell back to anthropic/claude-opus-5

The control plane's transcript view shows an outcome as a block with a severity-colored edge, its title and its detail, and a recovery as a one-line timeline entry. Beyond the transcript, a failed workflow step, fan-out unit or run records the outcome that caused it (its cause) on the run record, so a failure is stored as “refused · cyber” and not only as an error string. The control plane does not display cause yet.

The recovery ladder

When an error outcome occurs, rupu works down four rungs and stops at the first one that gets the run moving again:

What each outcome does on rung 0, which later rungs it may use, and what happens to the partial reply:

• A rung-0 budget is per logical turn. A turn that is continued three times and then moves on through tool calls starts the next turn with fresh budgets.
• Still paused after five continuations reclassifies the reply as incomplete (titled incomplete reply · still paused after 5 continuations). It skips the rung-0 retry and goes straight to the fallback hops, and the paused partial reply is kept, not discarded. With no fallback left, the run fails.
• A discarded partial reply is written to the transcript but left out of the conversation: it is not sent back to the model, and a later rupu run --continue does not replay it.
• One ceiling of 20 recovery actions per run. Rung 0 and every hop count against it. At the ceiling the run goes straight to rung 3.
• A fallback that cannot be built (its provider is not configured, say) is recorded as skipped with the reason, counts as an action, and the next fallback is tried.
• Hops are sticky. Once the run has moved to a fallback, later turns stay on it. When the conversation ends on an assistant message, the fallback model is told that a previous attempt stopped and why, and is asked to check the state and continue.
• Rungs 1 and 2 need a fallback chain and a way to build a provider for it. A run with no chain still gets rung 0 and then rung 3.

Configuring fallbacks

A fallback chain is an ordered list of models. Each entry names a model and, optionally, a provider. Declare it on an agent:

---
name: dependency-auditor
provider: anthropic
model: claude-opus-5
fallbacks:
  - model: claude-sonnet-5-5          # same provider as the run → rung 1
  - provider: openai-codex            # another provider → rung 2
    model: gpt-5.6-cyber
---

…or once, for every agent that declares none, in config.toml:

[recovery]
fallbacks = [
  { model = "claude-sonnet-5-5" },
  { provider = "openai-codex", model = "gpt-5.6-cyber" },
]
server_side_fallback = true

• Precedence. An agent's fallbacks: wins over [recovery].fallbacks (default []). They are not merged: when the agent declares a chain, the config table is not consulted for that agent, and fallbacks: [] is an empty chain. No chain in either place means rungs 1 and 2 are unavailable. Both forms reject unknown keys, so a typo in an entry is a load error and not a silent no-op. A rupu release that predates fallbacks: refuses to load an agent file that sets it, so upgrade remote hosts before adding it.
• Which rung an entry belongs to. An entry without a provider means the provider the run started on, even if the run has since moved to another provider; it is a rung-1 candidate, and so is an entry naming that provider. Every other entry is rung 2. Within a rung the chain's order decides. The model the run started on and any entry already tried are never offered again. The split matters because an outcome decides which rungs it may use: a rate_limited error skips the same-provider models (the provider is the problem) and goes to rung 2; a refusal may use both.
• What a hop uses. A hop is built through the same credential resolver, [providers.<name>] table and network logging as the run's own provider, so a same-provider hop differs only in model. The agent's anthropicOauthPrefix and anthropicPromptCache settings carry over to every hop (providers they do not apply to ignore them), and so does effort. The agent's auth: mode applies only to a hop on the provider the run started on. A hop does not inherit the agent's maxTokens or contextWindowTokens: its limits come from [[providers.X.models]] and the provider's model list. A hop to a different model does not send contextWindow, anthropicSpeed, anthropicTaskBudget or anthropicContextManagement either; a hop that keeps the original model's name on another provider keeps them.

Anthropic server-side fallback

Anthropic can fall back to another model on its own side when the model you asked for refuses. rupu opts in for you, which saves a round trip and keeps the reply in one request.

• Models: claude-fable-5, claude-fable-5-1, claude-opus-5, claude-opus-5-5 and claude-sonnet-5-5. A [1m] suffix on the model id is ignored when matching. The list is exact; a newer model is added deliberately.
• Auth: API-key auth only. A run on Anthropic SSO (OAuth) never sends the opt-in.
• On the wire: the request carries the server-side-fallback-2026-07-01 beta and "fallbacks": "default".
• In the transcript: when the fallback answers, the turn proceeds normally. The refusal is recorded as an outcome titled refused · served by <model>, followed by a recovery with action served_by_fallback on rung 1. The reply's fallback boundary is written in place as an assistant_block event, and replay puts it back where it was, since Anthropic requires it echoed in that position on the next request.
• Declined mid-output: when the requested model stops part way through its reply, the API passes only the partial's text on to the fallback model. Tool calls and reasoning from before the boundary are abandoned: rupu never runs those tool calls and leaves them out of the conversation. They are still written to the transcript, as assistant_block events marked abandoned.

Turn it off with [recovery].server_side_fallback = false (default true); the setting applies to the run's own provider and to any Anthropic hop. If the API refuses the opt-in (a 400 that names fallbacks, which an account or gateway that does not accept the beta returns), rupu stops sending it, retries the request once, and writes a notice of kind server_side_fallback_disabled (“server-side fallback refused by the API — disabled for this run”). The run is not affected otherwise; set the key to false for that account to avoid the refused request on every run.

When nothing is left

When the ladder runs out, the run ends with status error. The failure message is the outcome's title followed by a hint; for a provider error it is instead the provider's own error text (provider: <error>) followed by the hint. The hint depends on where the run was started:

# rupu run <agent> (a dispatched sub-agent's own transcript and run record get the same hint)
refused · cyber; no recovery left — continue with a prompt on the same model: rupu run dependency-auditor --continue <run id> "<instruction>", or with another model: rupu run dependency-auditor --continue <run id> --model <model> [--provider <provider>]

# a workflow step
no recovery left — add fallbacks: to the agent or [recovery].fallbacks to try other models

# a session turn (the conversation is kept, so the next message continues it)
no recovery left — send another message to continue, or start a new session on another model

A failed run's final output is empty: its last text is an interim message or a cut-off reply, not an answer. A workflow step tolerated by continue_on_error: publishes steps.<id>.output as "" and the reason as steps.<id>.error; a failed dispatched sub-agent returns "ok": false, "output": "" and the reason as "error" to the agent that dispatched it. That reason leaves out the rupu run … --continue hint, which the dispatching agent cannot act on. How to continue a failed run is under Continue a failed run on another model below.

Context overflow

rupu compacts proactively — summarizing older turns once the conversation reaches compactAtPercent (default 80) of the model's input limit, with room left for a full-length reply. When a provider still rejects a request as too long, rupu reads the real maximum out of the error text (Anthropic, OpenAI, Copilot, vLLM and Gemini formats are each recognized), lowers the run's input limit to it, writes a model_limits_clamped notice, compacts with a summary and retries. Deleting the oldest exchanges is the last resort.

• An Anthropic error saying input plus max_tokens exceeds the window is treated as an output-reservation problem: the turn is retried once with a smaller max_tokens before anything is compacted.
• An Anthropic 429 refusing a 1M-context request on an account without extra-usage billing is not retried with the beta: rupu drops to the standard 200K window for the rest of that client's life, says so, and retries once.
• An observed limit is never written to the shared model cache — it reflects the account, not the model.

Recover on interrupt

A closed terminal, a killed process, a crash or a reboot should not throw away an hour of agent work. rupu keeps enough on disk to continue interrupted work rather than restart it.

Continue an agent run

rupu run <agent> --continue <agent_run_id>

rupu rebuilds the conversation from the run's transcript, drops the turn that was in flight when it stopped, and tells the agent it was interrupted (work from that turn may be partially applied, so the agent re-checks before going on). The new run's transcript links back to the old one rather than copying it. If the run had actually finished, its recorded answer is printed without calling the model. A sub-agent's sub_… id works too: rupu finds its transcript under the run that dispatched it. The continued run also takes on the children the interrupted one launched, under the same handles: finished ones as they ended, the rest continued.

• The command is restored. Every flag the run was launched with (mode, model, provider, findings profile, engagement, skills and the rest) and its repo target come back, recorded as launch_args on the transcript's run_start. A flag given on the continue replaces only its own setting; a repeatable flag (--skill, --engagement-profile) replaces its whole list.
• A repo target reuses the checkout the original run cloned when it is still there (--tmp clones afresh). Run the command from the same directory as the original run.
• The new run needs its own run id: a --run-id that already has a transcript is refused rather than overwritten.
• With no prompt, a run that failed is not continued: a run that failed on a provider-side outcome is continued only with --model or --provider (below), and any other failed run only with a prompt. A run whose last turn ends on a tool call with no result cannot be continued at all.

Continue with a prompt

A continue can be told something new. A positional that parses as a run target is still the target; anything else is the prompt, and --prompt is the explicit form:

rupu run <agent> --continue <agent_run_id> "now write the fix"
rupu run <agent> --continue <agent_run_id> --prompt "now write the fix"

A prompt is new input, so every ended run can take one:

The prompt is never part of the restored command: launch_args records no prompt, and the new run's launch_args are the merged command, still without it. The prompt lives in the new run's transcript as a user_message, and so in the conversation a later --continue of that run rebuilds — continuing it again never re-sends the prompt. With no prompt (or a blank one), nothing changes.

Continue a failed run on another model

rupu run <agent> --continue <agent_run_id> --model <model> [--provider <provider>]

A run that failed on an outcome — a refusal, a reply that kept getting cut off — has an intact conversation, so it can be picked up on a different model. The rebuilt conversation leaves out any discarded partial reply. The new model is told that the earlier attempt stopped, why, and which model it is continuing on.

• It applies only to a run that ended in failure with an outcome recorded. A run that failed for another reason (max turns, say) cannot be continued this way; give it a prompt instead.
• Only a --model or --provider given on the continue counts: the ones a bare --continue restores from the original command do not, so a bare --continue still refuses a failed run.
• --model and --provider override the agent's own model: and provider: for any rupu run, not only a continuation.
• They also drop the agent's settings that were chosen for its own model. A different --model drops maxTokens, contextWindowTokens, compactAtPercent, contextWindow, anthropicSpeed, anthropicTaskBudget and anthropicContextManagement; a different --provider drops the agent's auth: mode. The new model's limits are discovered instead.

Resume a workflow

rupu workflow resume <run-id> (also rupu run resume) takes a failed, cancelled, rejected or paused run back and re-runs only what did not finish. Steps with a recorded result are skipped, and for_each units that already succeeded are replayed from their checkpoints. For each linear step and each for_each unit without a recorded result, resume reads the latest agent attempt's transcript and picks one of three plans:

Resume prints one line per step it planned something for (e.g. each: 1 continued · 1 recovered · 1 restarted) before dispatching; the live view marks a continued or recovered step or unit with ↩ continued / ↩ recovered (a restarted one looks like any fresh start), and the run's events.jsonl records each as an attempt_resumed event naming the interrupted agent run it picked up. Every agent attempt is recorded in a per-run attempts.jsonl ledger as it starts; a run interrupted before the ledger existed is covered by deriving the same attempts from its events.jsonl.

rupu workflow resume run_01EXAMPLE                         # continue interrupted work (the default)
rupu workflow resume run_01EXAMPLE --restart-interrupted   # start every interrupted step and for_each unit over from its prompt
rupu workflow resume run_01EXAMPLE --mode readonly --plain  # override the permission mode; plain line printer instead of the live graph view

A run left Running by a runner that has since died (SIGKILL, an out-of-memory kill, a reboot) is marked Failed first — the same transition rupu cp serve's sweep makes — so the CLI alone can recover it. A run whose recorded runner is still alive is refused.

From the control plane

The control plane records a control action and lets rupu cp serve carry it out, so a 200 or 202 means recorded; the run's own status is what confirms it. See Control plane and the HTTP API.

• Resume (POST /api/runs/:id/resume, a paused run) sets a marker; the resume worker inside cp serve (polling every 4 s) spawns a detached rupu workflow resume --if-unfinished, which refuses a run that finished in the meantime rather than retry it. Work the dead runner left mid-flight is continued from its transcript, as above. Resuming a run that isn't paused is a 409.
• Retry (POST /api/runs/:id/retry, a failed, rejected or cancelled run) records the retry, then spawns a detached rupu workflow resume <id> — plus --restart-interrupted when the body sets restart_interrupted — in the body's mode, else the run's own launch mode. A run left Running by a dead runner is marked Failed first and retried. A standalone agent run is a 409 here: continue it instead. Local only.
• Continue (POST /api/runs/agents/:id/continue, a standalone run_… or a sub-agent's sub_…) spawns a detached rupu run <agent> --continue <id> under a new run id, in the directory the run was launched from. Its optional body — prompt, model, provider, mode, findings_profile, engagement_profiles, skills, preload_skills, messaging — follows the CLI's rules: every original flag is restored and each field given replaces its own; with a prompt any ended run carries on, without one a failed run is continued only with model or provider, and a finished run answers with its recorded answer. 409 while the run is still running. Local only.

Model limits discovery

Every run needs its model's real input limit (for proactive compaction) and output cap (the per-request max_tokens). Per field, rupu takes the first of:

1. the agent's contextWindowTokens / maxTokens frontmatter;
2. a [[providers.<name>.models]] entry in config;
3. the provider's live model list, from the cache;
4. unknown — and the run says so.

A value of 0 counts as unset at every level. Each run writes a model_limits notice to its transcript with the values and where each came from. With an unknown input limit, compaction is off; with an unknown output cap, Anthropic requests carry 8192 and other providers get no cap (the model's own maximum).

Live lists are cached per provider (or per named account) at ~/.rupu/cache/models/<provider>.json for one hour. A run refreshes a stale or missing entry itself, bounded at 10 seconds; a failed refresh is remembered for 5 minutes so a provider outage doesn't cost every launch the timeout. Providers that expose no listing — a Gemini CLI / Code Assist login, for one — leave limits unknown unless you declare them.

rupu models list                              # CONTEXT, OUTPUT, SOURCE and FETCHED per model
rupu models list --provider anthropic-work     # a named account works too
rupu models refresh                           # re-fetch every live list; non-zero exit if none refreshed
rupu models refresh --provider openai

In the control plane, Settings → Models shows the same catalog with per-provider Refetch buttons. See Providers & Auth for where each provider's limits come from.