Send Failures and Fallback
Last updated
What happens when a model call times out, is rate limited, runs out of quota or context, and how fallback picks the next model.
Send failure handling
Agent.Send() failures escalate instead of retrying blindly:| Failure | Behavior |
|---|---|
| Timeout | Up to MAX_RETRY_TIMES (3) retries on the same model — four attempts in all since v1.1.2, which was three — spaced by SEND_TIMEOUT_RETRY_INTERVAL (15 s) |
| Rate limit (HTTP 429 or a rate-limit message) | Register a 30-minute cooldown, retry the same model after 5 s, 10 s, then 15 s; then switch to the next fallback |
| Quota exhausted (HTTP 402 / 403 or a quota / billing message) | Register a 30-minute cooldown and switch to the next fallback immediately |
| Reasoning level rejected (an error naming thinking / reasoning / effort as unsupported or invalid) | Since v1.3.0, retry the same model one level away — down from medium and above, up from below it, then on in that direction — up to MAX_RETRY_TIMES (3) times; the footer and the done event report the level actually used |
| Context-length exceeded | Since v1.3.0, learn the model's real limit from the error (the smallest number of at least 4,096 in it, kept per model for the life of the process and used by the compaction threshold from then on), cut history to 80 % of its size (compact.TrimRatio) and resend; since v1.3.1 a decision model first drops tool steps the request no longer needs, then the oldest exchanges go. Abort only when nothing is left to trim. Removed in v1.3.0: the v1.1.1 step that compacted history first |
| No response while streaming | Since v1.2.3 go-llm-router watches the stream (WatchIdle): a probe every 30 s, a recheck every 10 s after a probe finds no new data (keep-alive frames or data with no parsable content do not count), and after 3 failed probes, or when the stream degenerates into repetition, the stream fails with a [Timeout] error that is handled like a timeout above. Removed in v1.2.3: Agenvoy's own watchdog (UNRESPONSIVE_PROBE_INTERVAL) |
| Any other error | Switch to the next fallback model; abort when no healthy fallback remains |
Fallback candidates come from the priority-ordered list
exec/model.Route builds (ResolveAgent before v1.2.1), which since v1.1.0 excludes pass-tier models. exec/model.Next (nextAgent before v1.2.1) skips models from the failed model's provider (skipping models whose context window cannot hold the input was removed in v1.2.1), health-checks each candidate (HEALTH_CHECK_TIMEOUT, 10 s), and rebuilds the list until MAX_RETRY_TIMES (3) retry rounds have run — four passes in all since v1.1.2, which was three. A session bound to a specific model (anything other than auto) never falls back.Switching models clears
ToolHistories (or applies compact.RawToolFallback), resets the dedupe map, and restarts the counters — a fresh model never inherits the failed model's partial state.Timeout layers
| Layer | Value | Catches |
|---|---|---|
| Provider HTTP client | set inside go-llm-router per provider | Transport-level stalls |
AGENT_SEND_TIMEOUT_SEC | limits.agent_send_timeout_seconds in config.json, default 600; since v1.1.2 a smaller value is ignored | Exec-layer ceiling via context.WithTimeout |
Stream idle check (go-llm-router WatchIdle) | probe every 30 s; after a failed probe, recheck every 10 s; 3 failures end the stream as a timeout | A stream that hangs, sends only keep-alives, or repeats itself without erroring |
| Health check | 10 s | Liveness probe of each fallback candidate |