Gateway Routing And Reliability

Capability filtering happens before scoring or recovery. Provon never routes a request to a target that cannot satisfy its endpoint and requested features.

Resolution Precedence#

Direct Provider#

openai/gpt-5-mini pins provider resolution to openai. Provon still selects among eligible Provider Keys and credentials for that provider, but it does not substitute a different provider through project Model Policies.

Use this form as the default when the application should make provider intent explicit.

Model Policies#

A plain model such as support-model is matched against project Model Policies. A policy can define:

• one primary Provider Key or provider;
• ordered same-model recovery targets;
• ordered cross-model substitution targets.

Model patterns support *. Rules are evaluated in configured order, and the first matching rule wins.

If no policy matches, provider registry inference uses known model namespaces and patterns. Treat registry inference as a convenience, not a substitute for an explicit production policy.

Auto Router#

provon/auto asks Provon to select from enabled model mappings on BYOK Provider Keys. Auto Router:

1. filters mappings by requested endpoint and features;
2. applies the optional allowed-model patterns;
3. excludes unavailable targets;
4. scores eligible targets using priority, latency, health, and estimated cost;
5. uses round-robin ordering to break ties.

The Selection bias control moves the scoring balance between quality-oriented runtime signals and estimated cost. It does not override capability requirements or hard usage policies.

Provider Keys#

A Provider Key is the routable upstream target. It contains:

• provider and display name;
• enabled state and priority;
• one or more upstream credentials;
• optional endpoint override;
• optional request timeout;
• optional header overrides;
• zero or more model mappings;
• optional retry, fallback, circuit-breaker, and load-balance policies.

Within a provider, drag Provider Keys in Providers to set their priority. Matching requests use the first eligible target after policy, health, and load-balancing decisions.

Multiple Credentials And Load Balancing#

A Provider Key can store more than one upstream credential. The runtime uses the credential priority and the key's load-balance policy to choose which credential sends a given request. This is useful when:

• an upstream account provides multiple API keys and you want to spread load;
• you are rotating a secret and need both old and new keys available briefly;
• different keys have different rate-limit headroom.

Load balance strategies are round-robin and random. When load balancing is disabled, the highest priority enabled credential is used.

Model Mapping Modes#

Each mapping can declare:

• public AI Gateway model ID;
• upstream model ID;
• eligible endpoint families;
• required features;
• context window and maximum output metadata;
• pricing profile metadata.

Model mappings are routing truth. Pricing definitions only enrich estimates and traces; they do not make a model routable.

Configure Model Policies#

In the Workbench:

1. Connect the required Provider Keys under Providers.
2. Open Routing.
3. Under Model Policies, add a rule for the plain model pattern.
4. Choose the Primary Provider key.
5. Optionally enable Same model recovery and order alternate targets.
6. Optionally enable Model substitution and declare each alternate model explicitly.
7. Save the policy.

Example intent:

support-model
  -> OpenAI primary / gpt-5-mini
  -> Azure recovery / gpt-5-mini
  -> Anthropic substitution / claude-sonnet

Same-model recovery preserves the requested model identity across targets. Model substitution changes model behavior and must be enabled explicitly.

Configure Auto Router#

In Routing:

1. Enable Auto Router.
2. Leave Allowed models empty to consider all enabled mappings, or add wildcard patterns such as openai/* and anthropic/claude-*.
3. Set Selection bias between Quality and Cost.
4. Save the configuration.
5. Send requests with "model": "provon/auto".

Before relying on Auto Router, verify that each candidate mapping declares accurate endpoints, features, and upstream model IDs.

Reliability Layers#

Provon distinguishes several failure-handling mechanisms:

Retries and fallback are disabled unless configured for the target. This avoids silently repeating non-idempotent work or changing providers after application errors.

Reliability Configuration#

Reliability policies are attached to a Provider Key. They can be set through the Workbench, the Gateway CLI, or the Provider Key management API.

Example Provider Key configuration:

{
  "provider": "openai",
  "displayName": "OpenAI production",
  "timeoutMs": 60000,
  "retryPolicy": {
    "enabled": true,
    "maxRetries": 2,
    "statusCodes": [429, 500, 502, 503],
    "networkErrors": true,
    "followRetryAfter": true,
    "maxRetryAfterMs": 30000,
    "baseDelayMs": 500,
    "maxDelayMs": 8000
  },
  "fallbackPolicy": {
    "enabled": true,
    "statusCodes": [429, 500, 502, 503],
    "networkErrors": true
  },
  "circuitBreakerPolicy": {
    "enabled": true,
    "failureThreshold": 5,
    "openStateDurationMs": 30000,
    "networkErrors": true
  },
  "loadBalance": {
    "enabled": true,
    "strategy": "round-robin"
  }
}

Retry policy fields#

Fallback policy fields#

Circuit breaker policy fields#

Load balance policy fields#

Keep the overall Gateway request deadline larger than the sum of retry and fallback attempts. Streaming responses can only recover before the first chunk is sent to the client.

Retry And Fallback Guidance#

• Retry only transient network failures, 429, and selected 5xx responses.
• Keep retry counts and total request deadlines bounded.
• Honor Retry-After only within the configured maximum delay.
• Use same-model fallback before cross-model substitution.
• Do not fallback on authentication errors, invalid requests, or model permission failures.
• Verify that every fallback target supports the endpoint and requested features.
• Inspect attempt spans after changing recovery policy.

For streaming calls, recovery is only safe before a response has been committed to the client.

Capability-Aware Selection#

Provon derives request requirements from the endpoint and body, including:

• streaming;
• tool calling;
• JSON mode and structured outputs;
• vision input or audio input/output;
• reasoning.

Query eligible models before deploying a new request shape:

export PROVON_API_URL="https://api.provon.dev/v1"

curl \
  "$PROVON_API_URL/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"

See Gateway API for the full discovery contract.

Verify Routing#

After a test request, open its trace and verify:

• requested, Gateway, and upstream model IDs;
• selected Provider Key and credential fingerprint;
• target priority and routing source;
• fallback index and retry count;
• upstream status, duration, and error classification;
• circuit, health, or guardrail events that changed execution.

The final HTTP response shows the winning attempt. The trace explains why that attempt ran.