# Gateway Routing And Reliability

Provon separates model intent from upstream execution. A request first resolves its `model` value
into eligible Provider Key targets, then orders those targets using project policy and runtime
signals.

Capability filtering happens before scoring or recovery. Provon never routes a request to a target
that cannot satisfy its endpoint and requested features.

## Resolution Precedence

| Request model        | Resolver                                | Typical use                                   |
| -------------------- | --------------------------------------- | --------------------------------------------- |
| `<provider>/<model>` | Direct Provider                         | Explicit, portable application integration    |
| `<model>`            | Model Policies, then registry inference | Stable logical model names owned by a project |
| `provon/auto`        | Auto Router                             | Dynamic selection across mapped BYOK targets  |

### Direct Provider

`openai/gpt-5-mini` pins provider resolution to `openai`. Provon still selects among eligible
Provider Keys and credentials for that provider, but it does not substitute a different provider
through project Model Policies.

Use this form as the default when the application should make provider intent explicit.

### Model Policies

A plain model such as `support-model` is matched against project Model Policies. A policy can define:

- one primary Provider Key or provider;
- ordered same-model recovery targets;
- ordered cross-model substitution targets.

Model patterns support `*`. Rules are evaluated in configured order, and the first matching rule
wins.

If no policy matches, provider registry inference uses known model namespaces and patterns. Treat
registry inference as a convenience, not a substitute for an explicit production policy.

### Auto Router

`provon/auto` asks Provon to select from enabled model mappings on BYOK Provider Keys. Auto Router:

1. filters mappings by requested endpoint and features;
2. applies the optional allowed-model patterns;
3. excludes unavailable targets;
4. scores eligible targets using priority, latency, health, and estimated cost;
5. uses round-robin ordering to break ties.

The **Selection bias** control moves the scoring balance between quality-oriented runtime signals
and estimated cost. It does not override capability requirements or hard usage policies.

## Provider Keys

A Provider Key is the routable upstream target. It contains:

- provider and display name;
- enabled state and priority;
- one or more upstream credentials;
- optional endpoint override;
- optional request timeout;
- optional header overrides;
- zero or more model mappings;
- optional retry, fallback, circuit-breaker, and load-balance policies.

Within a provider, drag Provider Keys in **Providers** to set their priority. Matching requests use
the first eligible target after policy, health, and load-balancing decisions.

### Multiple Credentials And Load Balancing

A Provider Key can store more than one upstream credential. The runtime uses the credential priority
and the key's load-balance policy to choose which credential sends a given request. This is useful
when:

- an upstream account provides multiple API keys and you want to spread load;
- you are rotating a secret and need both old and new keys available briefly;
- different keys have different rate-limit headroom.

Load balance strategies are `round-robin` and `random`. When load balancing is disabled, the highest
priority enabled credential is used.

### Model Mapping Modes

| Mode         | Behavior                                               |
| ------------ | ------------------------------------------------------ |
| **All**      | Route models through the key without explicit mappings |
| **Specific** | Route only model IDs declared by enabled mappings      |

Each mapping can declare:

- public AI Gateway model ID;
- upstream model ID;
- eligible endpoint families;
- required features;
- context window and maximum output metadata;
- pricing profile metadata.

Model mappings are routing truth. Pricing definitions only enrich estimates and traces; they do not
make a model routable.

## Configure Model Policies

In the Workbench:

1. Connect the required Provider Keys under **Providers**.
2. Open **Routing**.
3. Under **Model Policies**, add a rule for the plain model pattern.
4. Choose the **Primary Provider key**.
5. Optionally enable **Same model recovery** and order alternate targets.
6. Optionally enable **Model substitution** and declare each alternate model explicitly.
7. Save the policy.

Example intent:

```text
support-model
  -> OpenAI primary / gpt-5-mini
  -> Azure recovery / gpt-5-mini
  -> Anthropic substitution / claude-sonnet
```

Same-model recovery preserves the requested model identity across targets. Model substitution
changes model behavior and must be enabled explicitly.

## Configure Auto Router

In **Routing**:

1. Enable **Auto Router**.
2. Leave **Allowed models** empty to consider all enabled mappings, or add wildcard patterns such as
   `openai/*` and `anthropic/claude-*`.
3. Set **Selection bias** between **Quality** and **Cost**.
4. Save the configuration.
5. Send requests with `"model": "provon/auto"`.

Before relying on Auto Router, verify that each candidate mapping declares accurate endpoints,
features, and upstream model IDs.

## Reliability Layers

Provon distinguishes several failure-handling mechanisms:

| Mechanism       | Scope                      | Behavior                                                         |
| --------------- | -------------------------- | ---------------------------------------------------------------- |
| Request timeout | One Provider Key attempt   | Abandons an attempt after its configured duration                |
| Retry           | Same upstream target       | Repeats configured status or network failures with bounded delay |
| Fallback        | Ordered target chain       | Moves to the next eligible target after configured failures      |
| Circuit breaker | Across requests per target | Temporarily skips a repeatedly failing target                    |
| Target health   | Across requests per target | Removes unavailable targets from health-aware routing            |

Retries and fallback are disabled unless configured for the target. This avoids silently repeating
non-idempotent work or changing providers after application errors.

## Reliability Configuration

Reliability policies are attached to a Provider Key. They can be set through the Workbench, the
[Gateway CLI](../cli/gateway.md), or the Provider Key management API.

Example Provider Key configuration:

```json
{
  "provider": "openai",
  "displayName": "OpenAI production",
  "timeoutMs": 60000,
  "retryPolicy": {
    "enabled": true,
    "maxRetries": 2,
    "statusCodes": [429, 500, 502, 503],
    "networkErrors": true,
    "followRetryAfter": true,
    "maxRetryAfterMs": 30000,
    "baseDelayMs": 500,
    "maxDelayMs": 8000
  },
  "fallbackPolicy": {
    "enabled": true,
    "statusCodes": [429, 500, 502, 503],
    "networkErrors": true
  },
  "circuitBreakerPolicy": {
    "enabled": true,
    "failureThreshold": 5,
    "openStateDurationMs": 30000,
    "networkErrors": true
  },
  "loadBalance": {
    "enabled": true,
    "strategy": "round-robin"
  }
}
```

### Retry policy fields

| Field                        | Purpose                                          |
| ---------------------------- | ------------------------------------------------ |
| `enabled`                    | Turn retries on or off                           |
| `maxRetries`                 | Maximum attempts on the same target              |
| `statusCodes`                | HTTP statuses that trigger a retry               |
| `statusRanges`               | Ranges of statuses that trigger a retry          |
| `networkErrors`              | Retry on DNS, timeout, or connection failures    |
| `followRetryAfter`           | Honor the upstream `Retry-After` header          |
| `maxRetryAfterMs`            | Cap the delay from `Retry-After`                 |
| `baseDelayMs` / `maxDelayMs` | Back-off bounds when no `Retry-After` is present |

### Fallback policy fields

| Field           | Purpose                                                  |
| --------------- | -------------------------------------------------------- |
| `enabled`       | Turn fallback on or off                                  |
| `statusCodes`   | Final statuses that trigger the next target in the chain |
| `statusRanges`  | Ranges of statuses that trigger fallback                 |
| `networkErrors` | Fallback when the current target cannot be reached       |

### Circuit breaker policy fields

| Field                          | Purpose                                         |
| ------------------------------ | ----------------------------------------------- |
| `enabled`                      | Turn the breaker on or off                      |
| `failureThreshold`             | Consecutive failures before opening the breaker |
| `openStateDurationMs`          | How long the breaker stays open before a probe  |
| `statusCodes` / `statusRanges` | Which responses count as failures               |
| `networkErrors`                | Whether network errors count as failures        |

### Load balance policy fields

| Field      | Purpose                                          |
| ---------- | ------------------------------------------------ |
| `enabled`  | Turn load balancing across credentials on or off |
| `strategy` | `round-robin` or `random`                        |

Keep the overall Gateway request deadline larger than the sum of retry and fallback attempts.
Streaming responses can only recover before the first chunk is sent to the client.

## Retry And Fallback Guidance

- Retry only transient network failures, `429`, and selected `5xx` responses.
- Keep retry counts and total request deadlines bounded.
- Honor `Retry-After` only within the configured maximum delay.
- Use same-model fallback before cross-model substitution.
- Do not fallback on authentication errors, invalid requests, or model permission failures.
- Verify that every fallback target supports the endpoint and requested features.
- Inspect attempt spans after changing recovery policy.

For streaming calls, recovery is only safe before a response has been committed to the client.

## Capability-Aware Selection

Provon derives request requirements from the endpoint and body, including:

- streaming;
- tool calling;
- JSON mode and structured outputs;
- vision input or audio input/output;
- reasoning.

Query eligible models before deploying a new request shape:

```bash
export PROVON_API_URL="https://api.provon.dev/v1"

curl \
  "$PROVON_API_URL/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

See [Gateway API](../api/gateway.md#endpoint-discovery) for the full discovery contract.

## Verify Routing

After a test request, open its trace and verify:

- requested, Gateway, and upstream model IDs;
- selected Provider Key and credential fingerprint;
- target priority and routing source;
- fallback index and retry count;
- upstream status, duration, and error classification;
- circuit, health, or guardrail events that changed execution.

The final HTTP response shows the winning attempt. The trace explains why that attempt ran.