Provon AI Gateway
The AI Gateway sits between your application and the model providers you connect. Your application sends an OpenAI-compatible request to the Gateway; the Gateway authenticates the caller, selects a configured upstream provider, applies guardrails and usage limits, forwards the request, and records the attempt as trace evidence. Applications keep one base URL and one API key while upstream providers, retries, fallback, and policy decisions stay on the Gateway.
The Gateway governs the model-call path only. It does not manage MCP tool registries, run local scripts, inspect the browser, or replace project permissions. Tool-side rules live in the MCP Gateway; management routes and credentials live in the management API.
What it does#
- Single caller contract. Clients use the OpenAI-compatible shape against the Gateway base URL; the Gateway translates provider-specific requests where a provider requires it.
- Provider-neutral routing. Configure one or more provider keys; model policies and the Auto Router choose among them, retry on failure, and fall over to a backup target.
- Policy on the request path. Usage limits cap requests, tokens, and spend per caller scope; guardrails inspect prompts and completions and block or degrade a call before it reaches the client.
- Traceable attempts. Every attempt records model, provider, latency, tokens, status, and any guardrail decision, linked to the caller context supplied by the client. Captured evidence feeds Rules and Findings in the existing review workflow.
What it does not do#
- It does not resell models or bundle paid quotas. Upstream credentials belong to the providers you configure; the Gateway holds them server-side and uses them only on model calls.
- It does not execute agent tools, local shell commands, or browser actions. Registered tool calls are governed by the MCP Gateway; local shell and browser actions belong to the agent host itself.
- It does not migrate historical product data, and request bodies are not retained indefinitely. Payload capture follows the observability policy you configure.
Objects you configure#
| Object | Purpose | Where you manage it |
|---|---|---|
| Provider key | One upstream connection: provider, base URL, secret, and enabled state | Workbench Providers; provon gateway providers |
| Model binding | Maps a Gateway model id to the upstream model served by a provider key | Provider details; provon gateway providers bind-model |
| Model Policies | Ordered primary target, same-model fallbacks, and substitutions per model pattern | Routing → Model Policies; provon gateway routing model-policies |
| Auto Router | Load-balanced, cost- and health-aware selection among compatible targets | Routing → Auto Router; provon gateway routing auto-router |
| Usage limits | Per-scope request rate or token / spend budgets | Project settings; provon gateway settings update |
| Guardrails | Deterministic and semantic rules over prompts, completions, and tool results | Guardrails; provon gateway guardrails |
| Trace Capture | Which requests keep full evidence, and which headers travel into spans | Providers → Request traces; provon gateway observability |
Request lifecycle#
- The client sends a request to the Gateway with a project API key.
- The Gateway resolves the caller and checks the capability and project-wide enabled state.
- Routing resolves the requested model to a concrete upstream target, applying policy rules and the Auto Router.
- Guardrails and usage limits evaluate the request; a violation produces a structured error or a monitored decision before the upstream call.
- The upstream call executes with retry, circuit state, and fallback attempts recorded separately.
- The response streams back to the caller; the attempt's usage, status, latency, and decisions become trace evidence under the request's conversation or trace id.
Scope boundaries#
| Boundary | Belongs to the Gateway | Belongs elsewhere |
|---|---|---|
| Model requests | /v1/chat/completions, /v1/responses, embeddings, images, audio, Realtime WebSocket |
MCP tool registration, tool policy, and publication under /mcp |
| Caller identity | Project API key with gateway:invoke; session users through Workbench permissions |
Account, organization membership, and sign-in configuration |
| Upstream credentials | Provider key secrets held server-side; never returned to callers | OAuth connector tokens and repository integrations |
| Decisions on traffic | Route selection, retries, limits, guardrails, fallback | Storage retention, export pipelines, and alert rules |
| Playground use | The Playground exercises the Gateway model path for experimentation | Direct upstream calls outside the Gateway bypass every Gateway control |
Task map#
| Want to | Go to |
|---|---|
| Connect a provider and send the first request | Gateway quickstart |
| Switch an existing SDK from direct provider calls to the Gateway | Gateway migration |
| Choose a provider type, add a custom OpenAI-compatible target, or test connectivity | Model providers |
| Define policies, fallbacks, retries, and Auto Router behavior | Routing |
| Set request budgets and write guardrail rules | Limits and guardrails |
| See which requests keep traces and which attributes attach | Gateway tracing |
| Use voice and Realtime model sessions over WebSocket | Realtime Gateway |
| Run the Gateway safely in a self-hosted deployment | Production Gateway |
| Identify a failing call from its status code and body | Gateway troubleshooting |
| Adapt an OpenAI SDK, framework, or caller context header | SDK integrations |
Next steps#
- Complete the quickstart and confirm a trace appears after the first call.
- Review Routing before connecting a second provider.
- Turn on Trace Capture so policy decisions carry evidence.