Gateway Troubleshooting

Fast Triage#

For one failing request, record:

• HTTP status and complete response error body;
• x-provon-request-id, when present;
• request endpoint and model value;
• whether the failure happens before or after streaming starts;
• the project API key ID, not its secret;
• the selected trace and provider-attempt spans, when present.

Then follow this sequence:

1. No request ID and 401 or 403: verify client authentication and key capability.
2. Gateway error before an upstream attempt: inspect model resolution, limits, guardrails, or project state.
3. Provider attempt exists: inspect its status, error category, selected Provider Key, and upstream model.
4. Model response succeeds but no trace appears: inspect Trace Capture and telemetry delivery.

Do not enable retries or fallback until the primary failure is understood.

Common Statuses#

Use error.code as the primary signal. The same status can represent different layers.

An upstream provider can return the same statuses. When a trace exists, the root span shows the final Gateway result and child attempt spans show upstream responses.

Authentication Fails With 401 Or 403#

Verify all of these:

Base URL:      https://gateway.provon.dev/v1
Authorization: Bearer <Provon project API key>
Capability:    gateway:invoke

Common mistakes:

• sending the upstream provider key to Provon;
• pointing the client at the provider's original base URL;
• using a session cookie or connector credential in an application;
• configuring a client that sends only x-api-key;
• appending a second /v1;
• using a key from a different Provon project.

Discovery calls under /v1/gateway/* require workspace:read. Provider Key write calls require gateway:manage. These are separate from inference.

Gateway Returns UPSTREAM_NOT_CONFIGURED#

UPSTREAM_NOT_CONFIGURED means model resolution produced no eligible attempts. Check:

1. The request uses the intended form: <provider>/<model>, a configured plain model, or provon/auto.
2. The Provider Key is enabled and has a usable credential.
3. Specific mapping mode contains an enabled mapping for the request model.
4. The mapping allows the requested endpoint.
5. Required features such as streaming, tools, structured output, vision input, or audio are declared.
6. A Model Policy or Auto Router allowlist is not excluding the target.
7. Target health or circuit state has not removed every candidate.

Query the current project truth:

export PROVON_API_URL="https://api.provon.dev/v1"

curl "$PROVON_API_URL/gateway/capability-matrix?includeDisabled=false" \
  -H "Authorization: Bearer $PROVON_API_KEY"

curl \
  "$PROVON_API_URL/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"

These calls require workspace:read.

Gateway Returns 429#

Inspect:

• error.code: rate_limit_exceeded, gateway_budget_exceeded, or another project-state error;
• error.reason and error.policyId;
• Retry-After;
• RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset.

Do not automatically retry a cost or lifetime budget rejection. Retry a resettable rate limit only when the application can wait until the indicated window.

If error.code is INGESTION_SUSPENDED, resolve the project ingestion state instead of changing an API-key limit.

A Guardrail Returns 409#

An enforced guardrail block uses:

error.code: PROVON_GATEWAY_GUARD_TRIGGERED
x-provon-guard-triggered: true
x-provon-guard-plugin-id: <plugin>
x-provon-guard-action: <action>

Do not retry the same payload. Inspect the rule direction, category, match evidence, and action. Move a new rule back to Monitor while evaluating false positives. Keep application authorization separate from Gateway tool-shape policy.

The Provider Attempt Fails#

Open the request trace and inspect the child attempt:

• Provider Key and credential fingerprint;
• Gateway and upstream model IDs;
• upstream status and normalized error category;
• attempt timeout;
• retry count and fallback index;
• circuit-breaker or target-health events.

Typical actions:

Do not configure fallback for upstream authentication, permission, or malformed-request errors.

Streaming Stops Or Cannot Recover#

Test the same request with streaming disabled first. Then verify:

• the target declares streaming;
• the client accepts the provider-compatible event stream;
• reverse proxies do not buffer or terminate the stream;
• client, Provider Key, and overall Gateway deadlines are compatible;
• response guardrails have the intended streaming action.

Retry or fallback is only safe before response bytes have been committed. Once a stream has started, the client may receive a partial response instead of a transparent recovery.

The Model Call Succeeds But No Trace Appears#

Check the evidence path independently from inference:

1. Open Providers and set Request traces to All.
2. Confirm the request used the same project API key as the Workbench project.
3. Search Traces by timestamp and x-provon-request-id.
4. Wait for asynchronous delivery. In split-runtime deployments (the hosted edge and some self-hosted topologies) the Gateway hands captured evidence to a separate telemetry runtime. Traces commonly appear within seconds but can take up to a minute under load. Refresh before investigating deeper.
5. Verify the configured telemetry sink and delivery path.
6. Inspect delivery logs when traces continue to fail.

In Filtered mode, a successful request is persisted only when at least one filter matches. Off intentionally persists no Gateway trace.

The Trace Exists But Conversation Diagnosis Is Incomplete#

Verify:

• every turn uses the same x-otel-gen-ai-conversation-id;
• unrelated user goals do not share one conversation ID;
• agent IDs identify the actual participant;
• application tool, handoff, retrieval, and queue spans are emitted separately;
• the same model operation is not recorded by both Gateway and application instrumentation;
• sampling or filtered capture did not remove required turns.

Gateway capture sees model requests. It cannot reconstruct application operations that were never instrumented.

Escalation Record#

Preserve this non-secret bundle when escalating:

timestamp:
project ID:
request ID:
endpoint:
model:
HTTP status:
error code:
streaming:
Provider Key ID:
attempt error category:
trace URL or trace ID:

Remove prompts, responses, credentials, raw authorization headers, and personal data unless they are explicitly required and approved for the investigation.