Skip to content

Troubleshooting

Start with the first failed verification step. A Gateway response, an ingested trace, a diagnostic Run, a Finding, and a GitHub handoff are separate stages.

View as Markdown Open the plain-text version of this page.

Success Check (healthy state)#

Work through this list top to bottom; a healthy deployment satisfies every item:

  • Sign-in succeeds and the selected project exists in the Workbench.
  • Gateway requests return OpenAI-shaped responses, or OTLP exports return 200/accepted.
  • Traces lists recent evidence with a stable conversation id.
  • Enabled Rules either publish candidates or stay silent on healthy traffic.
  • Handoff items exist in the tracker and resolve back to the Finding after closure.

Quickstart and onboarding#

I cannot sign in or create a workspace#

  • For hosted Provon, confirm you are at https://app.provon.dev.
  • For self-hosted, confirm the server is running and the Workbench origin matches the configured PROVON_WORKBENCH_ORIGINS.
  • If you are behind a reverse proxy, forward the original Host and X-Forwarded-Proto headers.

The onboarding Discover step finds no trace#

  1. Complete the Connect step and run one agent call or sync first.
  2. Confirm the API key in the snippet matches the current project.
  3. For local development, confirm PROVON_API_URL points to the API port (3001 by default), not the Workbench port (3000).
  4. If the trace does not appear in Traces, see the path-specific sections below.

I installed the CLI but provon is not found#

If you used the self-hosted installer, either:

  • reopen your shell so the PATH change takes effect, or
  • source the installer-generated env file before running commands.

If you are working from the source repository, run the CLI through node:

bash
pnpm --filter @provon/cli build
node cli/dist/bin.js --help

The Workbench loads but API calls fail#

  • Read the pnpm dev output for the Workbench and API origins.
  • Confirm PROVON_API_ORIGIN was not overridden to an unavailable server.
  • When using a reverse proxy, forward the original host and scheme.
  • Confirm the signed-in user belongs to the requested project.

Provon does not start#

Check:

bash
node --version
pnpm --version
pnpm dev
  • Node.js must be 22 or newer.
  • The local Workbench port defaults to 3000; the development API uses a separate internal port.
  • Another process or Provon instance must not hold the configured DuckDB file lock.
  • Relative runtime storage paths resolve from the repository root.

Use a custom Workbench port when needed:

bash
pnpm dev -- --port 3100

Installer server management#

  • The installer provides no provon-server stop subcommand. Stop a foreground process with Ctrl+C; stop a --daemon service with launchctl unload (macOS) or systemctl --user stop provon-server (Linux).
  • Start a manual foreground session in a new shell: source ~/.provon/env then ~/.provon/bin/provon-server. On Windows, run env.ps1 and provon-server.cmd in a separate PowerShell window; the Windows installer registers no service.
  • Service logs live under <data-dir>/logs; data files live under <data-dir> and blobs under <data-dir>/blobs.
  • On Linux, a user service that exits after logout needs loginctl enable-linger <user>.

OTLP returns 401 or 403#

Verify the header format:

bash
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $PROVON_API_KEY"

The key must be a project API key with telemetry:ingest. Provider API keys and GitHub tokens are not valid OTLP credentials.

OTLP succeeds but no trace appears#

Check these in order:

  1. The exporter uses OTLP/HTTP, not OTLP/gRPC.
  2. The endpoint is the Provon origin, or the signal-specific URL ends in /v1/traces.
  3. The exporter is flushed before the process exits.
  4. The trace is in the same project as the API key.
  5. The selected Workbench time range includes the span timestamps.

On deployments that stage ingestion asynchronously (for example the Cloudflare runtime), a 200 means the payload was durably staged and queued; trace summaries can appear after the raw write completes. The background worker parses and validates individual records after the response. If a known trace never appears, inspect ingest-worker and dead-letter logs for malformed OTLP envelopes or rejected records.

OTLP returns 413, 422, 501, or 503#

  • 413: reduce the exporter batch size. The default decoded request limit is 10 MiB.
  • 422: use a supported content type and content encoding, and confirm protobuf decoding is configured.
  • 501: the deployed runtime does not enable the requested signal.
  • 503: retry with exponential backoff and honor Retry-After when queue admission is saturated.

Do not retry authentication or validation errors without changing the request.

See the OTLP/HTTP API for the complete acceptance and retry contract.

Traces appear without conversation grouping#

Add a stable gen_ai.conversation.id to the agent spans. Reusing a trace ID is not a substitute for conversation identity because one conversation may contain multiple traces. session.id is retained as an attribute but is not the primary Provon conversation key.

Also verify:

  • the identifier is attached consistently across turns;
  • timestamps and parent span IDs preserve event order;
  • user, assistant, and tool content was retained when diagnosis requires it.

Provon also recognizes conversation.id, conversation_id, provon.conversation_id, thread.id, and thread_id.

A trace is visible but not diagnosable#

Trace visibility only requires valid spans. Diagnostic Rules need a reconstructable trajectory. Verify that the trace includes:

  • a bounded root operation and ordered child spans;
  • the user goal in gen_ai.input.messages;
  • the terminal answer in gen_ai.output.messages;
  • tool call IDs, arguments, results, failures, and retries;
  • error span status plus error.type;
  • raw evidence allowed by the payload-retention policy.

See Trace model and instrumentation. Use the diagnosis-ready tracing checklist to review the complete run.

Trace search misses an older trace#

The CLI traces list and traces search default to a 1h range; pass a wider range explicitly:

bash
provon traces list --range 7d

In the Workbench, widen the range picker in the Traces filter bar before assuming the data is missing.

Content, input, and output search uses projected previews and remains available with projected-only retention. It can match only the retained previews; the full raw body is unavailable for later inspection.

Agent transcript sync finds no files#

  • Point provon ingest sync at a directory containing .json or .jsonl transcripts.
  • Use --once --dry-run --print-sample to inspect detection without uploading.
  • Confirm the process can read nested source directories.
  • Wait for an actively written JSONL record to finish; the scanner ignores an incomplete final line until the file has been quiet long enough.
  • Inspect the command's failures and warnings arrays.

AI Gateway#

Use Gateway troubleshooting for the request-ordered diagnostic path:

  1. verify project API-key authentication and capability;
  2. inspect model resolution and project policy;
  3. inspect the selected provider attempts;
  4. verify Trace Capture and telemetry delivery.

It covers 401, 402, 403, 404, 409, 413, 415, 429, and 503 responses, streaming failures, provider errors, missing traces, and incomplete conversation evidence.

No Findings appear from automatic diagnosis#

This can be correct. Healthy conversations and weak candidates should not create Findings.

Verify:

  1. At least one Rule is enabled in Findings → Rules. Automatic diagnosis is off only when every Rule is disabled.
  2. The relevant Rules are enabled.
  3. The conversation has been inactive for at least 10 minutes.
  4. A stable conversation identifier and usable trajectory evidence exist.
  5. Rule keywords match the captured conversation text.
  6. Signal and diagnostic-confidence thresholds are not filtering the candidate.
  7. Model review has a usable deployment-default review model (or the Rule's configured reviewModel).

To separate scheduling from detection, open Rules and run an enabled Rule through the backfill dialog (range presets: last 24 hours, 7 days, 30 days, 90 days, or all time; default 30 days). Use provon diagnose <trace-or-transcript-path> --mode rule-signals when testing locally without a review model; that profile never performs semantic review and returns only deterministic Findings.

A Rule Run is skipped#

Common skip reasons:

  • the Rule was disabled;
  • every Rule was disabled, so automatic scheduling was off;
  • the conversation no longer exists;
  • newer activity superseded the queued snapshot;
  • no usable trajectory could be reconstructed;
  • configured keywords did not match.

Skipped Runs do not publish Findings.

Create repair issue fails#

Repair handoff requires an enabled issue-tracker connector. Provon supports GitHub, GitLab, Linear, and Jira.

Check:

  • the issue tracker of your choice on the Findings page is enabled;
  • a token or OAuth connection is stored;
  • the required default destination is set (for GitHub/GitLab an owner and repository/project path; for Jira a project; for Linear a team and project);
  • the token can create issues in that destination;
  • the deployment has an integration secret for encrypted connector credentials.

For GitHub and GitLab OAuth, configure the matching integration client ID and secret. For local testing, expand the connector and provide a token manually.

A Finding is handed off but never resolves#

Provon stores a provider-specific external reference (for example, owner/repo#123 for GitHub or GitLab, or a Jira/Linear issue key). Resolution sync needs:

  • the connector to remain enabled;
  • permission to read the work item and its associated pull requests or linked changes;
  • the background repair-status loop to be running;
  • a merged or completed change associated with the work item, or the work item itself being closed or moved to a done status.

Closing a Finding in the Workbench is not a substitute for the external repair outcome, and an external item being closed is not, by itself, proof that the agent task improved.

Seed data fails#

pnpm seed:demo intentionally requires empty metadata, metering, and telemetry databases. It will not merge demo data into an existing workspace.

Stop the server before seeding so DuckDB is not locked. Use a separate configured database if you need to preserve existing data.

Still Blocked#

Capture:

  • the failing command or HTTP method and path;
  • the response status, error code, and request ID;
  • the runtime shape (Node or Cloudflare);
  • the relevant project ID;
  • whether the previous stage in the loop succeeded.

Do not include project API keys, provider credentials, GitHub tokens, prompts, or private payloads in a public issue.