Skip to content

Telemetry Ingestion Boundaries

This page describes what can enter Provon telemetry, what each boundary observes, when an accepted payload is considered ready to read, and which retention and size choices users and operators control. It replaces the older internal storage walkthrough: implementation internals (queue names, storage adapters, and maintenance jobs) are documented for maintainers, not for operators of the public intake.

View as Markdown Open the plain-text version of this page.

For application instrumentation, start with Tracing and OpenTelemetry setup. The wire contract lives in the OTLP/HTTP API; query surfaces in the Trace read API and Telemetry Read API.

Three ingestion boundaries#

AI Gateway proxy Provon intake OTLP/HTTP exporter CLI hooks and transcript sync Traces, Sessions, Users, Findings
Boundary Observes Does not observe Prerequisites
AI Gateway proxy Real upstream calls: request, response, provider attempts, latency, tokens, cost, guardrail decisions, error categories Internal tool loops, queue steps, verification spans the Gateway never executes Gateway base URL and key; capture mode enabled
OTLP/HTTP exporter Whatever the SDK or Collector emits: spans, logs, metrics with recognized attributes Guardrail decisions; provider attempt details unless the exporter records them telemetry:ingest key; OTLP/HTTP target
CLI hooks and transcript sync Persisted or streamed agent session events: messages, tool calls, results, usage, session identity Upstream provider response bodies beyond the transcript; guardrail facts Authenticated CLI; local transcript files or live hook support

All three converge on the same project-scoped trace records, but each boundary's evidence differs. A Finding based on a transcript record cannot claim a Gateway guardrail fired, and a Gateway-only trace cannot show an internal tool step.

Accepted signals and endpoints#

Endpoint Signal Notes
POST /v1/traces Traces Primary signal for the Workbench, Sessions, and diagnostic Rules
POST /v1/logs Logs Optional; listed and queried through the Telemetry Read API; do not enrich the span trajectory
POST /v1/metrics Metrics Optional; aggregate usage and activity queries

A runtime that has not enabled a signal rejects it with 501 OTLP_SIGNAL_NOT_ENABLED. OTLP/gRPC is not exposed; accepted transport encodings are identity, gzip, and deflate, with JSON or protobuf payloads.

Acceptance versus readiness#

Stage Meaning Evidence
Accepted Decoded, authorized, staged, and queued; producer receives 200 { queued: true, jobId } Not yet queryable
Processed Records normalized, projected, and indexed; trace summaries refreshed Records appear in Traces / Sessions
Readable Summary-backed list, search, and detail views return the record Verified in the Workbench or CLI read-back

A 200 therefore never means "visible now." Summary materialization runs asynchronously; if an accepted payload never appears, use the intake failure checklist in OpenTelemetry setup and the troubleshooting guide.

Intake response codes#

Code Meaning Producer action
200 Staged and queued Do not resend the same batch
401 / 403 Invalid key or missing telemetry:ingest Fix credentials; do not retry unchanged
402 INGESTION_SUSPENDED Project ingestion suspended Resolve the suspension
413 PAYLOAD_TOO_LARGE Encoded declaration or decoded stream over the limit Reduce exporter batch size
422 VALIDATION_FAILED Unsupported content type, encoding, or protobuf capability Fix exporter configuration
501 OTLP_SIGNAL_NOT_ENABLED Runtime does not accept that signal Enable it or stop sending it
503 OTLP_STAGING_UNAVAILABLE Staging temporarily unavailable Retry with exponential backoff
503 INGEST_QUEUE_BACKPRESSURE Queue admission saturated Honor Retry-After and back off

Delivery is at-least-once at the transport layer; records are deduplicated by stable identity (payload hash for OTLP batches, source/session-derived IDs for transcripts), so safe retries do not create duplicate traces.

Size and payload boundaries#

Bound Default Notes
Decoded OTLP request body 10 MiB Applied after gzip/deflate decompression; a small compressed request can still exceed it
CLI sync upload batch 8 MiB (--max-payload-bytes) Configurable per sync run
Transcript string budget 32,000 chars (truncated); 1,000 (none); 256,000 (full) See Agent transcripts
Spans per trace read 10,000 Exceeding a budget returns an explicit error; narrow the range
Events per trace read 10,000 Same budget rule
Bytes per trace read 16 MiB Same budget rule
Candidate traces per summary query 10,000 rows / 16 MiB Same budget rule

Retention choices#

Two independent decisions control what survives:

Decision Question it answers Setting / control
Payload retention Are raw message, tool, and provider bodies retained, or only projected fields? PROVON_TELEMETRY_PAYLOAD_RETENTION=all (default) or projected-only; Cloudflare/Worker deployments only — Node deployments keep all with no toggle
Data retention How long projected telemetry rows remain queryable Resolved from the project plan and deployment configuration; rows expire by UTC event_date
  • Worker: all retains projected query fields plus raw payload evidence needed for span detail and diagnosis; an unset variable defaults to all, and an invalid value fails startup.
  • Worker: projected-only retains projected fields used by lists, search, summaries, dashboards, and usage queries, but not full raw bodies; it cannot be upgraded back to raw bodies after intake.
  • Node deployments have no projected-only toggle: the store defaults to retaining raw payload bodies under the Node retention policy regardless of this variable.
  • Transient staged upload payloads are removed after successful processing or after a job fails permanently; they are not a long-term store.
  • Test enabled Rules against the chosen payload mode before production rollout: projected-only retention can remove the exact tool result a Rule depends on.

Read contract#

  • Trace list and search default to the last 24 hours when no range is supplied; logs and metrics require an explicit named range or startMs/endMs.
  • Trace, conversation, and user lists are keyset-paginated; use cursors rather than offsets.
  • Conversation and user views are derived from trace summaries at query time; a missing summary row yields an empty read model, not a fallback raw scan.
  • Public read routes require telemetry:read where API-key capabilities apply; see Trace read API and Telemetry Read API.

Failure semantics users should rely on#

  • Retry 503 responses with backoff and Retry-After; never retry 401, 402, 403, 413, 422, or 501 without changing configuration.
  • One failed record or job does not block unrelated records in the same batch.
  • Transcript sync skips unchanged files by content hash and reuses stable trace and span IDs on re-run.
  • Guardrail summaries are interpreted from recorded events; missing evidence never counts as a hit, and a healthy conversation with no failures yields no Finding.

Limitations#

  • This page states user-visible boundaries, not storage internals; deployment-specific storage and queue bindings belong to the self-hosting and operator documentation.
  • Boundary coverage is additive: combining Gateway, OTLP, and CLI records does not fill gaps each source left empty.
  • Read budgets are safety limits, not a guarantee that a single long session should be treated as one trace; keep turns bounded per the trace model.