Self-hosting Operations

Define Service Objectives#

Set explicit targets for:

Derive alert thresholds and capacity from these targets; do not infer them from default polling intervals.

Health Checks#

Public request-facing deployments expose GET /healthz with runtime status, version, and deployment target:

curl --fail --silent https://provon.example.com/healthz

Use it for the Node process and the API, Gateway, MCP, and OTLP Workers. Background-only deployments differ: the telemetry pipeline, diagnostics, and maintenance Workers return a plain healthz: ok response, and the warehouse Worker exposes GET /health.

Health checks do not prove that metadata queries work, telemetry can be written or read, queues are draining, a model provider is reachable, diagnostics are completing, or a Connector can perform its external action. Add synthetic checks with a dedicated project:

1. Query an authenticated API endpoint.
2. Send a small OTLP trace with a unique trace ID.
3. Poll until the trace is queryable within the evidence-visibility objective.
4. Send a low-cost Gateway canary and correlate x-provon-request-id to its trace.
5. Periodically run a deterministic diagnostic Rule over known evidence.

Keep synthetic data clearly labeled and outside production diagnostic inputs when possible.

Monitor The Pipeline#

Node#

Capture stdout and stderr from the Node process. Alert on repeated error labels for:

• [ingest-worker];
• [gateway-telemetry-worker];
• [telemetry-summary-materialization];
• [diagnostics];
• [finding-work-item-sync];
• [data-retention];
• [meta-store-maintenance];
• [gateway-usage-reconciliation].

Also monitor process restarts, filesystem free space, file-descriptor usage, memory, CPU, and DuckDB query latency.

Cloudflare#

Monitor each deployment independently:

• API, Gateway, OTLP, MCP, pipeline, diagnostics, and maintenance Worker errors;
• Queue backlog, oldest message age, retry exhaustion, and the ingest DLQ;
• Maintenance cron trigger success;
• Container starts, failures, and write latency;
• R2 SQL query errors;
• Data Catalog append, compaction, snapshot-expiration, and retention failures;
• D1, R2, KV, Durable Object, Worker, Queue, and Container usage.

A healthy API Worker can coexist with a stalled background Worker. Alert on the oldest unfinished work, not only on request error rates.

Back Up The Recovery Set#

Define one coordinated recovery point that includes:

1. Metadata;
2. Telemetry;
3. Blobs and staged payloads;
4. Persistent queue or runtime state;
5. PROVON_AUTH_SECRET and required storage credentials;
6. Deployment configuration and application version.

Store the secret recovery copy separately from encrypted metadata, and test restoration in an isolated environment on a schedule.

Node Backup#

For local SQLite and DuckDB, use a quiesced backup:

1. Stop new Gateway and OTLP traffic at the ingress.
2. Allow in-flight requests and background work to finish.
3. Send SIGTERM and wait for graceful shutdown.
4. Snapshot the complete persistent data directory.
5. Record the Provon version and configuration revision.
6. Restart and run the normal verification checks.

Include:

• the file behind PROVON_META_DB_URL;
• the file behind PROVON_METERING_DB_URL;
• the file behind PROVON_GATEWAY_DB_URL;
• the file behind PROVON_JOBS_DB_URL when queued work should survive the restore; it holds only in-flight background jobs, so restoring without it loses queued work and nothing else;
• the DuckDB file when local;
• PROVON_BLOBS_DIR and its parent runtime-state files.

Use the libSQL provider's consistent export or snapshot mechanism for remote metadata and usage-event stores. Do not copy a live DuckDB file from storage with uncertain snapshot semantics.

Cloudflare Backup#

Treat each managed service according to its recovery model:

• export D1 metadata and usage events before upgrades and test D1 point-in-time recovery procedures;
• retain the Wrangler configs, Durable Object migration history, and resource IDs;
• protect R2 blobs and the Data Catalog bucket with account-level access and lifecycle policy;
• preserve Data Catalog maintenance settings;
• record Queue and DLQ state expectations in the recovery plan;
• keep deployment and secret inventory outside the Cloudflare account for account-recovery events.

Create a pre-upgrade D1 export:

pnpm exec wrangler d1 export provon-demo-meta \
  --remote \
  --output provon-demo-meta-pre-upgrade.sql \
  --config services/cloudflare/wrangler.api.toml

pnpm exec wrangler d1 export provon-demo-metering \
  --remote \
  --output provon-demo-metering-pre-upgrade.sql \
  --config services/cloudflare/wrangler.maintenance.toml

R2 Data Catalog tables are transactionally managed Iceberg data. Do not manually delete or rewrite catalog objects as a backup or cleanup method; doing so can leave table metadata inconsistent.

Restore#

Restore into an isolated origin first:

1. Restore metadata with the original PROVON_AUTH_SECRET.
2. Restore usage events to the matching recovery point.
3. Restore telemetry and blobs to the matching recovery point.
4. Restore Node runtime state or recreate Cloudflare bindings and queues.
5. Start the same application version that created the backup.
6. Verify sign-in, project membership, API keys, stored provider credentials, traces, Findings, and Connector decryption.
7. Upgrade only after the recovered version is healthy.
8. Switch traffic after the end-to-end synthetic checks pass.

If only metadata is restored, trace and attachment references may be missing. If only telemetry is restored, tenant ownership and access records may be missing. Document which degraded restore modes are acceptable.

Upgrade Node#

Use an immutable release or container tag.

1. Read release and migration notes.
2. Build the new Node image or source artifact.
3. Run the Node server tests and relevant workspace checks.
4. Quiesce traffic and create the recovery set.
5. Stop the old process gracefully.
6. Start one new process against the persistent data.
7. Let startup migrations complete.
8. Verify health, sign-in, API, OTLP, Gateway, traces, diagnostics, and Connectors.
9. Retain the prior artifact until the release is accepted.

Do not run old and new Node processes concurrently against one local data directory during a rolling upgrade.

Upgrade Cloudflare#

1. Read release and migration notes.
2. Run pnpm test:cf.
3. Export D1 metadata and usage events.
4. Apply any required Data Catalog maintenance change.
5. Run pnpm release:build && pnpm release:check && pnpm release:publish.
6. Verify each deployment and the complete asynchronous evidence path.
7. Watch queue lag, D1 errors, warehouse writes, and R2 SQL reads through the rollout window.

Application publishing does not run D1 initialization; prepare the current baseline explicitly with pnpm release:db-init --remote. The project keeps only the current initialization baseline: no historical data migrations, old schema compatibility, or release aliases are maintained. Application rollback does not restore D1, R2, Queue, or Durable Object state; preserve an isolated recovery copy before any live schema change and verify recovery separately.

Capacity And Scaling#

Node#

Scale vertically and control evidence volume: CPU and memory for request handling and DuckDB queries; disk capacity and IOPS for metadata, telemetry, blobs, and queues; body and batch limits for ingest; trace capture and payload retention; and background loop batch sizes. When one process no longer meets availability or throughput objectives, move to shared adapter infrastructure or the Cloudflare runtime. Starting more default Node replicas creates independent local work queues and schedulers.

Cloudflare#

Scale each constrained service: Gateway Worker concurrency and provider deadlines; OTLP admission and Queue backlog; consumer concurrency and batch size; warehouse Container instances and append coalescing; R2 SQL read latency and summary freshness; D1 and Durable Object hot keys; Data Catalog small-file count and compaction. Tune from platform telemetry; raising producer admission without increasing consumer and warehouse capacity only moves the bottleneck into queue age.

Retention Operations#

Verify three mechanisms independently:

Retention is not backup expiration: keep recovery copies under a separate policy and document when they are destroyed.

On Cloudflare, retention runs inside the Maintenance Worker cron trigger (0 */6 * * *, every six hours) alongside metadata maintenance and work-item sync. On Node, the retention job runs on the interval set by PROVON_DATA_RETENTION_INTERVAL_MS (default six hours).

Run the maintenance configuration in dry-run mode after changes:

pnpm configure:cf:telemetry-maintenance

Defaults behind the script: OTLP and Gateway staging objects expire after 7 days; Data Catalog snapshot expiration keeps at least a 15-day window (required for queue-replay idempotency) and retains the last 10 snapshots; compaction targets 64 MB files. Do not shorten the snapshot window independently.

Incident Procedures#

Gateway Incident#

1. Preserve x-provon-request-id and the failing request shape.
2. Pause affected project traffic or route the application back to its prior provider endpoint.
3. Check provider attempts, guardrail events, and usage reservations.
4. Keep captured evidence for diagnosis.

Ingest Backlog#

1. Separate acceptance errors from post-acceptance visibility lag.
2. Measure oldest pending work and dependency errors.
3. Stop increasing admission limits.
4. Restore the writer, telemetry store, or queue consumer.
5. Verify replay and deduplication before clearing dead letters.

Credential Exposure#

1. Revoke the exposed workload, provider, Connector, storage, or deployment credential at its authority.
2. Disable the affected integration or project path.
3. Search logs and trace evidence for accidental secret capture.
4. Replace the credential and verify the new path.
5. Treat PROVON_AUTH_SECRET exposure as deployment-wide encrypted-credential exposure and plan a full credential reauthorization or re-encryption event.

Storage Recovery#

1. Stop writes.
2. Preserve the current failed state for investigation.
3. Restore into an isolated environment.
4. Validate tenant ownership and cross-store references.
5. Resume traffic only after end-to-end checks pass.

Release Gate#

• Recovery set and restore procedure are current.
• Health, queue, background job, storage, and provider alerts are active.
• Synthetic API, OTLP, and Gateway checks pass.
• The new version and config revision are recorded.
• Current initialization baseline prepared explicitly before the first release.
• Evidence visibility and diagnostic latency remain within objectives.
• Retention and Data Catalog maintenance completed successfully.
• The previous artifact and rollback procedure remain available.