Trace Diagnosis With CLI

trace source -> rule signals -> optional Agent review -> Finding candidates

In the reviewed-diagnosis profile, the calling Agent supplies semantic judgment through a hash-bound structured response for the candidates the rules surfaced.

Execution Boundary#

Source-based diagnosis:

• requires no Provon API key;
• does not upload the source;
• does not persist a Finding to the server;
• does not write to an external issue tracker;
• does not modify the repository or source files.

It returns evidence-backed Finding candidates on stdout. Server-side Rule Runs and the Workbench Findings inbox remain a separate remote workflow; see Findings CLI.

Prerequisites#

• A trace source: an exported bundle, transcript file/directory, OTLP file, span file, or stdin.
• For reviewed-diagnosis: the structured review response from the previous run's reviewQueue; rule-signals needs nothing further.

Supported Inputs#

provon diagnose <source>

Input formats (--input-format, default auto): auto, transcript, otlp-json, otlp-protobuf, provon-spans.

Automatic detection:

provon diagnose ./trace-evidence
provon diagnose ~/.claude/projects/example/session.jsonl
provon diagnose ./otlp-traces.json
provon diagnose ./traces.pb

Override detection for stdin or an ambiguous extension:

cat ./spans.jsonl | provon diagnose - --input-format provon-spans
provon diagnose ./capture.bin --input-format otlp-protobuf

Duplicate trace/span identity is deduplicated before diagnosis.

Built-in Rules#

All five built-in Rules run by default. Select one or more with repeated --rule:

provon diagnose ./trace-evidence \
  --rule runtime-reliability \
  --rule tool-correctness

Override publication thresholds for this execution (each between 0 and 1):

provon diagnose ./trace-evidence \
  --min-score 0.5 \
  --min-confidence 0.7

Diagnosis Profiles#

--mode rule-signals
--mode reviewed-diagnosis   (default)

rule-signals:

• returns only candidates supported without semantic review;
• emits no review request and makes no model call;
• is predictable for CI and high-volume screening;
• intentionally has lower recall for grounding, fulfillment, and causal questions.

The default reviewed-diagnosis profile can emit hash-bound review requests. --review-response is rejected in rule-signals mode (diagnose.review_response_without_review_mode).

Signal Review#

In reviewed-diagnosis, the first pass may return status: needs_review with a reviewQueue:

{
  "ok": true,
  "data": {
    "status": "needs_review",
    "findings": [],
    "reviewQueue": [
      {
        "schemaVersion": "provon-signal-review-v1",
        "diagnosisInputHash": "<hash>",
        "instructions": "<review instructions>",
        "evidenceBrief": "<evidence summary>",
        "signals": [],
        "evidence": [],
        "responseContract": {}
      }
    ]
  },
  "nextActions": [
    {
      "label": "Finalize with current Agent review",
      "command": "provon diagnose ./trace-evidence --review-response <response.json>"
    }
  ]
}

The request contains the only candidates eligible for confirmation. The reviewer judges whether each candidate is a genuine fault, the earliest uncorrected cause, and not corrected or compensated for by a later step. It must not invent a candidate or reuse trace/span/conversation ids or timestamps as signalSignature.

Response Shape#

Create one response per diagnosisInputHash; a single object or a responses envelope are both accepted:

{
  "responses": [
    {
      "schemaVersion": "provon-signal-review-v1",
      "diagnosisInputHash": "copy-from-the-request",
      "outcome": "failed",
      "confidence": 0.92,
      "signals": [
        {
          "failureModeKind": "goal_not_completed",
          "signalSignature": "copy-a-candidate-signature",
          "severity": "high",
          "confidence": 0.9,
          "title": "Required verification was skipped",
          "summary": "The agent claimed completion without running the required verification.",
          "rootCauseSummary": "The first uncorrected fault is the omitted verification step.",
          "suggestedFix": "Run and report the required verification before claiming completion.",
          "failedActor": "implementation-agent",
          "causeStep": "Claimed completion before verification",
          "causeStepIndex": 8,
          "genuineFault": true,
          "earliestUncorrected": true,
          "notCorrected": true,
          "causalCheckReason": "No later step performed or compensated for the verification."
        }
      ]
    }
  ]
}

Use values supplied by the request; an unknown failure mode or non-matching signature cannot confirm that candidate. Re-run with the response:

provon diagnose ./trace-evidence --review-response ./review-response.json

The command recomputes the diagnosis input hash; a changed trace, Rule, threshold, or evidence projection invalidates stale review.

Output#

Success envelopes are { ok, data, warnings, nextActions }.

Parse and rejected-record messages appear in warnings; when review is required, nextActions includes the follow-up command. Use --include-assessments to inspect promoted, suppressed, pending, and rejected candidates.

Success Check#

ok: true and data.summary.findingCount plus data.funnel show how many signals were promoted. With status: completed, act on data.findings; with needs_review, complete the reviewQueue response and re-run with --review-response.

CI Pattern#

provon diagnose ./artifacts/agent-trace.jsonl \
  --input-format provon-spans \
  --mode rule-signals \
  --min-score 0.5 \
  --min-confidence 0.7

CI consumes the structured JSON result; define repository-specific severity or count gates explicitly rather than relying on a built-in gate.

Evidence Requirements#

Diagnosis needs more than valid OTLP: a stable conversation boundary; the user's goal and constraints; ordered model, tool, retrieval, and participant activity; tool arguments, results, failures, and retries; a terminal answer or explicit non-completion; and trace/span ids for evidence references. Inputs with spans but no diagnosable trajectory fail with diagnose.no_trajectories.

Troubleshooting#