Diagnostic Rules

The unified finding taxonomy#

The catalog uses one set of 14 finding types; a detected type is exported verbatim with no remapping layer. Each type sits on three orthogonal axes:

• type — what went wrong (the user-facing defect class);
• stage — where in the agent loop it surfaced (6 stages);
• attribution — who is responsible: agent, environment, or input.

A weak causal role prior (root / symptom) seeds root-cause ranking; actual causality comes from detector-declared edges and shared evidence. The Detector column distinguishes types a deterministic family can close (signal) from types reachable only through model discovery (review).

Stages, in agent-loop order: understanding, planning, execution, verification, final-answer, runtime.

Built-in Rules#

capability-unavailable is not bound to a built-in Rule's signal set; it can be proposed by model discovery (whole_trajectory policy) and reviewed like any other candidate. Signal overrides apply only to types the Rule actually detects; overrides for unknown types are ignored.

What the pure-rule sweep can recall#

On the no-model rule-signals profile (provon diagnose --mode rule-signals), a candidate publishes only when a detector closes it from structured or lexical evidence. The deterministic sweep covers:

• safety-policy-block — an explicit provider content-filter error (content filter, Azure content-management policy, jailbreak detected) or a self-reported rendered gate (anti-bot challenge page).
• repetition-no-progress — the same operation repeated past a threshold, or a blind run of consecutive operations returning no usable observation.
• incoherent-answer — a terminal answer that is empty, garbled, or an uncontextualized bare value with no sentence structure; a surface-form test, so a task that genuinely needs a bare value is not a defect.
• unsupported-claim, invalid-tool-precondition, tool-result-misread, plan-step-omitted, constraint-violated, and capability-mismatch in their high-confidence closed cases.

The remaining types require semantic or causal inference and are reachable only via model discovery or review: goal-misunderstood, retrieval-insufficient, input-underspecified, capability-unavailable, and the bulk of tool-result-misread. A budget marker ("max time/rounds reached") is detected, but whether it is the independent root cause versus a symptom needs attribution, so it is reviewed rather than auto-published.

Configuration options#

A Rule PATCH sends a complete configuration object (it replaces the stored configuration rather than merging), so include every key together. Production Rules always run the reviewed-diagnosis profile; the no-model deterministic screen is only available locally through provon diagnose --mode rule-signals.

Signal triage and admission#

Each signal passes two deterministic layers. The detector-layer triage reads only the signal's own evidence and uses no review vocabulary:

• admissible with an evidence kind:
  • mechanical_predicate — the Rule can establish the finding mechanically;
  • verifiable_predicate — the finding holds once its deterministic proof closes;
  • lexical_lead — a concrete lexical seed the Rules cannot close themselves.
• proof_not_closed — the deterministic proof for a verifiable_predicate signal does not close.
• withheld with an objective mechanical reason (below threshold, recovered, no precise evidence, no terminal impact, and so on).

The outer admission layer applies runtime policy:

• model review enabled: every admissible signal goes to the reviewer (mechanical/verifiable ones for confirmation; lexical leads for closure judgment);
• no-model rule-signals profile: mechanical_predicate and closed verifiable_predicate signals publish as deterministic; a lexical_lead is withheld as model_review_unavailable.

Evidence requirements apply per review family (the signal's evidence-collection strategy), not per type. Production is fail-closed: no available model withholds as model_review_unavailable; a throwing call or execution failure becomes model_review_error; a model returning no definite verdict becomes model_review_inconclusive. No candidate is promoted, and the run retains audit counters.

Publication funnel#

Every diagnosis reports an unambiguous, model-first funnel returned on the discovery API, printed by provon diagnose (funnel), emitted as diagnostics.* span attributes, and aggregated in eval metadata.

Reconciliation: under reviewed mode, review.acceptedCount = publication.publishedCount + review.postGateRejectedCount. A one-event lexical lead without corroboration is withheld at the mechanical gate as insufficient_corroboration (weakLeadWithheldCount); it differs from model_review_unavailable, the fail-closed withhold of a candidate the review boundary never closed.

How to tune#

Reduce false positives#

1. Increase minDiagnosticConfidence for the Rule.
2. Disable noisy finding types with signalOverrides.
3. Add keywords so the Rule only runs on relevant conversations.

Example API patch (the whole configuration is replaced):

{
  "configuration": {
    "keywords": ["payment", "refund"],
    "minSignalScore": 0.25,
    "minDiagnosticConfidence": 0.7,
    "reviewerDiscoveryPolicy": "candidates_only",
    "reviewModel": null,
    "signalOverrides": {
      "repetition-no-progress": { "enabled": false },
      "incoherent-answer": { "severity": "info" }
    },
    "lexicon": {},
    "detectorSettings": {}
  }
}

Control cost in high-volume pipelines#

The default candidates_only policy bounds cost: a clean conversation where no Rule fired makes no model call; any conversation with a candidate is still reviewed.

• Switch to whole_trajectory only when you want issues the Rules never hinted; it makes one model call on every conversation, including clean ones.
• Production Rules always run the reviewed-diagnosis profile; there is no server-side no-model option. Use the local rule-signals profile for deterministic screening.

CI screening#

provon diagnose --mode rule-signals skips the model and publishes only signals confirmable from structured evidence; expect lower recall for review-gated types such as unsupported-claim. Server-side Rules stay on the candidates_only default for persisted Findings.

Severity levels#

Limitations#

• Review-only types never publish under the local deterministic profile; absence locally is not absence on the reviewed server profile.
• Keyword filters narrow scope but also suppress every type on unmatched conversations.
• A high-confidence deterministic signal still follows the Rule's configured confidence and publication gates.