Diagnostic Rules

Provon ships five built-in Diagnostic Rules. Each Rule detects a family of failure modes, applies thresholds, and decides which signals require model adjudication.

View as Markdown Open the plain-text version of this page.

Rules are configured per project. You can enable or disable each Rule, change its thresholds, and override individual signal categories.

Built-In Rules#

Rule	Key	Detects
Runtime reliability	`runtime-reliability`	Runtime errors, incomplete termination, and false success after execution failures
Tool correctness	`tool-correctness`	Failed tool loops, ignored negative results, and incorrect tool-result interpretation
Task fulfillment	`task-fulfillment`	Missing required actions, skipped verification, constraint failures, and premature completion
Answer grounding	`answer-grounding`	Unsupported final answers and unsupported external claims
Conversation health	`conversation-health`	User corrections, dissatisfaction, runaway context growth, and failure to converge

Configuration Options#

Option	Default	Description
`keywords`	`[]`	One per line. When non-empty, the Rule only evaluates conversations containing at least one keyword. Matching is case-insensitive over captured conversation text.
`minSignalScore`	`0.25`	Minimum signal score for a signal to enter the diagnosis pipeline. Signals below this are discarded before adjudication and causal grouping.
`minDiagnosticConfidence`	`0.25`	Minimum diagnostic confidence required to publish a Finding candidate. Candidates below this are filtered out.
`adjudicationMode`	`auto`	`auto` confirms deterministic signals directly and escalates judgment signals to a model. `deterministic-only` makes no model calls.
`adjudicationModel`	`null`	Uses `provon/auto` by default. Select a connected model when `adjudicationMode` is `auto`.
`signalOverrides`	`{}`	Per-category overrides keyed by category key. Each override can set `enabled: false` or change `severity`.

Signal Categories By Rule#

A category marked requires adjudication is only published in auto mode. In deterministic-only mode it is suppressed.

If the selected model is unavailable in auto mode, deterministic detection continues and Workbench reports the feature as Limited. Signals that require model adjudication are not promoted until a compatible model is configured.

Runtime reliability#

Category	Family	Default severity	Requires adjudication
`runtime_error`	runtime	high	no
`explicit_non_completion`	conversation	high	no
`no_terminal_answer`	runtime	medium	no
`unsupported_final_claim`	answer	high	yes

Tool correctness#

Category	Family	Default severity	Requires adjudication
`repeated_failed_tool_use`	tool_use	high	no
`tool_result_misread`	tool_use	high	no
`negative_tool_result_ignored`	tool_use	medium	no
`policy_precondition_skipped`	planning	high	no
`non_convergence`	conversation	medium	no

Task fulfillment#

Category	Family	Default severity	Requires adjudication
`required_action_missing`	planning	medium	no
`goal_not_verified`	planning	medium	yes
`policy_precondition_skipped`	planning	high	no
`subagent_handoff_unfulfilled`	coordination	high	no
`plan_step_not_executed`	planning	high	no
`premature_finalization`	planning	medium	yes

Answer grounding#

Category	Family	Default severity	Requires adjudication
`unsupported_final_claim`	answer	high	yes
`unsupported_external_claim`	answer	medium	no

Conversation health#

Category	Family	Default severity	Requires adjudication
`user_explicit_correction`	conversation	medium	no
`implicit_user_dissatisfaction`	conversation	medium	yes
`context_growth`	conversation	low	no
`non_convergence`	conversation	medium	no

How To Tune#

Reduce false positives#

Increase minDiagnosticConfidence for the Rule.
Disable noisy categories with signalOverrides.
Add keywords so the Rule only runs on relevant conversations.

Example API patch:

{
  "configuration": {
    "minDiagnosticConfidence": 0.7,
    "keywords": ["payment", "refund"],
    "signalOverrides": {
      "implicit_user_dissatisfaction": { "enabled": false },
      "context_growth": { "severity": "info" }
    }
  }
}

Reduce cost in high-volume pipelines#

Switch to deterministic-only. This skips all model adjudication and publishes only signals that can be confirmed from structured evidence. Expect lower recall for categories like unsupported_final_claim and implicit_user_dissatisfaction.

CI screening#

Use provon diagnose with --deterministic-only for local or CI checks. Server-side Rules can stay in auto mode for persisted Findings.

Severity Levels#

Severity	When to use
`critical`	Agent produced a wrong or harmful outcome with high confidence.
`high`	Clear failure mechanism that likely affects user value.
`medium`	Detected problem that may be recoverable or context-dependent.
`low`	Minor degradation, noise, or best-practice violation.
`info`	Observation that does not require immediate action.