# Improve a model from trace evidence

The model improvement loop turns reviewed production behavior into Dataset examples, a fine-tuned
checkpoint, and a new Gateway target whose behavior can be observed again.

## Before you start

Dataset collection, review, and export work in both hosted and self-hosted Provon. Training a
fine-tuned checkpoint and deploying it behind a `self/` Gateway target require a Node deployment
with model inference and fine-tuning services. Self-hosted model execution and training are not
available in the Cloudflare runtime.

You need:

- diagnosis-ready conversations;
- a Dataset with reviewed chat examples (hosted or self-hosted);
- for training: a Node deployment, a base model supported by the local runtime, and the optional
  `services/python-inference` and `services/python-fine-tuning` services.

A runnable Dataset collection example is in
[`examples/dataset-collection`](https://github.com/provon/provon/tree/main/examples/dataset-collection).

## 1. Choose what to learn

Create a Dataset with one collection objective:

- preserve successful behavior;
- correct failed conversations;
- learn from user corrections;
- teach project knowledge.

Use filters to constrain the evidence when the objective supports them. Collection is based on
trace-backed producers; Findings and Knowledge remain independent domain artifacts rather than
Dataset source types.

See the [Dataset quickstart](../datasets/quickstart.md).

## 2. Review the examples

Inspect the input, expected output, optional rejected output, tags, and evidence provenance. Remove
examples that are unsupported, duplicated, private beyond the intended training boundary, or
inconsistent with the desired behavior.

Do not use example count alone as a readiness signal. A smaller coherent Dataset is preferable to a
larger contradictory one.

## 3. Train a checkpoint

Open model fine-tuning, select a base model and Dataset, then create an SFT job. Provon freezes the
selected examples into an immutable run manifest so later Dataset edits do not change the meaning of
the job.

Track the job until it completes and inspect its checkpoint lineage. See
[Model fine-tuning](../ai-models/fine-tuning.md) and [Self-hosted models](../ai-models/self-hosted-models.md).

## 4. Deploy through the Gateway

Deploy the checkpoint to the local model runtime. Once its inference service is healthy, route a
provider-qualified or project-owned logical model name to the new `self/` target.

Keep the previous target available until the new checkpoint passes representative requests. See
[AI Models](../ai-models/index.md) and [Gateway routing](../ai-gateway/routing.md).

## 5. Close the loop

Send new traffic through the deployed target and compare:

- task completion and Finding recurrence;
- latency, token usage, and errors;
- behavior on held-out evaluation examples;
- regressions outside the behavior represented in the training Dataset.

The new traces become evidence for the next review, repair, or training cycle.

## Example: collect a Dataset from traces

The [`examples/dataset-collection`](https://github.com/provon/provon/tree/main/examples/dataset-collection)
example emits an OpenTelemetry trace with a stable `conversation.id`, creates a Dataset with a
`preserve_successful_behavior` collection objective, and polls for produced Dataset examples. You
can adapt it to other collection objectives before moving on to training.