# Datasets

Provon Datasets turn reviewed agent behavior into structured Examples for evaluation, fine-tuning,
and export. Every automatically collected Example retains evidence provenance so its expected
behavior can be audited.

Datasets are consumption-neutral. Creating a Dataset does not commit it to one training engine,
evaluation framework, or model provider.

## When To Use A Dataset

Use a Dataset when you want to:

- preserve production behavior that should not regress;
- turn diagnosed failures into corrected examples;
- learn from explicit user corrections;
- teach a model reviewed project Knowledge;
- maintain one versionable source for evaluation and fine-tuning.

## Core Concepts

| Concept               | Responsibility                                                           |
| --------------------- | ------------------------------------------------------------------------ |
| **Dataset**           | Owns a schema, collection policy, state, and directly owned Examples     |
| **Example**           | Stores model-facing input, expected output, and optional preference data |
| **Collection policy** | Defines which behavior the Dataset should collect                        |
| **Provenance**        | Links an Example to traces, conversations, imports, or manual input      |
| **Run manifest**      | Freezes selected Examples for a particular evaluation or training run    |

## Collection Objectives

Each automatically collecting Dataset has one objective:

| Objective                      | Intended behavior                                              | Best for                                                                |
| ------------------------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `preserve_successful_behavior` | Retain verified conversations that already work                | Keeping high-quality production behavior from regressing                |
| `correct_failed_conversations` | Pair diagnosed failures with corrected expected responses      | Turning Findings into supervised corrections                            |
| `learn_from_user_corrections`  | Preserve rejected answers and verified user-driven corrections | Capturing explicit thumbs-down / follow-up corrections                  |
| `teach_project_knowledge`      | Build examples grounded in selected Learning Goals             | Teaching the model reviewed project facts, constraints, and preferences |

`preserve_successful_behavior` is deterministic and does not require a model. The other objectives
select a generation model from the project's connected Gateway models. If that model is not ready,
the Dataset remains enabled but reports **Setup required** and does not enqueue collection work.

Use `preserve_successful_behavior` when you already have strong traces and want the model to repeat
that behavior. Use `correct_failed_conversations` after a diagnostic Rule has identified a recurring
failure and you have written the corrected response. Use `learn_from_user_corrections` when users
explicitly reject or rewrite answers. Use `teach_project_knowledge` after you have reviewed
Knowledge Goals and want the model to ground responses in project truth.

Findings and Knowledge participate as producer capabilities. They do not become Dataset source
types: automatically derived Examples still use neutral conversation provenance.

See the [Dataset quickstart](./quickstart.md) to create and review the first collection.

## Example Structure

Each Example separates provenance, labels, and model-facing content:

```json
{
  "id": "dsex_123",
  "projectId": "project_123",
  "datasetId": "ds_123",
  "source": {
    "kind": "conversation",
    "traceIds": ["trace_123"],
    "conversationId": "conversation_123",
    "derivationKey": "snapshot:policy:fingerprint"
  },
  "tags": ["objective:preserve_successful_behavior"],
  "payload": {
    "input": {
      "type": "chat",
      "messages": [{ "role": "user", "content": "Question" }]
    },
    "expectedOutput": {
      "type": "chat_message",
      "message": { "role": "assistant", "content": "Answer" }
    }
  }
}
```

`input` may be `chat`, `prompt`, `instruction`, `text`, or `json`.
`expectedOutput` may be an assistant chat message, text, JSON, or classification label. Optional
`rejectedOutput` and `rubric` fields support preference training and evaluation.

Currently only the `chat` schema (`{ "type": "chat", "version": 1 }`) is supported. The schema is
selected at Dataset creation and cannot be changed later, so choose it based on the training or
evaluation formats you intend to use.

## Provenance And Review

`source` supports `manual`, `conversation`, `import`, and `synthetic` origins. The metadata store
derives a stable canonical `sourceKey` and enforces uniqueness within `(projectId, datasetId)`. The
key is internal and is not part of the public Example object.

Automatic collection creates `conversation` sources with trace identity and a deterministic
derivation key. Reprocessing the same evidence under the same policy revision is idempotent.

Before using a Dataset, review Examples for correctness, privacy, contradictions, and coverage. A
high example count does not compensate for unsupported or inconsistent expected outputs.

## Data Quality Principles

A useful Dataset is smaller, coherent, and auditable rather than large and noisy.

- **Prefer correctness over volume.** Remove examples where the expected output is only partially
  correct or relies on missing context.
- **Check for contradictions.** Two examples with similar inputs must not teach opposite behavior.
- **Respect privacy boundaries.** Strip or exclude PII, credentials, and internal identifiers that
  should not appear in training or evaluation artifacts.
- **Preserve provenance.** Keep trace references so reviewers can verify the expected output against
  the original evidence.
- **Balance coverage.** Make sure the Dataset represents the behavior you want, not just the most
  common or most recent conversations.
- **Tag deliberately.** Use tags such as `reviewed`, `objective:*`, or `source:*` to track review
  state and filter examples during export.

## Exports And Runs

Canonical Examples can be exported to several training and evaluation formats:
`openai_chat_sft`, `chat_prompt_completion`, `alpaca_instruction`, `preference_chat_dpo`, and
`evaluation_jsonl`. Each exporter validates the required input and output shapes before producing
output.

Fine-tuning and evaluation jobs do not consume the mutable Dataset directly. They materialize a
versioned run manifest that freezes the selected Example IDs, the export format, and deterministic
sampling metadata. Subsequent edits to the Dataset do not change an already-created run. See
[Dataset export formats](./formats.md) for the shape and constraints of each format.

## Collection Mechanics

Automatic collection is asynchronous. After a conversation becomes inactive, the runtime evaluates
eligible evidence against each enabled Dataset's policy. Reprocessing the same evidence under the
same policy revision is idempotent, so collection does not produce unbounded duplicates. You can
disable collection on a Dataset to freeze its working set while continuing to review or export
existing Examples.