Datasets

Datasets are consumption-neutral. Creating a Dataset does not commit it to one training engine, evaluation framework, or model provider.

When To Use A Dataset#

Use a Dataset when you want to:

• preserve production behavior that should not regress;
• turn diagnosed failures into corrected examples;
• learn from explicit user corrections;
• teach a model reviewed project Knowledge;
• maintain one versionable source for evaluation and fine-tuning.

Core Concepts#

Collection Objectives#

Each automatically collecting Dataset has one objective:

preserve_successful_behavior is deterministic and does not require a model. The other objectives select a generation model from the project's connected Gateway models. If that model is not ready, the Dataset remains enabled but reports Setup required and does not enqueue collection work.

Use preserve_successful_behavior when you already have strong traces and want the model to repeat that behavior. Use correct_failed_conversations after a diagnostic Rule has identified a recurring failure and you have written the corrected response. Use learn_from_user_corrections when users explicitly reject or rewrite answers. Use teach_project_knowledge after you have reviewed Knowledge Goals and want the model to ground responses in project truth.

Findings and Knowledge participate as producer capabilities. They do not become Dataset source types: automatically derived Examples still use neutral conversation provenance.

See the Dataset quickstart to create and review the first collection.

Example Structure#

Each Example separates provenance, labels, and model-facing content:

{
  "id": "dsex_123",
  "projectId": "project_123",
  "datasetId": "ds_123",
  "source": {
    "kind": "conversation",
    "traceIds": ["trace_123"],
    "conversationId": "conversation_123",
    "derivationKey": "snapshot:policy:fingerprint"
  },
  "tags": ["objective:preserve_successful_behavior"],
  "payload": {
    "input": {
      "type": "chat",
      "messages": [{ "role": "user", "content": "Question" }]
    },
    "expectedOutput": {
      "type": "chat_message",
      "message": { "role": "assistant", "content": "Answer" }
    }
  }
}

input may be chat, prompt, instruction, text, or json. expectedOutput may be an assistant chat message, text, JSON, or classification label. Optional rejectedOutput and rubric fields support preference training and evaluation.

Currently only the chat schema ({ "type": "chat", "version": 1 }) is supported. The schema is selected at Dataset creation and cannot be changed later, so choose it based on the training or evaluation formats you intend to use.

Provenance And Review#

source supports manual, conversation, import, and synthetic origins. The metadata store derives a stable canonical sourceKey and enforces uniqueness within (projectId, datasetId). The key is internal and is not part of the public Example object.

Automatic collection creates conversation sources with trace identity and a deterministic derivation key. Reprocessing the same evidence under the same policy revision is idempotent.

Before using a Dataset, review Examples for correctness, privacy, contradictions, and coverage. A high example count does not compensate for unsupported or inconsistent expected outputs.

Data Quality Principles#

A useful Dataset is smaller, coherent, and auditable rather than large and noisy.

• Prefer correctness over volume. Remove examples where the expected output is only partially correct or relies on missing context.
• Check for contradictions. Two examples with similar inputs must not teach opposite behavior.
• Respect privacy boundaries. Strip or exclude PII, credentials, and internal identifiers that should not appear in training or evaluation artifacts.
• Preserve provenance. Keep trace references so reviewers can verify the expected output against the original evidence.
• Balance coverage. Make sure the Dataset represents the behavior you want, not just the most common or most recent conversations.
• Tag deliberately. Use tags such as reviewed, objective:*, or source:* to track review state and filter examples during export.

Exports And Runs#

Canonical Examples can be exported to several training and evaluation formats: openai_chat_sft, chat_prompt_completion, alpaca_instruction, preference_chat_dpo, and evaluation_jsonl. Each exporter validates the required input and output shapes before producing output.

Fine-tuning and evaluation jobs do not consume the mutable Dataset directly. They materialize a versioned run manifest that freezes the selected Example IDs, the export format, and deterministic sampling metadata. Subsequent edits to the Dataset do not change an already-created run. See Dataset export formats for the shape and constraints of each format.

Collection Mechanics#

Automatic collection is asynchronous. After a conversation becomes inactive, the runtime evaluates eligible evidence against each enabled Dataset's policy. Reprocessing the same evidence under the same policy revision is idempotent, so collection does not produce unbounded duplicates. You can disable collection on a Dataset to freeze its working set while continuing to review or export existing Examples.