> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/helicone/helicone/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets
> Create evaluation and fine-tuning datasets from production LLM traffic
Helicone Datasets let you capture, curate, and export production LLM requests for evaluation, fine-tuning, and analysis. Transform your production logs into high-quality training data with just a few clicks.
## Why Use Datasets
Create training datasets from your best production requests to fine-tune custom models
Build consistent test sets to evaluate model performance and compare versions
Curate high-quality examples to improve prompt engineering and model outputs
Export structured data for external analysis, research, and compliance
## Quick Start
Use [custom properties](/features/advanced-usage/custom-properties), scores, or feedback ratings to find your best examples
Check boxes next to requests you want in your dataset
Click "Add to Dataset" to create a new dataset or add to an existing one
Review examples, remove poor quality ones, and export in your preferred format
## Creating Datasets
### From the Dashboard
The easiest way to build datasets is through the Helicone UI:
1. Navigate to [helicone.ai/requests](https://us.helicone.ai/requests)
2. Apply filters to find high-quality examples:
* **Custom properties**: Tag production traffic (e.g., `feature: "customer-support"`)
* **Scores**: Filter by evaluation metrics (e.g., `accuracy > 90`)
* **Feedback**: Select highly-rated responses (e.g., `feedback: true`)
* **User**: Focus on specific users or use cases
3. Select requests using checkboxes
4. Click "Add to Dataset" and choose or create a dataset
### Via API
Create and manage datasets programmatically for automated workflows:
```typescript theme={null}
// Create a new dataset with requests
const response = await fetch('https://api.helicone.ai/v1/helicone-dataset', {
method: 'POST',
headers: {
'Authorization': `Bearer ${HELICONE_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
datasetName: 'Customer Support Q1 2024',
requestIds: [
'f47ac10b-58cc-4372-a567-0e02b2c3d479',
'6ba7b810-9dad-11d1-80b4-00c04fd430c8',
'6ba7b811-9dad-11d1-80b4-00c04fd430c9'
],
meta: {
description: 'High-quality customer support examples',
tags: ['support', 'q1-2024']
}
})
});
const { datasetId } = await response.json();
console.log('Created dataset:', datasetId);
```
```typescript theme={null}
// Add more requests to existing dataset
const response = await fetch(
`https://api.helicone.ai/v1/helicone-dataset/${datasetId}/mutate`,
{
method: 'POST',
headers: {
'Authorization': `Bearer ${HELICONE_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
addRequests: ['new-request-id-1', 'new-request-id-2'],
removeRequests: [] // Optional: remove requests
})
}
);
```
```typescript theme={null}
// Retrieve dataset contents
const response = await fetch(
`https://api.helicone.ai/v1/helicone-dataset/${datasetId}/query`,
{
method: 'POST',
headers: {
'Authorization': `Bearer ${HELICONE_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
limit: 100,
offset: 0
})
}
);
const rows = await response.json();
console.log('Dataset rows:', rows.length);
```
```typescript theme={null}
// Edit request/response in dataset
const response = await fetch(
`https://api.helicone.ai/v1/helicone-dataset/${datasetId}/request/${requestId}`,
{
method: 'POST',
headers: {
'Authorization': `Bearer ${HELICONE_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
requestBody: {
messages: [{
role: 'user',
content: 'Updated prompt text'
}]
},
responseBody: {
choices: [{
message: {
role: 'assistant',
content: 'Updated response text'
}
}]
}
})
}
);
```
### Rate Limits
**Dataset creation limits**: You can add up to 1,000 dataset rows per day per organization. The limit resets at midnight UTC.
## Curating Quality Datasets
### The Curation Process
Raw production logs contain noise—curation transforms them into valuable training data:
Add many potential examples initially. It's easier to remove poor examples than to find good ones later.
Examine each request/response pair for:
* **Accuracy**: Is the response correct and helpful?
* **Consistency**: Does it match the style and format you want?
* **Completeness**: Does it fully address the user's request?
* **Relevance**: Is this the behavior you want to reinforce?
Delete requests that contain:
* Incorrect or misleading responses
* Off-topic or irrelevant content
* Inconsistent formatting or style
* Edge cases that might confuse the model
* Sensitive or inappropriate content
Ensure you have:
* Examples covering all common use cases
* Both simple and complex queries
* Appropriate distribution matching real usage patterns
* Diverse input styles and edge cases
**Quality beats quantity**: 50-100 carefully curated examples often outperform thousands of uncurated ones. Focus on consistency and correctness over volume.
### Dataset Dashboard
Manage all your datasets at [helicone.ai/datasets](https://us.helicone.ai/datasets):
From the dashboard you can:
* **Track progress**: Monitor dataset size and last updated time
* **Access datasets**: Click to view and curate contents
* **Export data**: Download datasets when ready for fine-tuning
* **Delete datasets**: Remove datasets you no longer need
## Exporting Data
### Export Formats
Download your datasets in formats optimized for different use cases:
Perfect for OpenAI fine-tuning format:
```json theme={null}
{"messages": [{"role": "user", "content": "What is quantum computing?"}, {"role": "assistant", "content": "Quantum computing is..."}]}
{"messages": [{"role": "user", "content": "Explain machine learning"}, {"role": "assistant", "content": "Machine learning is..."}]}
```
Ready to use directly with:
* OpenAI's fine-tuning API
* Anthropic Claude fine-tuning
* Custom training pipelines
Structured format for spreadsheet analysis:
```csv theme={null}
request_id,created_at,model,prompt_tokens,completion_tokens,cost,user_message,assistant_response,properties
req_123,2024-01-15,gpt-4o,50,100,0.002,"What is AI?","AI is...","{feature: support}"
req_124,2024-01-15,gpt-4o-mini,45,95,0.0001,"Help me","I'd be happy to help!","{feature: support}"
```
Import into:
* Excel or Google Sheets
* Python pandas for analysis
* Data visualization tools
* Research databases
### Programmatic Export
Retrieve dataset contents via API:
```python theme={null}
import requests
import json
# Query dataset rows
response = requests.post(
f"https://api.helicone.ai/v1/helicone-dataset/{dataset_id}/query",
headers={"Authorization": f"Bearer {HELICONE_API_KEY}"},
json={"limit": 1000, "offset": 0}
)
rows = response.json()
# Format for fine-tuning
training_data = []
for row in rows:
# Fetch full request/response from signed URL
data = requests.get(row['signed_url']).json()
training_data.append({
"messages": data['request']['messages']
})
# Save as JSONL
with open('training_data.jsonl', 'w') as f:
for item in training_data:
f.write(json.dumps(item) + '\n')
```
## Use Cases
### Replace Expensive Models with Fine-Tuned Alternatives
The most common use case—train cheaper models on expensive model outputs:
Start logging successful requests from GPT-4, Claude Sonnet, or other expensive models
```typescript theme={null}
const response = await openai.chat.completions.create(
{ model: "gpt-4o", messages },
{
headers: {
"Helicone-Property-Task": "customer-support",
"Helicone-Property-Quality": "production"
}
}
);
```
Create separate datasets for different tasks:
* Customer support responses
* Code generation
* Data extraction
* Content summarization
Review examples to ensure responses follow the same format, style, and quality standards
Export JSONL and fine-tune models that are 10-50x cheaper:
* GPT-4o-mini (10x cheaper than GPT-4o)
* Gemini 2.5 Flash (50x cheaper than Gemini Pro)
* Claude Haiku (30x cheaper than Claude Sonnet)
Continue collecting examples from your fine-tuned model to improve it over time
A fine-tuned GPT-4o-mini can often match or exceed GPT-4o performance on specific tasks while costing 90% less. Start with 50-100 examples and iterate.
### Task-Specific Evaluation Sets
Build test datasets to evaluate model performance consistently:
```typescript theme={null}
// Create evaluation datasets for different capabilities
const evalDatasets = {
reasoning: {
name: 'Complex Reasoning',
description: 'Multi-step problems with verified solutions',
requestIds: [] // Add IDs of reasoning examples
},
extraction: {
name: 'Data Extraction',
description: 'Structured data extraction with known correct outputs',
requestIds: [] // Add IDs of extraction examples
},
creativity: {
name: 'Creative Writing',
description: 'Creative writing with human-rated quality scores',
requestIds: [] // Add IDs of creative examples
}
};
// Use these to:
// - Compare model versions before deploying
// - Test prompt changes against consistent examples
// - Identify model weaknesses and blind spots
```
### Continuous Improvement Pipeline
Build a data flywheel for model improvement:
1. **Tag production requests** with custom properties for filtering
```typescript theme={null}
{ headers: { "Helicone-Property-Feature": "chat" } }
```
2. **Score outputs** based on automated metrics or user feedback
```typescript theme={null}
await fetch(`/v1/request/${id}/score`, {
body: JSON.stringify({ scores: { quality: 95 } })
});
```
3. **Filter high-quality examples** using scores and feedback
```sql theme={null}
-- In Helicone dashboard filters
scores.quality > 90 AND feedback = true
```
4. **Auto-add to datasets** when examples meet quality thresholds
5. **Regular retraining** with newly curated examples every week/month
6. **A/B test** new models against production traffic before full rollout
### Research and Compliance
Export datasets for research, auditing, or compliance:
````python theme={null}
# Export dataset with metadata for research
import pandas as pd
rows = query_dataset(dataset_id)
df = pd.DataFrame(rows)
# Add analysis columns
df['response_length'] = df['assistant_response'].str.len()
df['prompt_complexity'] = df['user_message'].apply(calculate_complexity)
df['contains_code'] = df['assistant_response'].str.contains('```')
# Export for analysis
df.to_csv('research_dataset.csv', index=False)
# Generate statistics
stats = df.groupby('model').agg({
'cost': 'sum',
'prompt_tokens': 'mean',
'completion_tokens': 'mean'
})
print(stats)
````
## Best Practices
Choose fewer, high-quality examples rather than large datasets with mixed quality
Create separate datasets for different use cases rather than one general dataset
Continuously add new examples as your application evolves and improves
Document what makes a "good" example for each dataset's specific purpose
Create new dataset versions when making significant changes to examples
Include varied inputs, edge cases, and different user types in your datasets
## API Reference
### Key Endpoints
| Endpoint | Method | Description |
| ----------------------------------------------- | ------ | -------------------------------- |
| `/v1/helicone-dataset` | POST | Create new dataset with requests |
| `/v1/helicone-dataset/query` | POST | List all datasets |
| `/v1/helicone-dataset/{id}/query` | POST | Get dataset rows |
| `/v1/helicone-dataset/{id}/mutate` | POST | Add/remove requests |
| `/v1/helicone-dataset/{id}/request/{requestId}` | POST | Update request data |
| `/v1/helicone-dataset/{id}/delete` | POST | Delete dataset |
[View full API documentation →](/rest/dataset/post-v1helicone-dataset)
## Related Features
Track evaluation metrics to identify best examples for datasets
Use user ratings to find high-quality examples automatically
Tag requests to make dataset creation easier with filtering
Include full conversation context in your datasets
***
Datasets turn your production LLM logs into valuable training and evaluation resources. Start small with a focused use case, then expand as you see the benefits of curated, high-quality data.