Dataset-Creation-Skill
Dataset Creation & Training
Dataset Creation & Training
Complete guide for building high-quality datasets and training models. Quality data consistently outperforms model size and hyperparameter tuning — this skill covers the full lifecycle from raw collection to trained model.
When to Use
- Creating a new dataset for ML training or evaluation
- Preparing existing raw data for model consumption
- Cleaning, deduplicating, or augmenting datasets
- Generating synthetic training examples via LLMs
- Splitting datasets into train/validation/test sets
- Formatting data for instruction tuning (Alpaca, ShareGPT, ChatML)
- Training or fine-tuning models with prepared datasets
Iron Laws
- Split before touch — Separate train/validation/test before ANY preprocessing. Data leakage through preprocessing is the #1 ML bug.
- Data card required — Every dataset must document source, collection method, known biases, and usage restrictions.
- Version everything — Datasets, labels, and preprocessing code must be versioned together for reproducibility.
- Class balance awareness — Know your class distribution. Imbalanced datasets need explicit handling (oversampling, weighting, stratified splitting).
- Quality > quantity — Noisy, biased, or leaked data is worse than less clean data.
Workflow
Phase 1: Define & Collect
1.1 Define the objective
Before collecting data, answer:
- What task will the model perform? (classification, generation, Q&A, reasoning)
- What are the input/output types?
- What is the target performance metric?
- What are the constraints? (privacy, licensing, latency, budget)
1.2 Identify data sources
| Source Type | Best For | Considerations |
|---|---|---|
| Existing public datasets | Baseline, benchmarking | Check license, verify quality |
| Web scraping | Domain-specific text | Legal review required, clean aggressively |
| Internal logs / databases | Production-aligned data | PII risk, access controls |
| Human annotation | High-precision labels | Expensive, needs guidelines |
| Synthetic generation | Augmentation, bootstrapping | Risk of model collapse, verify quality |
1.3 Collect raw data
import pandas as pd
# Example: Load from multiple sources
sources = [
pd.read_csv("source_a.csv"),
pd.read_json("source_b.jsonl", lines=True),
pd.read_parquet("source_c.parquet"),
]
raw = pd.concat(sources, ignore_index=True)
print(f"Collected {len(raw)} raw records")
print(f"Columns: {raw.columns.tolist()}")
Related skills
- Downloads
- Security audit
- Pass
- Last updated
- 1mo ago
- Current version
- v1.0.0
- License
- MIT-0
