Fine-Tuning

Fine-tune LLMs with data preparation, provider selection, cost estimation, evaluation, and compliance checks.

Security Scan

The skill's content matches a fine-tuning guide, but it omits declaring needed credentials and runtime assumptions (API keys, CLI setup, network access), so its requirements and instructions are not fully coherent or transparent.

Purpose & Capability

The name/description match the provided files: the SKILL.md and supporting docs are a thorough fine-tuning playbook (data prep, providers, costs, evaluation, compliance). The requested capabilities are appropriate for a fine-tuning helper.

Instruction Scope

The runtime instructions and example code assume use of provider SDKs/CLIs (OpenAI client/CLI, Bedrock, huggingface-cli, pip installs/transfers for air-gapped systems) and local dataset files. The SKILL metadata declares no required env vars or credentials, yet the instructions require API keys, CLI auth, access to local dataset files, and network access to cloud providers — a scope mismatch. The docs also show uploading datasets and calling model endpoints which would transmit potentially sensitive data to third-party providers if the user follows examples.

Install Mechanism

This is instruction-only (no install spec), so nothing will be automatically written or executed by installation. However, the docs instruct users to run package installs, pip downloads, huggingface and OpenAI CLI commands and to pre-download model artifacts for air-gapped setups — all of which require manual actions and network access. There is no automatic installer, which lowers supply-chain risk but does not remove runtime risks if users execute the suggested commands.

Credentials

The registry metadata lists no required environment variables or primary credential, yet multiple code snippets and CLI commands clearly require provider credentials (OPENAI API keys, AWS credentials for Bedrock/Bedrock via AWS, Hugging Face auth) and may encourage enabling 'data sharing' discounts. Asking users to run these without declaring them is a transparency gap. The compliance guidance also emphasizes scanning/removing PII, but example eval/generation snippets call out to remote providers (client.chat/completions.create), which would send dataset content off-host unless an on-prem option is used.

Persistence & Privilege

The skill is not always-enabled and does not request special persistence or system-wide changes. It does not modify other skills or claim elevated privileges. Autonomous invocation is allowed (platform default) but is not combined with other major red flags here.

What to consider before installing

This skill is a comprehensive fine-tuning guide but is missing explicit declarations of operational needs. Before installing/using it: 1) Expect to need provider credentials (OpenAI key, AWS creds for Bedrock, Hugging Face token) and ensure you only provide those in a controlled environment; the skill metadata does not list them. 2) Be careful with real datasets: the examples upload local data to remote APIs and could send PII to third parties — follow the compliance.md PII-remediation steps and test on non-sensitive samples first. 3) The skill is instruction-only (no install) so it won't auto-run code, but following its examples will run CLI/SDK commands and pip installs — review commands before executing and prefer isolated/sandboxed environments. 4) If you require strict privacy, prefer the on‑prem instructions in compliance.md and verify how to authenticate and download models offline. 5) Because the source/homepage is unknown, exercise extra caution: verify code snippets and provider commands against official provider docs before use.

When to Use

User wants to fine-tune a language model, evaluate if fine-tuning is worth it, or debug training issues.

Quick Reference

Topic	File
Provider comparison & pricing	providers.md
Data preparation & validation	data-prep.md
Training configuration	training.md
Evaluation & debugging	evaluation.md
Cost estimation & ROI	costs.md
Compliance & security	compliance.md

Core Capabilities

1. Decide fit — Analyze if fine-tuning beats prompting for the use case
2. Prepare data — Convert raw data to JSONL, deduplicate, validate format
3. Select provider — Compare OpenAI, Anthropic (Bedrock), Google, open source based on constraints
4. Estimate costs — Calculate training cost, inference savings, break-even point
5. Configure training — Set hyperparameters (learning rate, epochs, LoRA rank)
6. Run evaluation — Compare fine-tuned vs base model on task-specific metrics
7. Debug failures — Diagnose loss curves, overfitting, catastrophic forgetting
8. Handle compliance — Scan for PII, configure on-premise training, generate audit logs

Decision Checklist

Before recommending fine-tuning, ask:

• What's the failure mode with prompting? (format, style, knowledge, cost)
• How many training examples available? (minimum 50-100)
• Expected inference volume? (affects ROI calculation)
• Privacy constraints? (determines provider options)
• Budget for training + ongoing inference?

Fine-Tune vs Prompt Decision

Signal	Recommendation
Format/style inconsistency	Fine-tune ✓
Missing domain knowledge	RAG first, then fine-tune if needed
High inference volume (>100K/mo)	Fine-tune for cost savings
Requirements change frequently	Stick with prompting
<50 quality examples	Prompting + few-shot

Critical Rules

• Data quality > quantity — 100 great examples beat 1000 noisy ones
• LoRA first — Never jump to full fine-tuning; LoRA is 10-100x cheaper
• Hold out eval set — Always 80/10/10 split; never peek at test data
• Same precision — Train and serve at identical precision (4-bit, 16-bit)
• Baseline first — Run eval on base model before training to measure actual improvement
• Expect iteration — First attempt rarely optimal; plan for 2-3 cycles

Common Pitfalls

Mistake	Fix
Training on inconsistent data	Manual review of 100+ samples before training
Learning rate too high	Start with 2e-4 for SFT, 5e-6 for RLHF
Expecting new knowledge	Fine-tuning adjusts behavior, not knowledge — use RAG
No baseline comparison	Always test base model on same eval set
Ignoring forgetting	Mix 20% general data to preserve capabilities