🔬 Automatic Skill Refiners for AI Coding Agents

Claude Code plugin · SkillOpt repo · arXiv paper

The deployment-time companion to SkillOpt. Harvests your Claude Code/Codex sessions, mines recurring tasks, replays them offline, and consolidates learned improvements into validated skill edits. Nothing changes until you approve.

How One "Night" Works

Key Features

• Claude Code plugin ready — /plugin marketplace add ./skillopt-sleep-plugin
• Validation gate — edits only accepted when held-out score strictly improves
• Nothing live until you adopt — /skillopt-sleep adopt with backup
• Free dry-run — mock backend costs nothing, try the plumbing first
• Experience replay — recall past similar tasks to strengthen nightly updates
• Deficient skills went 0.00 → 1.00 on held-out tasks in 1-2 nights

Quick Start

git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
# In Claude Code:
/plugin marketplace add ./skillopt-sleep-plugin
/plugin install skillopt-sleep@skillopt-sleep
/skillopt-sleep dry-run   # preview what it would learn
/skillopt-sleep run       # full cycle, stages proposal
/skillopt-sleep adopt     # apply changes (with backup)

Blog post · GitHub (Claude Code) · Best practices learned

Specifically built to optimize CLAUDE.md, .clinerules, and cursor rules. Uses an iterative prompt learning loop with LLM-as-judge evaluation — the closest tool to what you're already doing manually.

How It Works

Results

• +10-15% accuracy on SWE-bench for Cline
• +10% boost for Claude Code with no architecture changes
• Has specific cookbooks for Claude Code and Cline
• Full observability via Phoenix dashboard

Requires setting up eval tasks — more involved than SkillOpt-Sleep but gives you more control over what "good" means.

Plugin page · Blog post · SKILL.md

Anthropic's official plugin with four composable agents (Executor, Grader, Comparator, Analyzer) for data-driven skill refinement. Semi-automated — human-in-the-loop for approval.

Modes

• Create — extracts workflow from conversation and builds a skill
• Eval — defines test prompts and checks skill behavior
• Improve — analyzes eval results and suggests targeted improvements
• Benchmark — runs standardized assessments with variance analysis
• Blind A/B comparison between skill versions

github.com/microsoft/SkillOpt · arXiv paper · pip install skillopt

The full offline optimizer. Treats skill documents as trainable parameters — runs epochs with (mini-)batch size, learning rate, and validation gates. Produces a compact best_skill.md artifact.

Results

• Best or tied-best on all 52 evaluated cells (model × benchmark × harness)
• On GPT-5.5: +23.5 pts direct chat, +24.8 Codex, +19.1 Claude Code
• Skills transfer across models and harnesses

Need to define benchmarks/tasks. More work than Sleep plugin but more control.

github.com/alchaincyf/darwin-skill · npx skills add alchaincyf/darwin-skill

Inspired by Karpathy's autoresearch. Evaluates skills on 9 quality dimensions with human-in-the-loop checkpoints. Integrated SkillOpt's validation gate in v2.0.

Key Differences

• Human-in-the-loop by design — 3-layer checkpoint system
• 9-dimension rubric: failure encoding, actionable specificity, high-risk blacklist, etc.
• Anti-pattern blacklist: prevents AI from grading its own work (LLM self-eval = 46.4% accurate)
• Early stopping: auto-stops if gain < 1 point per round
• Results: 80.8 → 91.65 on real skills

Best for skills where you have strong aesthetic/quality opinions.

GEPA repo · gskill blog · gskill guide

Genetic-Pareto evolutionary search for skill optimization. Uses SWE-smith to auto-generate verifiable tasks from any GitHub repo. 90× cheaper than RL approaches.

• Maintains a Pareto frontier of candidate skills
• Skills transfer across models (train on cheap, deploy on expensive)
• Improved Jinja repo bug-fixing: 55% → 82% resolve rate
• Now integrated into DSPy

GitHub · arXiv paper

Failure-driven evolutionary skill discovery. Operates on skill folders compatible with Claude Code, Codex, OpenCode. Model stays frozen; only skills evolve.

• Skills materialized as structured folders (instructions + trigger metadata + helper scripts)
• Retained only if they improve held-out validation
• Maintains Pareto frontier of agent programs