---
title: "Skill Treatment Effect"
section: "concepts"
type: "concept"
created: "2026-08-27"
updated: "2026-08-27"
tags: ["agentic-patterns", "evaluation", "harness-engineering"]
canonical: "https://pyweb.dev/wiki/skill-treatment-effect"
---
# Skill Treatment Effect

The **Skill Treatment Effect** refers to the measured marginal utility of injecting procedural knowledge packages ([evals skills](/wiki/evals-skills), `SKILL.md`) into an agent's context during inference. Empirical benchmarks demonstrate that skills are not a universal productivity booster; their effectiveness depends heavily on domain specificity, task fit, and version compatibility.

```mermaid
flowchart LR
    A[Raw Skill Injection] --> B{Task & Domain Fit}
    B -->|Domain-Specific / Tool Procedure| C[+15% to +30% Lift]
    B -->|Generic Software Engineering| D[+1.2% to +4.5% Marginal Lift]
    B -->|Version Mismatch / Contradiction| E[-5% to -10% Regression & Bloat]
```

## Empirical Evidence

### SkillsBench (Cross-Domain Benchmark)
- **Scope:** 87 tasks across 8 domains evaluated over 18 model-harness configurations.
- **Aggregate Result:** Curated skills increased the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain).
- **Domain Variance:** Broad non-coding tasks saw substantial lift, but software engineering tasks experienced only a modest **+4.5 pp** improvement.
- **Composition Law:** Focused skills with $\le 3$ modules outperformed exhaustive or kitchen-sink bundles. Self-generated skills yielded zero net benefit.

### SWE-Skills-Bench (Software Engineering Focus)
- **Scope:** 49 public SWE skills across ~565 task instances in real-world GitHub repositories with deterministic verifiers.
- **Aggregate Result:** Mean pass-rate gain across all skills was only **+1.2%**.
- **Distribution:**
  - **39 of 49 skills:** Zero pass-rate improvement while increasing token overhead by up to 451%.
  - **7 specialized skills:** Delivered meaningful gains (up to +30%) by providing toolchain-specific API knowledge or complex domain flows.
  - **3 mismatched skills:** Degraded performance (up to -10%) due to outdated assumptions or version incompatibilities conflicting with the repo.

## Practical Implications for Harness Engineering

1. **Skills Are Code Interventions:** Treat skills like code dependencies. Version them, evaluate them on paired benchmark tasks (with vs. without skill), and track token cost vs. pass-rate delta.
2. **Beware Prompt Bloat:** Generic advice (e.g., "write clean modular code") induces [prompt bloat](/wiki/prompt-bloat) and increases inference latency without moving verification needles.
3. **Progressive Disclosure:** Pre-load only skill names and triggers in the root prompt; load the full procedural body only when the task explicitly routes to that skill.

---

## Related Concepts
- [evals skills](/wiki/evals-skills) — Evaluating and testing agent skills.
- [agent harness engineering](/wiki/agent-harness-engineering) — Harness architecture and runtime orchestration.
- [prompt bloat](/wiki/prompt-bloat) — Degradation of performance from excessive prompt instructions.
- [progressive disclosure](/wiki/progressive-disclosure) — Staged context loading to avoid token saturation.

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 117.7

### References (outbound)
- [Evals Skills](https://pyweb.dev/wiki/evals-skills.md)
- [Prompt Bloat](https://pyweb.dev/wiki/prompt-bloat.md)
- [Agent Harness Engineering](https://pyweb.dev/wiki/agent-harness-engineering.md)
- [Progressive Disclosure](https://pyweb.dev/wiki/progressive-disclosure.md)

### Referenced by (inbound)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)

### Evidence (verified primary sources)
- raw/papers/skillsbench-2026.md (internal workspace doc)
- raw/papers/swe-skills-bench-2026.md (internal workspace doc)
- [addy-osmani-agent-skills-2026](https://pyweb.dev/wiki/raw/articles/addy-osmani-agent-skills-2026.md) | origin: https://addyosmani.com/blog/agent-skills/ | ingested: 2026-08-27 | sha256: 8b3508b787f042786af718b0044ee01b0e16c202905d36e09a5403b16c80b54e

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt