---
title: "Karpathy LLM Wiki: Spec vs. Ecosystem — Deep Analysis"
section: "raw"
type: "source"
created: "2026-08-22"
updated: "2026-08-22"
canonical: "https://pyweb.dev/wiki/raw/articles/karpathy-wiki-ecosystem-analysis-2026"
---
# Karpathy LLM Wiki: Spec vs. Ecosystem — Deep Analysis

Sources analyzed (all fetched 2026-08-23):
- Karpathy gist `442a6bf555914893e9891c11519de94f` (the spec)
- `Astro-Han/karpathy-llm-wiki` (SKILL.md + README design boundaries; ~2k stars; production KB: 94 articles / 99 sources / daily since Apr 2026)
- `NicholasSpisak/second-brain` (SKILL.md, wiki-schema.md, tooling.md; Obsidian-first)
- `atomicstrata/llm-wiki-compiler` v1.1.0 (full README; CLI/SDK/MCP compiler)
- `wpzero/karpathy-llm-wiki` (lightweight skill fork, 2 stars)
- Hermes local `llm-wiki` skill v2.1.0 (the skill pyweb's wiki is built from)
- Local pyweb wiki: 75 pages, SCHEMA.md, Astro site + `scripts/sync-wiki.mjs`

---

## 1. What Karpathy actually proposed

The gist is deliberately an **idea file, not a spec** — "designed to be copy-pasted to your own LLM agent... the document's only job is to communicate the pattern." Core elements:

- **Three layers**: raw sources (immutable) → wiki (LLM-owned markdown) → schema (CLAUDE.md/AGENTS.md; "what makes the LLM a disciplined wiki maintainer rather than a generic chatbot").
- **Three operations**: Ingest (one source may touch 10–15 pages), Query (good answers get *filed back* — explorations compound), Lint (contradictions, orphans, stale claims, gaps).
- **Two spine files**: `index.md` (content catalog; "works surprisingly well at ~100 sources / hundreds of pages and avoids embedding-based RAG") and `log.md` (append-only, grep-parseable via `## [date] op | title`).
- **Tooling is explicitly optional and deferred**: qmd suggested only "as the wiki grows"; "vibe-code a naive search script as the need arises."
- **Why it works**: maintenance cost → 0. "Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don't get bored."
- Everything modular: "pick what's useful, ignore what isn't."

Karpathy's own workflow is **interactive and supervised**: ingest one source at a time, discuss takeaways, Obsidian on one side, agent on the other. Batch-ingest is mentioned as possible but not his preference.

## 2. Architectural patterns across implementations

| Dimension | Karpathy gist | Astro-Han | NicholasSpisak | atomicstrata | wpzero |
|---|---|---|---|---|---|
| Form | idea file | pure skill (SKILL.md + references + 1 script) | skill bundle (4 skills + agent-config templates) | hardcoded CLI (Node 24, TS) | pure skill (fork-style) |
| Layout | abstract | `raw/<topic>/` + `wiki/<topic>/`, 1 level only | `raw/ wiki/{sources,entities,concepts,synthesis} output/` | `sources/ wiki/{concepts,queries,<typed>} .llmwiki/ artifacts/` | per-domain `<vault>/<domain>/{raw,wiki}` + global index |
| Links | wikilinks (Obsidian) | plain relative markdown paths | wikilinks (Obsidian) | wikilinks + typed relation store (`wiki/graph/`) | wikilinks |
| Index/log | yes | yes, log entries machine-parseable ("no material" heading is an inventory key) | yes, lint every 10 ingests / monthly | yes + `state.json` hashes, ownership, eval history | yes |
| Freshness | not specified | **rejected** (see below) | frontmatter `updated` + lint schedule | first-class: source hashes → stale/orphaned flags, `refresh --stale` | not specified |
| Evidence | "citations" loosely | **Grounding Invariant** + `check_evidence.py` grep | frontmatter `sources:` list | line-range citations, lint-validated, eval-judged | summary/entity/concept/comparison/synthesis page types |
| Review | human-in-loop conversationally | human via lint reports | human via wizard + lint | **runtime-enforced** review queue, trust gates, fail-closed | human-in-loop |

## 3. Astro-Han: the production-tested pure skill (most instructive)

The standout artifact. Its README's **"Design Boundaries"** section is a list of things deliberately *not* built after 3 months of production logs and an ecosystem survey — effectively a negative-space map of where the naive reading of Karpathy over-engineers:

**Rejected after production experience:**
- **Source-hash freshness tracking** — "raw/ is immutable, so hashes guard against events that cannot happen. Genuinely new information arrives as new sources." *(Directly contradicts the Hermes llm-wiki v2.1 skill's sha256 raw frontmatter, which pyweb's SCHEMA.md inherited.)*
- **Persisted line-number citations** — every observed fidelity error was "value absent from source," catchable by whole-file grep. Anchor friction makes agents skip the rule. *(Contrast: atomicstrata went the other way — line-range citations are central.)*
- **Numeric confidence/quality scores** — "false precision with no calibration behind it. Evidence strength belongs in the prose." *(Also contradicts Hermes skill's `confidence:` frontmatter.)*
- **Per-article review dates** — maintenance driven by whole-wiki lint, not per-page timers.
- **Access-based decay** — "frequently asked is not the same as true."
- **Retract/bad-source machinery** — "has not happened yet. Handle it manually until it does."
- **Hooks/scheduled runs, MCP servers, UIs** — belong to the agent harness, not a tool-agnostic skill.
- **Vector/graph search** — at 50K–100K tokens of curated wiki, "grep and read are more reliable. Add search tooling only when recall measurably degrades."
- **Typed relationship ontologies** — "link semantics live in the prose around the link."

**What it kept and sharpened:**
- **The Grounding Invariant**: every load-bearing fact (numbers, dates, quotes) must exist *verbatim* in the linked raw file; compile-time "locate before you write" + one Python script (`check_evidence.py`) that greps high-signal literals across the whole wiki in seconds. Because raw/ is immutable, verification is stateless and total — no incremental state.
- **Triage with an explicit escape hatch**: New / Update / Disputed / **No material** — "Do not force an article out of a thin source." The no-material log heading doubles as a machine-readable inventory key, so lint can exclude those raws from the unreferenced-files report.
- **Cascade updates as full-text search, not index trust**: "Do not rely on the index alone" — search the whole wiki for entities/aliases/claims.
- **Status blocks instead of silent rewrites**: Outdated / Disputed annotations keep old claims for the record ("Never silently rewrite history").
- **Lint with tiered authority**: Safe auto-fixes (index/link repair) vs. mechanical reports (evidence script) vs. judgment reports (contradictions, orphans — report only, never auto-fix facts).
- **Query never writes files unless asked**; archived answers are new pages, never merged into compiled articles.

## 4. NicholasSpisak/second-brain: Obsidian-first onboarding

- **Four skills** (`/second-brain` setup wizard, `-ingest`, `-query`, `-lint`) — the setup wizard is the differentiator: interactive configuration of vault name, location, domain, **per-agent config generation** (CLAUDE.md, AGENTS.md, Cursor rules, GEMINI.md all templated from one canonical `wiki-schema.md` — "single source of truth" pattern).
- **Tool integration as progressive enhancement**: Obsidian Web Clipper (required — the human's ingestion path), `summarize` (pre-digestion), `qmd` (hybrid BM25/vector search, explicitly gated on "~100+ pages"), `agent-browser` (fallback when web fetch fails). Tools are documented in `tooling.md` with install + verify commands but the schema works without them.
- **Opinionated page taxonomy**: sources/entities/concepts/synthesis subdirectories; rule 7: "Keep source summary pages factual. Save interpretation for concept and synthesis pages." Rule 9: "Prefer updating existing pages over creating new ones."
- **Explicit lint cadence**: after every 10 ingests, monthly minimum, before any major synthesis.
- **Image handling worked out concretely** (clipper → raw/assets/ → describe image contents in text during ingest so knowledge survives in text form).
- 10 numbered "Rules" at the end of the schema — a compliance checklist agents can be held to.

## 5. atomicstrata/llm-wiki-compiler: the hardcoded-tooling pole

Everything the pure skills leave to prompt discipline, llmwiki moves into runtime code:

- **Two-phase LLM compile pipeline** (concept extraction → typed page generation) with incremental recompilation via source hashes/ownership in `.llmwiki/state.json`. Batch compile replaces Karpathy's one-at-a-time supervision.
- **Configurable Lifecycle Profiles (v1.0)**: validated `profile.json` declares typed entities, relations, lifecycle state machines, trust gates, workflows — "enforced by the runtime, not left as prompt conventions. Invalid profiles fail closed." Ships `autosci` and `newsroom` domain packs; signed template distribution (Ed25519).
- **Retrieval is built in**: chunk embeddings + BM25 rerank + wikilink-graph expansion, content-hash-aware embedding updates, lexical fallback. Exactly the vector search Astro-Han deferred — because llmwiki targets larger/team corpora and context-pack generation for *other* agents.
- **Trust/safety as a write-path concern**: review queue, `--review` holds, connector confinement, symlink-escape confinement for wiki files read into prompts, hash-pinned artifacts. This is the team/untrusted-input scenario, absent from personal-wiki skills.
- **Eval harness**: `llmwiki eval` — health score, citation coverage/precision, regression deltas. The compiler *measures* what skills only assert.
- **Self-aware scope limits**: "Do not use as a static-site generator, heavy ontology DB, or for high-churn firehoses where raw search is enough and compiled structure goes stale faster than it can be reviewed."
- Costs: Node 24, provider API keys, real complexity budget. Overkill below ~hundreds of sources or for a single-user interactive workflow.

## 6. Prompt discipline vs. hardcoded tooling — the real tradeoff

**Where pure prompt/skill approaches win (proven by Astro-Han's production logs):**
- Single-user, interactive, Obsidian-sidecar workflow — Karpathy's actual described usage.
- Tool portability: one SKILL.md runs on Claude Code, Cursor, Codex, Hermes unchanged.
- Iteration speed: schema co-evolves with the domain by editing markdown, not code.
- The failure modes that actually occurred in production (fabricated values, index drift, dead links) were all catchable by *grep + one Python script + tiered lint*, not by infrastructure.

**Where prompt discipline struggles (what pushes projects toward tooling):**
- **Compliance drift**: rules like provenance markers and confidence fields get skipped by agents under context pressure (Astro-Han: "annotation friction makes agents skip the rule"). Anything that matters must be *verifiable mechanically* or it will silently decay. This is the strongest argument for one small checker script inside an otherwise pure skill.
- **Batch/CI/team operation**: unattended compile, review gating of untrusted imports, quality thresholds in CI — impossible as prompt conventions; llmwiki's entire reason to exist.
- **Scale past index-navigation**: Karpathy and Astro-Han agree index+grep works to ~100 sources / 50–100K tokens. Beyond that, embeddings/search stop being optional.
- **State that prompts can't reliably maintain**: incremental compilation (what changed since last compile) needs hashes somewhere — llmwiki puts them in state.json; pure skills avoid needing them by re-verifying everything cheaply each time (the immutable-raw trick).

**The synthesis both ends converge on**: keep the *artifact* dumb (plain markdown, relative links, append-only log) and put whatever intelligence you need in the *cheapest place that can enforce it* — prompt rule if judgment is required, script if it's mechanical, runtime gate only if adversaries/scale/CI demand it.

## 7. Lessons for pyweb's LLM wiki

pyweb already runs a healthy Hermes-skill-derived wiki (75 pages, 3-domain scope with negative filter, Astro-published). Concrete takeaways, in priority order:

1. **Adopt the Grounding Invariant and `check_evidence.py`-style lint.** This is the single highest-value production lesson. pyweb's SCHEMA has provenance markers (`^[raw/...]`) but nothing verifies them. A small grep-based script (verbatim numbers/dates/quotes must appear in linked raw files) converts a prompt rule into an enforced one — the pattern that survived production.
2. **Add the "No material" disposition.** Prevents thin sources from forcing junk pages — important for pyweb's strict 3-domain scope with negative filter; log-only ingests keep the raw capture without polluting the index.
3. **Add Status blocks (Outdated/Disputed) instead of silent rewrites.** pyweb covers fast-moving AI topics; current SCHEMA has `contradictions:` frontmatter but no in-body convention. Astro-Han's inline Status block is the battle-tested version.
4. **Reconsider sha256 raw frontmatter and `confidence:` scores.** Two experienced implementations disagree: Hermes skill (pyweb's base) mandates them; Astro-Han explicitly rejected both after production use ("guards against events that cannot happen" / "false precision"). For pyweb's single-author interactive use, Astro-Han's argument is strong — candidates for removal at next SCHEMA revision to cut ingest friction. (Keep if pyweb moves to automated/cron ingests, where drift detection becomes real.)
5. **Tier the lint by authority** (auto-fix links/index; report facts/orphans/contradictions). pyweb's lint currently inherits the flat Hermes list; tiering prevents an agent from "fixing" facts during lint.
6. **Don't adopt llmwiki-style compilation.** pyweb's wiki is a *content source for an Astro site* (`sync-wiki.mjs` → content collections) with human-in-loop ingests — squarely the skill sweet spot. The compiler's value (batch compile, review queues, OKF exchange, MCP context packs) only materializes if pyweb later wants unattended multi-source pipelines or to serve context packs to other agents. If that day comes, llmwiki can point at the same vault (it's Obsidian-compatible); no migration needed now.
7. **Keep vector search deferred.** At 75 pages pyweb is far below the ~100-source threshold where Karpathy/Astro-Han/NicholasSpisak all agree grep+index suffice. The trigger to revisit is measurable recall degradation, not page count anxiety.
8. **Borrow the single-source-of-truth schema pattern** from second-brain if pyweb grows more agent configs: one canonical `wiki-schema.md` that AGENTS.md/CLAUDE.md/skill files *reference* rather than duplicate — pyweb already has SCHEMA.md + hermes/AGENTS.md that can drift apart.
9. **Publish-side synergy**: llmwiki's exports (llms.txt, Marp, GraphML) mirror what pyweb's Astro build already produces (`llms-full.txt` exists in dist/). pyweb's static-site output is effectively the "viewer" that llmwiki had to build — a genuine advantage of the pyweb architecture; keep the wiki markdown-native so the Astro pipeline stays a pure projection.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt