Karpathy LLM Wiki: Spec vs. Ecosystem — Deep Analysis

Sources analyzed (all fetched 2026-08-23):

Karpathy gist 442a6bf555914893e9891c11519de94f (the spec)
Astro-Han/karpathy-llm-wiki (SKILL.md + README design boundaries; ~2k stars; production KB: 94 articles / 99 sources / daily since Apr 2026)
NicholasSpisak/second-brain (SKILL.md, wiki-schema.md, tooling.md; Obsidian-first)
atomicstrata/llm-wiki-compiler v1.1.0 (full README; CLI/SDK/MCP compiler)
wpzero/karpathy-llm-wiki (lightweight skill fork, 2 stars)
Hermes local llm-wiki skill v2.1.0 (the skill pyweb’s wiki is built from)
Local pyweb wiki: 75 pages, SCHEMA.md, Astro site + scripts/sync-wiki.mjs

1. What Karpathy actually proposed

The gist is deliberately an idea file, not a spec — “designed to be copy-pasted to your own LLM agent… the document’s only job is to communicate the pattern.” Core elements:

Three layers: raw sources (immutable) → wiki (LLM-owned markdown) → schema (CLAUDE.md/AGENTS.md; “what makes the LLM a disciplined wiki maintainer rather than a generic chatbot”).
Three operations: Ingest (one source may touch 10–15 pages), Query (good answers get filed back — explorations compound), Lint (contradictions, orphans, stale claims, gaps).
Two spine files: index.md (content catalog; “works surprisingly well at ~100 sources / hundreds of pages and avoids embedding-based RAG”) and log.md (append-only, grep-parseable via ## [date] op | title).
Tooling is explicitly optional and deferred: qmd suggested only “as the wiki grows”; “vibe-code a naive search script as the need arises.”
Why it works: maintenance cost → 0. “Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don’t get bored.”
Everything modular: “pick what’s useful, ignore what isn’t.”

Karpathy’s own workflow is interactive and supervised: ingest one source at a time, discuss takeaways, Obsidian on one side, agent on the other. Batch-ingest is mentioned as possible but not his preference.

2. Architectural patterns across implementations

Dimension	Karpathy gist	Astro-Han	NicholasSpisak	atomicstrata	wpzero
Form	idea file	pure skill (SKILL.md + references + 1 script)	skill bundle (4 skills + agent-config templates)	hardcoded CLI (Node 24, TS)	pure skill (fork-style)
Layout	abstract	`raw/<topic>/` + `wiki/<topic>/`, 1 level only	`raw/ wiki/{sources,entities,concepts,synthesis} output/`	`sources/ wiki/{concepts,queries,<typed>} .llmwiki/ artifacts/`	per-domain `<vault>/<domain>/{raw,wiki}` + global index
Links	wikilinks (Obsidian)	plain relative markdown paths	wikilinks (Obsidian)	wikilinks + typed relation store (`wiki/graph/`)	wikilinks
Index/log	yes	yes, log entries machine-parseable (“no material” heading is an inventory key)	yes, lint every 10 ingests / monthly	yes + `state.json` hashes, ownership, eval history	yes
Freshness	not specified	rejected (see below)	frontmatter `updated` + lint schedule	first-class: source hashes → stale/orphaned flags, `refresh --stale`	not specified
Evidence	“citations” loosely	Grounding Invariant + `check_evidence.py` grep	frontmatter `sources:` list	line-range citations, lint-validated, eval-judged	summary/entity/concept/comparison/synthesis page types
Review	human-in-loop conversationally	human via lint reports	human via wizard + lint	runtime-enforced review queue, trust gates, fail-closed	human-in-loop

3. Astro-Han: the production-tested pure skill (most instructive)

The standout artifact. Its README’s “Design Boundaries” section is a list of things deliberately not built after 3 months of production logs and an ecosystem survey — effectively a negative-space map of where the naive reading of Karpathy over-engineers:

Rejected after production experience:

Source-hash freshness tracking — “raw/ is immutable, so hashes guard against events that cannot happen. Genuinely new information arrives as new sources.” (Directly contradicts the Hermes llm-wiki v2.1 skill’s sha256 raw frontmatter, which pyweb’s SCHEMA.md inherited.)
Persisted line-number citations — every observed fidelity error was “value absent from source,” catchable by whole-file grep. Anchor friction makes agents skip the rule. (Contrast: atomicstrata went the other way — line-range citations are central.)
Numeric confidence/quality scores — “false precision with no calibration behind it. Evidence strength belongs in the prose.” (Also contradicts Hermes skill’s confidence: frontmatter.)
Per-article review dates — maintenance driven by whole-wiki lint, not per-page timers.
Access-based decay — “frequently asked is not the same as true.”
Retract/bad-source machinery — “has not happened yet. Handle it manually until it does.”
Hooks/scheduled runs, MCP servers, UIs — belong to the agent harness, not a tool-agnostic skill.
Vector/graph search — at 50K–100K tokens of curated wiki, “grep and read are more reliable. Add search tooling only when recall measurably degrades.”
Typed relationship ontologies — “link semantics live in the prose around the link.”

What it kept and sharpened:

The Grounding Invariant: every load-bearing fact (numbers, dates, quotes) must exist verbatim in the linked raw file; compile-time “locate before you write” + one Python script (check_evidence.py) that greps high-signal literals across the whole wiki in seconds. Because raw/ is immutable, verification is stateless and total — no incremental state.
Triage with an explicit escape hatch: New / Update / Disputed / No material — “Do not force an article out of a thin source.” The no-material log heading doubles as a machine-readable inventory key, so lint can exclude those raws from the unreferenced-files report.
Cascade updates as full-text search, not index trust: “Do not rely on the index alone” — search the whole wiki for entities/aliases/claims.
Status blocks instead of silent rewrites: Outdated / Disputed annotations keep old claims for the record (“Never silently rewrite history”).
Lint with tiered authority: Safe auto-fixes (index/link repair) vs. mechanical reports (evidence script) vs. judgment reports (contradictions, orphans — report only, never auto-fix facts).
Query never writes files unless asked; archived answers are new pages, never merged into compiled articles.

4. NicholasSpisak/second-brain: Obsidian-first onboarding

Four skills (/second-brain setup wizard, -ingest, -query, -lint) — the setup wizard is the differentiator: interactive configuration of vault name, location, domain, per-agent config generation (CLAUDE.md, AGENTS.md, Cursor rules, GEMINI.md all templated from one canonical wiki-schema.md — “single source of truth” pattern).
Tool integration as progressive enhancement: Obsidian Web Clipper (required — the human’s ingestion path), summarize (pre-digestion), qmd (hybrid BM25/vector search, explicitly gated on “~100+ pages”), agent-browser (fallback when web fetch fails). Tools are documented in tooling.md with install + verify commands but the schema works without them.
Opinionated page taxonomy: sources/entities/concepts/synthesis subdirectories; rule 7: “Keep source summary pages factual. Save interpretation for concept and synthesis pages.” Rule 9: “Prefer updating existing pages over creating new ones.”
Explicit lint cadence: after every 10 ingests, monthly minimum, before any major synthesis.
Image handling worked out concretely (clipper → raw/assets/ → describe image contents in text during ingest so knowledge survives in text form).
10 numbered “Rules” at the end of the schema — a compliance checklist agents can be held to.

5. atomicstrata/llm-wiki-compiler: the hardcoded-tooling pole

Everything the pure skills leave to prompt discipline, llmwiki moves into runtime code:

Two-phase LLM compile pipeline (concept extraction → typed page generation) with incremental recompilation via source hashes/ownership in .llmwiki/state.json. Batch compile replaces Karpathy’s one-at-a-time supervision.
Configurable Lifecycle Profiles (v1.0): validated profile.json declares typed entities, relations, lifecycle state machines, trust gates, workflows — “enforced by the runtime, not left as prompt conventions. Invalid profiles fail closed.” Ships autosci and newsroom domain packs; signed template distribution (Ed25519).
Retrieval is built in: chunk embeddings + BM25 rerank + wikilink-graph expansion, content-hash-aware embedding updates, lexical fallback. Exactly the vector search Astro-Han deferred — because llmwiki targets larger/team corpora and context-pack generation for other agents.
Trust/safety as a write-path concern: review queue, --review holds, connector confinement, symlink-escape confinement for wiki files read into prompts, hash-pinned artifacts. This is the team/untrusted-input scenario, absent from personal-wiki skills.
Eval harness: llmwiki eval — health score, citation coverage/precision, regression deltas. The compiler measures what skills only assert.
Self-aware scope limits: “Do not use as a static-site generator, heavy ontology DB, or for high-churn firehoses where raw search is enough and compiled structure goes stale faster than it can be reviewed.”
Costs: Node 24, provider API keys, real complexity budget. Overkill below ~hundreds of sources or for a single-user interactive workflow.

6. Prompt discipline vs. hardcoded tooling — the real tradeoff

Where pure prompt/skill approaches win (proven by Astro-Han’s production logs):

Single-user, interactive, Obsidian-sidecar workflow — Karpathy’s actual described usage.
Tool portability: one SKILL.md runs on Claude Code, Cursor, Codex, Hermes unchanged.
Iteration speed: schema co-evolves with the domain by editing markdown, not code.
The failure modes that actually occurred in production (fabricated values, index drift, dead links) were all catchable by grep + one Python script + tiered lint, not by infrastructure.

Where prompt discipline struggles (what pushes projects toward tooling):

Compliance drift: rules like provenance markers and confidence fields get skipped by agents under context pressure (Astro-Han: “annotation friction makes agents skip the rule”). Anything that matters must be verifiable mechanically or it will silently decay. This is the strongest argument for one small checker script inside an otherwise pure skill.
Batch/CI/team operation: unattended compile, review gating of untrusted imports, quality thresholds in CI — impossible as prompt conventions; llmwiki’s entire reason to exist.
Scale past index-navigation: Karpathy and Astro-Han agree index+grep works to ~100 sources / 50–100K tokens. Beyond that, embeddings/search stop being optional.
State that prompts can’t reliably maintain: incremental compilation (what changed since last compile) needs hashes somewhere — llmwiki puts them in state.json; pure skills avoid needing them by re-verifying everything cheaply each time (the immutable-raw trick).

The synthesis both ends converge on: keep the artifact dumb (plain markdown, relative links, append-only log) and put whatever intelligence you need in the cheapest place that can enforce it — prompt rule if judgment is required, script if it’s mechanical, runtime gate only if adversaries/scale/CI demand it.

7. Lessons for pyweb’s LLM wiki

pyweb already runs a healthy Hermes-skill-derived wiki (75 pages, 3-domain scope with negative filter, Astro-published). Concrete takeaways, in priority order:

Adopt the Grounding Invariant and check_evidence.py-style lint. This is the single highest-value production lesson. pyweb’s SCHEMA has provenance markers (^[raw/...]) but nothing verifies them. A small grep-based script (verbatim numbers/dates/quotes must appear in linked raw files) converts a prompt rule into an enforced one — the pattern that survived production.
Add the “No material” disposition. Prevents thin sources from forcing junk pages — important for pyweb’s strict 3-domain scope with negative filter; log-only ingests keep the raw capture without polluting the index.
Add Status blocks (Outdated/Disputed) instead of silent rewrites. pyweb covers fast-moving AI topics; current SCHEMA has contradictions: frontmatter but no in-body convention. Astro-Han’s inline Status block is the battle-tested version.
Reconsider sha256 raw frontmatter and confidence: scores. Two experienced implementations disagree: Hermes skill (pyweb’s base) mandates them; Astro-Han explicitly rejected both after production use (“guards against events that cannot happen” / “false precision”). For pyweb’s single-author interactive use, Astro-Han’s argument is strong — candidates for removal at next SCHEMA revision to cut ingest friction. (Keep if pyweb moves to automated/cron ingests, where drift detection becomes real.)
Tier the lint by authority (auto-fix links/index; report facts/orphans/contradictions). pyweb’s lint currently inherits the flat Hermes list; tiering prevents an agent from “fixing” facts during lint.
Don’t adopt llmwiki-style compilation. pyweb’s wiki is a content source for an Astro site (sync-wiki.mjs → content collections) with human-in-loop ingests — squarely the skill sweet spot. The compiler’s value (batch compile, review queues, OKF exchange, MCP context packs) only materializes if pyweb later wants unattended multi-source pipelines or to serve context packs to other agents. If that day comes, llmwiki can point at the same vault (it’s Obsidian-compatible); no migration needed now.
Keep vector search deferred. At 75 pages pyweb is far below the ~100-source threshold where Karpathy/Astro-Han/NicholasSpisak all agree grep+index suffice. The trigger to revisit is measurable recall degradation, not page count anxiety.
Borrow the single-source-of-truth schema pattern from second-brain if pyweb grows more agent configs: one canonical wiki-schema.md that AGENTS.md/CLAUDE.md/skill files reference rather than duplicate — pyweb already has SCHEMA.md + hermes/AGENTS.md that can drift apart.
Publish-side synergy: llmwiki’s exports (llms.txt, Marp, GraphML) mirror what pyweb’s Astro build already produces (llms-full.txt exists in dist/). pyweb’s static-site output is effectively the “viewer” that llmwiki had to build — a genuine advantage of the pyweb architecture; keep the wiki markdown-native so the Astro pipeline stays a pure projection.