LLM Wiki Ecosystem: Spec vs. Implementations

Deep analysis of Karpathy’s LLM Wiki gist against four ecosystem implementations (Astro-Han, NicholasSpisak/second-brain, atomicstrata/llm-wiki-compiler, wpzero), filed as a query result. Full report with source links: [[source: karpathy-wiki-ecosystem-analysis-2026]](/wiki/raw/articles/karpathy-wiki-ecosystem-analysis-2026). Extends llm wiki pattern.

The core finding

Karpathy’s gist is deliberately a pattern, not a spec — an idea file meant to be instantiated per-agent. The implementations split into two poles:

• Pure skills (Astro-Han, second-brain, wpzero): the schema is a markdown document the agent reads; enforcement is prompt discipline plus at most one small checker script. Optimized for single-user, interactive, Obsidian-sidecar workflows.
• Hardcoded compiler (atomicstrata/llm-wiki-compiler): enforcement moved into runtime code — incremental compilation via source hashes, review queues, trust gates, embeddings, eval harness. Optimized for batch/CI/team operation.

Both poles converge on one principle: keep the artifact dumb (plain markdown, relative links, append-only log) and put enforcement in the cheapest place that works — prompt for judgment, a script for mechanical checks, runtime gates only for CI/team/untrusted-input scale.

Astro-Han’s production-tested design boundaries

After 3 months of daily production use (94 articles / 99 sources), Astro-Han explicitly rejected:

• Source-hash freshness tracking — raw/ is immutable; “hashes guard against events that cannot happen.”
• Persisted line-number citations — every observed fidelity error was catchable by whole-file grep; anchor friction makes agents skip the rule.
• Numeric confidence scores — “false precision with no calibration behind it.”
• Per-article review dates — maintenance driven by whole-wiki lint, not timers.
• Vector/graph search — at 50K–100K tokens of curated wiki, “grep and read are more reliable. Add search tooling only when recall measurably degrades.”

What survived production:

• The Grounding Invariant — every load-bearing fact must exist verbatim in the linked raw file; one Python script greps high-signal literals across the wiki in seconds. Stateless and total because raw/ is immutable.
• “No material” triage — an explicit escape hatch: log the raw, stop, “do not force an article out of a thin source.”
• Status blocks (Outdated/Disputed) — “never silently rewrite history.”
• Tiered lint authority — auto-fix links and index entries; report facts, contradictions, and orphans without touching them.

Operational realities

• Prompt-only rules that aren’t mechanically verifiable silently decay. This is the strongest argument for one small checker script inside an otherwise pure skill — demonstrated today on this wiki: the first run of check_evidence.py found 4 suspect literals in 20 checked pages, of which 2 were real fidelity errors (a trailing period added to a Karpathy quote; a quote restated in the wrong words) and 2 were attribution/provenance gaps fixed in the page text.
• Index + grep scales to ~100 sources per Karpathy, Astro-Han, and second-brain — all three independently gate search tooling on the same threshold.
• llmwiki-style compilation only pays off for unattended multi-source pipelines, review-gated untrusted imports, or serving context packs to other agents. Its own README warns against it for interactive single-author use.

What changed in this wiki as a result

1. wiki/scripts/check_evidence.py — mechanical grounding-invariant lint.
2. SCHEMA.md Operational Rules: triage dispositions (incl. No material), Status blocks, grounding invariant, tiered lint authority.
3. Four fidelity fixes applied to existing pages (pages fixed, raw untouched).

Open question for this wiki

The Hermes llm-wiki skill (this wiki’s origin) mandates sha256 raw frontmatter and confidence: scores — both on Astro-Han’s rejected list. Kept for now: the wiki-update cron ingests unattended, where drift detection is real. Revisit if ingest friction becomes the binding constraint.

Related

• llm wiki pattern — the base pattern this analysis extends
• context engineering — the wiki as compiled agent context
• andrej karpathy — the pattern’s author
• error analysis and evals — evidence linting as an eval gate