---
title: "LLM Wiki Ecosystem: Spec vs. Implementations"
description: "Comparative query analyzing the LLM Wiki specification against real-world implementations, including this site."
section: "queries"
type: "query"
created: "2026-08-23"
updated: "2026-08-23"
tags: ["knowledge-management", "comparison", "agents", "principle"]
canonical: "https://pyweb.dev/wiki/llm-wiki-ecosystem-analysis"
---
# LLM Wiki Ecosystem: Spec vs. Implementations

Deep analysis of Karpathy's LLM Wiki gist against four ecosystem implementations
(Astro-Han, NicholasSpisak/second-brain, atomicstrata/llm-wiki-compiler, wpzero),
filed as a query result. Full report with source links:
` [[source: karpathy-wiki-ecosystem-analysis-2026]](/wiki/raw/articles/karpathy-wiki-ecosystem-analysis-2026)`. Extends [llm wiki pattern](/wiki/llm-wiki-pattern).

## The core finding

Karpathy's gist is deliberately a **pattern, not a spec** — an idea file meant to be
instantiated per-agent. The implementations split into two poles:

- **Pure skills** (Astro-Han, second-brain, wpzero): the schema is a markdown document
  the agent reads; enforcement is prompt discipline plus at most one small checker
  script. Optimized for single-user, interactive, Obsidian-sidecar workflows.
- **Hardcoded compiler** (atomicstrata/llm-wiki-compiler): enforcement moved into
  runtime code — incremental compilation via source hashes, review queues, trust
  gates, embeddings, eval harness. Optimized for batch/CI/team operation.

Both poles converge on one principle: **keep the artifact dumb (plain markdown,
relative links, append-only log) and put enforcement in the cheapest place that
works** — prompt for judgment, a script for mechanical checks, runtime gates only
for CI/team/untrusted-input scale.

## Astro-Han's production-tested design boundaries

After 3 months of daily production use (94 articles / 99 sources), Astro-Han
explicitly *rejected*:

- **Source-hash freshness tracking** — raw/ is immutable; "hashes guard against
  events that cannot happen."
- **Persisted line-number citations** — every observed fidelity error was catchable
  by whole-file grep; anchor friction makes agents skip the rule.
- **Numeric confidence scores** — "false precision with no calibration behind it."
- **Per-article review dates** — maintenance driven by whole-wiki lint, not timers.
- **Vector/graph search** — at 50K–100K tokens of curated wiki, "grep and read are
  more reliable. Add search tooling only when recall measurably degrades."

What survived production:

- **The Grounding Invariant** — every load-bearing fact must exist verbatim in the
  linked raw file; one Python script greps high-signal literals across the wiki in
  seconds. Stateless and total because raw/ is immutable.
- **"No material" triage** — an explicit escape hatch: log the raw, stop, "do not
  force an article out of a thin source."
- **Status blocks (Outdated/Disputed)** — "never silently rewrite history."
- **Tiered lint authority** — auto-fix links and index entries; *report* facts,
  contradictions, and orphans without touching them.

## Operational realities

- **Prompt-only rules that aren't mechanically verifiable silently decay.** This is
  the strongest argument for one small checker script inside an otherwise pure
  skill — demonstrated today on this wiki: the first run of `check_evidence.py`
  found 4 suspect literals in 20 checked pages, of which 2 were real fidelity
  errors (a trailing period added to a Karpathy quote; a quote restated in the
  wrong words) and 2 were attribution/provenance gaps fixed in the page text.
- **Index + grep scales to ~100 sources** per Karpathy, Astro-Han, and
  second-brain — all three independently gate search tooling on the same threshold.
- **llmwiki-style compilation only pays off** for unattended multi-source
  pipelines, review-gated untrusted imports, or serving context packs to other
  agents. Its own README warns against it for interactive single-author use.

## What changed in this wiki as a result

1. `wiki/scripts/check_evidence.py` — mechanical grounding-invariant lint.
2. SCHEMA.md Operational Rules: triage dispositions (incl. **No material**),
   Status blocks, grounding invariant, tiered lint authority.
3. Four fidelity fixes applied to existing pages (pages fixed, raw untouched).

## Open question for this wiki

The Hermes `llm-wiki` skill (this wiki's origin) mandates sha256 raw frontmatter
and `confidence:` scores — both on Astro-Han's rejected list. Kept for now: the
wiki-update cron ingests unattended, where drift detection is real. Revisit if
ingest friction becomes the binding constraint.

## Related

- [llm wiki pattern](/wiki/llm-wiki-pattern) — the base pattern this analysis extends
- [context engineering](/wiki/context-engineering) — the wiki as compiled agent context
- [andrej karpathy](/wiki/andrej-karpathy) — the pattern's author
- [error analysis and evals](/wiki/error-analysis-and-evals) — evidence linting as an eval gate

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 26.4

### References (outbound)
- [LLM Wiki Pattern](https://pyweb.dev/wiki/llm-wiki-pattern.md)
- [Context Engineering](https://pyweb.dev/wiki/context-engineering.md)
- [Andrej Karpathy](https://pyweb.dev/wiki/andrej-karpathy.md)
- [Error Analysis and Evals](https://pyweb.dev/wiki/error-analysis-and-evals.md)

### Referenced by (inbound)
- [Agentic Engineering Entity Accounting](https://pyweb.dev/wiki/agentic-engineering-entity-accounting.md)

### Evidence (verified primary sources)
- [karpathy-wiki-ecosystem-analysis-2026](https://pyweb.dev/wiki/raw/articles/karpathy-wiki-ecosystem-analysis-2026.md)

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt