First Principles

This app exists to solve one problem: "I want to easily understand and memorize AI engineering concepts." Everything on this site — every feature, down to the choice of database — is derived from that sentence, so that every design choice feels less like a decision and more like a consequence.

The problem, taken seriously

AI engineering is a wall of interlocking concepts: RAG, embeddings, agents, evals, rerankers, workflows. You have lived the failure mode: you read about embeddings three times, nod along each time, and still cannot explain them out loud a week later. Two distinct things went wrong there — one at comprehension time, one at retention time — and they need distinct fixes:

• Reading is not understanding. You hit a passage that assumes three things you half-know, and a generic explanation — ungrounded, unsourced — leaves you no better off.
• Understanding today is not retaining in three months. The memory literature is unambiguous: durable memory follows retrieval practice, spacing, and rich encoding — not re-reading. Exposure feels like learning; it isn't — learners systematically mistake fluent re-reading for durable knowledge (Bjork, Dunlosky & Kornell 2013; Soderstrom & Bjork 2015).

So the one problem splits into two epistemic needs: understand easily (a grounded explanation at the exact moment of confusion) and memorize durably (a system that forces retrieval on a schedule). Two are the core, not the whole. The same memory literature that justifies the second defines durable learning as a pair — retention and transfer — and recalling what a reranker is still doesn't compile into writing one that runs. So a third need, apply reliably, gets its own lever here: graded micro-builds at /practice, scheduled like memories. And every lever presupposes two things that aren't knowledge at all: that you keep showing up — the streaks and progress you'll meet serve that — and that you know what to reach for next (/next-up). Every learning feature below earns its place by falling out of one of these.

What understanding needs — and where it lives here

An explanation helps a learner only when it is both grounded in a real source and arrives on demand at the moment of confusion. A hallucinated or ungrounded explanation is worse than none: confident but false output systematically deflates a learner’s own epistemic work — automation-induced testimonial injustice (Bao 2026) — and unflagged incorrect help risks inducing the misconception (Große & Renkl 2007). The theoretical case is strong; the absence of a head-to-head RCT merely cautions against overclaiming (learning-science). The system’s answer is to meet each understanding need with a concrete mechanism, all built on the same retrieval substrate — LlamaIndex over an embedded Qdrant store (LlamaIndex primer).

• Read-time grounded retrieval – the “Explain this” flow: a reader selects a passage, the client posts it to app/api/highlight-query/route.ts, which proxies to a stateless or memory-backed RAG service that returns answers with source citations from the site’s own corpus so the learner can verify every claim.
• AI tutor chat – a RAG-grounded conversation (app/api/chat/route.ts) that retrieves lesson chunks via full-text and hybrid ranking, labels context by lesson, and instructs the model to cite the lesson title — keeping every answer anchored to the curriculum (glossary).
• Knowledge-graph search – Cmd+K full-text search over the exported concept graph (data/graph/graph.json), not vector/embedding search (that path was retired). Surfaces the missing prerequisite the learner half-knows, letting them jump along dependency edges.
• Grounded generation of reference pages – the faithfulness gate (FaithfulnessEvaluator in ground_first_principles.py) scores each body against retrieved context and earns one retry on failure; a refusal/shape gate catches non-answers; code lanes add a deterministic identifier check that requires real identifiers traceable to source. All applied at write-time, not read-time.
• Generated curriculum (PRD 05) – the research→outline→draft→review pipeline (kg/ground*.py) runs the same layered gates — faithfulness, refusal, identifier-check — so the curriculum a learner reads is grounded before it ever ships.
• Loci trainer as the encoding bridge – once a concept is understood, the route at /loci (or /memorize) turns atomic facts into palace cards with mnemonics, collision-checked and grounding-gated, scheduling them via FSRS-6 for durable recall.

The bring-your-own-materials study (BYOM) is proposed — extending the same grounded pipeline to learner-uploaded content.

What memorizing needs — and where it lives here

Memory has three levers — retrieval practice, spacing, encoding — and each one is a feature here, not a footnote:

• Retrieval practice → active-recall flashcards at /memorize, with plausible-but-wrong distractors, because retrieval itself — not restudy — is what drives retention (Karpicke & Roediger 2008).
• Spacing → every card is scheduled by FSRS, which models your forgetting curve and asks again just before you'd forget. Spacing is one of the most-replicated effects in the learning literature (Cepeda et al. 2006), and machine-learned schedulers like FSRS are its modern lineage (Settles & Meeder 2016). Daily interleaved review at /memorize/review mixes topics, because blocked practice feels more effective than it is, while interleaving produces the more durable learning — most where concepts are confusable (Brunmair & Richter 2019).
• Encoding → the /loci memory-palace trainer attaches concepts to spatial anchors — the technique memory athletes rely on (Maguire et al. 2003) and that training measurably instills (Dresler et al. 2017) — and grounded mnemonics and memory hooks give abstract terms something vivid to bind to (Atkinson & Raugh 1975).
• Recall everywhere → cues are woven into lesson prose and into the narrated audio, so even listening quizzes you.

The studies behind each lever are catalogued at /learning-science. Every item here is one of the three levers made concrete. The other things you'll notice — streaks, XP, progress — aren't memory levers at all; they're what keeps you coming back to them.

What applying needs — and where it lives here

Recall is not yet skill. You can define a reranker cleanly and still stall at a blank editor — procedural knowledge compiles through production, not retrieval, so applying earns the same rigor the other two got:

• Faded guidance → micro-builds at /practice open as a worked example, drop to a completion problem with the hard step blanked, then to the full problem on your own — the guidance faded exactly as competence grows, because studying worked examples beats unaided problem-solving early in a skill and solo problem-solving wins later (Sweller & Cooper 1985; Renkl & Atkinson 2003).
• Graded against real code → each exercise is generated by a LlamaIndex lane over the same single embedded Qdrant corpus, its expected identifiers checked against this site's actual source and dropped if ungrounded — the identical guarantee the reference pages carry.
• Spaced re-drills → a passed build isn't finished; the same FSRS scheduler that paces flashcards points at micro-builds too, so a skill you proved once returns just before it decays.

And a build you fail rarely tells you which prerequisite you were missing. /next-up closes that gap: it aggregates your own failure signal across cards and builds and walks a concept-prerequisite graph — grounded in the same corpus — so the next thing to study is ordered for you, not guessed.

The shared substrate

Look at what both lists do. "Explain this" retrieves passages. Grounded generation retrieves evidence. Flashcards, mnemonics, and audio cues are generated from retrieved chunks. Every derived feature is retrieval over one corpus. And retrieval needs exactly three things per chunk: an embedding (to find it by similarity, via an approximate-nearest-neighbor index like HNSW), the text (to hand the LLM or the learner), and metadata (to filter by lesson, section, source). The last question of the derivation is forced: how many places do those three things live, and who keeps them consistent?

The default split — and what it costs

Out of the box, LlamaIndex splits them: a vector store holds embedding + ID ("which chunks are similar?"), a docstore holds text + metadata, looked up by that ID afterwards. Retrieval becomes a two-step dance — similarity search returns IDs, a second hop turns IDs into text. The split buys flexibility, but it costs a second moving part, a second I/O hop per query, and a consistency obligation: nothing structurally prevents a vector pointing at a node the docstore no longer has.

Collapse the split

Qdrant stores a payload with every point. If the payload can hold the chunk text and metadata — and it can — the docstore stores nothing the vector DB couldn't. The second store is redundant by construction, so collapse it. Retrieval becomes one hop:

1. Query vector → top-k similar points in Qdrant.
2. The returned points already contain the chunk text in their payload.
3. LlamaIndex builds the context directly — no secondary fetch.

The real code

No flag even needs flipping: the Qdrant integration stores the chunk text in each point's payload by default (stores_text=True, in the field named by text_key) — that default is what makes the separate docstore redundant. index_doc_id=True additionally payload-indexes the document id, so doc-level filters and deletes stay fast. The code spells both defaults out because they are the contract:

from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client

client = qdrant_client.QdrantClient(path="local_qdrant")  # local binary, no Docker

vector_store = QdrantVectorStore(
    collection_name="my_docs",
    client=client,
    index_doc_id=True,   # payload-index the doc id (fast doc-level filters/deletes)
    text_key="text",     # payload field where the chunk text lands (stores_text default)
)

Then build the index with no separate docstore argument — everything lands inside Qdrant:

from llama_index.core import VectorStoreIndex, StorageContext

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

This is not a toy example — it is literally this site's wiring. roadmap-kg/kg/memory_common.py routes every content lane through one embedded Qdrant store at .embed-cache/qdrant; the docstore is empty by design, and the BM25 lanes that need raw nodes hydrate them through a store_nodes() helper over the same payloads.

What it wins, what it costs

Wins:

• One store instead of two moving parts.
• No drift — a vector can never point at a missing node, because the vector is the node's home.
• Backup is a snapshot of one directory.
• Zero extra latency — there is no second hop.

Costs:

• Payload storage grows beyond "vectors only".
• The stores are welded together — swapping vector databases means migrating documents too.
• No SQL over your documents — if you ever need joins, a DuckDB or Postgres docstore is the better home.

For a single site's corpus — lessons, specs, code — the trade is worth it.

The evidence

These claims are load-bearing, so here are the studies behind them — drawn from the same DOI-verified corpus that powers /learning-science:

• Testing effect — Karpicke & Roediger 2008, Science: retrieval, not restudy, drives retention. Meta-analysis: Rowland 2014.
• Spacing — Cepeda et al. 2006: the canonical distributed-practice meta-analysis. Settles & Meeder 2016 and Tabibian et al. 2019: forgetting curves learned and optimized by machine — the lineage FSRS belongs to.
• Interleaving — Brunmair & Richter 2019: helps most exactly where concepts are confusable. Also Rohrer 2012.
• Encoding & the method of loci — Dresler et al. 2017, Neuron: mnemonic training reshapes memory networks. Maguire et al. 2003: superior memorizers rely on spatial routes. Atkinson & Raugh 1975: the keyword mnemonic.
• Worked examples & faded guidance — Sweller & Cooper 1985: studying worked examples beats unaided problem-solving early in skill acquisition. Renkl & Atkinson 2003: fading worked steps into completion problems structures the transition to solo problem-solving.
• The illusion of learning — Bjork, Dunlosky & Kornell 2013 and Soderstrom & Bjork 2015: performance during study is a poor proxy for durable learning.
• Retrieval & grounding, the AI side — Lewis et al. 2020: retrieval-augmented generation. Robertson & Zaragoza 2009: BM25, the sparse leg of this site's hybrid retrieval. Blended RAG 2024: dense+sparse hybrids beat either alone. Malkov & Yashunin 2018: HNSW, the index behind "find it by similarity". Spens & Burgess 2024: biological memory consolidation modeled as compressive RAG.

Bottom line

One problem — understand easily, memorize durably, apply reliably — forced three epistemic needs; the needs derived the features (/memorize, /memorize/review, /loci, /practice, /next-up, "Explain this", the grounded reference pages); the features reduced to retrieval over one corpus; and the corpus reduced to one database holding embedding, text, and metadata together, consistent by construction. The whole learning machine stands on a single source of truth, and every step from the opening sentence to the payload that holds the text follows from the one before it. There is even a pleasing symmetry: recent neuroscience models biological memory consolidation itself as a form of compressive retrieval-augmented generation (Spens & Burgess 2024) — the memory being trained and the store doing the retrieving are, at bottom, the same shape.