LlamaIndex Primer
🦙 The RAG engine that grounds and generates every guide page — Documents and chunks, embeddings, and the vector index, in plain language first. Every section leads with the simple take and the real code; the dense system-design detail is tucked behind a “deep dive” toggle. Select any passage to get it explained.
What you’ll be able to explain
Why source files are split into small overlapping chunks before anything else — and what breaks without the overlap.
How code is split on function boundaries (not sentences) so the grounding check can verify that quoted identifiers are real.
How chunks become searchable vectors with a local embedding model and a cache — no embedding API involved.
The query path: retrieve the top chunks, drop low-relevance noise, reorder for position, then answer ONLY from what was retrieved.
How hybrid retrieval plus a cross-encoder reranker sharpen results, and how the verifier throws away any excerpt whose code isn't in the real source.
Key terms, in plain words
— one source file loaded as raw text, ready to be indexed.
— a bite-sized slice of a document — small enough to search precisely.
— a list of numbers capturing what a chunk means, so similar ideas land near each other.
— the searchable pile of all chunks, arranged by meaning rather than keywords.
— the lookup step: given a question, fetch the few most relevant chunks.
— retriever + language model: fetch the right chunks, then write an answer from ONLY those.
— a second, sharper judge that re-orders the fetched chunks so the best evidence comes first.
— the proofreader: every identifier in generated code must exist in the real source, or the excerpt is thrown away.
The learning science behind it
The RAG pipeline this primer teaches is, stage for stage, a working model of human memory science. Each lens below pairs one pipeline stage with the documented memory-science principle it mirrors, states the mechanism in plain terms, and links to the full write-up on the learning-science principles page. The prose is generated by LlamaIndex, grounded ONLY in that principles corpus — not paraphrased from memory.
Chunking
Working memory is limited to around four chunks, so recoding raw material into meaningful units multiplies capacity. In a retrieval-augmented generation pipeline, a splitter cuts large files into small overlapping chunks that each hold one graspable idea. This recoding into manageable units respects chunk limits, making retrieval feasible where a whole file would overwhelm working memory.
Mirrors Documents and chunks
Cowan (2001) · Elsner et al. (2026) · Ericsson et al. (1980) · Lee et al. (2025) · Miller (1956) · Sennrich et al. (2016)
Encoding specificity and transfer-appropriate processing: a cue works at retrieval only to the extent it was part of encoding. In a retrieval-augmented generation pipeline, the same model encodes both stored chunks and incoming questions, so the retrieval cue’s encoding exactly overlaps the space the material was encoded into. This ensures the cue can land near the right trace, matching the original encoding.
Mirrors Embeddings without an API
Godden & Baddeley (1975) · Kang et al. (2025) · Lewis et al. (2020) · Tulving & Thomson (1973)
The mechanism of retrieval practice (the testing effect) is that successful effortful retrieval strengthens and multiplies retrieval routes in a way re‑exposure does not. In a retrieval‑augmented generation pipeline, the query engine must actively pull relevant chunks from the index and answer solely from that retrieved content, rather than paraphrasing from the model’s memory. This effortful retrieval mirrors the principle: the answer is built from recalled information, not from re‑exposure to stored knowledge.
Mirrors Asking the index
Eldho Paul & Sunar (2026) · Karpicke & Blunt (2011) · Ramsauer et al. (2020) · Roediger & Karpicke (2006)
Imageability and dual coding makes memory stick because concrete, picturable content gets two independent retrieval routes—verbal and imaginal. This mirrors a retriever that reaches the same stored chunks through two independent routes, one matching semantic meaning and one matching exact keywords, then fuses the rankings, so material findable by either route gets recalled.
Mirrors Hybrid retrieval (dense + keyword fusion)
Brysbaert et al. (2014) · Paivio (1991) · Radford et al. (2021) · Roy et al. (2026) · Wu & Smith (2023)
The mechanism: memory dramatically rewards an item that differs from its neighbors. A second-pass reranker reads the question with each candidate chunk and promotes the one that genuinely stands apart from its lookalike neighbors. This directly implements Distinctiveness (von Restorff / isolation effect) by rejecting near-duplicates that first-pass similarity treats as equally good.
Mirrors Cross-encoder reranking
Atzert et al. (2026) · Hunt & McDaniel (1993) · Koneru & Powell (2026) · Lee et al. (2024) · Robinson et al. (2020) · von Restorff (1933)
Desirable difficulties states that conditions slowing apparent learning enhance retention because effortful processing builds storage strength over retrieval strength. A verification gate that checks each code excerpt against the source, feeds back failures, and retries mirrors this: the deliberate difficulty forces effortful verification, and because the system can overcome the difficulty, what survives is far more reliable.
Mirrors Trust but verify
Bengio et al. (2009) · Bjork (1994) · Hwang et al. (2026) · Lee et al. (2026) · Shu et al. (2026)
The generation effect states that self-produced material is remembered better than received material. In a retrieval-augmented generation pipeline, the page builds itself by generating content from sources rather than copying, making that self-produced text more memorable. A quality check mirrors the principle's trade-off: generated content from novices can be weak, so the pipeline ensures high-quality cues.
Mirrors From index to page
Shao et al. (2026) · Slamecka & Graf (1978) · Zelikman et al. (2022)
The spacing effect works because distributed practice flattens the forgetting curve more effectively than massed practice. A code-aware splitter that overlaps function boundaries re-encounters the same material across separate adjacent chunks, spacing those retrievals rather than massing them in a single chunk. This mirrors the principle by distributing the material’s appearances, strengthening memory through spaced rather than massed exposure.
Mirrors Code-aware splitting
Cepeda et al. (2006) · Cepeda et al. (2008) · Ebbinghaus (1885) · Rolnick et al. (2019) · Settles & Meeder (2016) · Spens & Burgess (2024) · Xiao & Wang (2024)
Elaboration and levels of processing holds that memory is a byproduct of processing depth: semantic processing outperforms surface forms, and elaboration is most effective when it forms an integrated, coherent structure. A retrieval-augmented generation index that files chunks by semantic vectors rather than surface words encodes deep meaning, enabling retrieval from many differently-worded but related queries, mirroring how integrated cues make memories recoverable.
Mirrors The index
Balepur et al. (2024) · Craik & Lockhart (1972) · Craik & Tulving (1975) · Devlin et al. (2019)
Interactive imagery makes memory stick because interacting items in a single scene produce far larger gains than separate or bizarre-but-separate images—interaction is the load-bearing variable. A retrieval-augmented generation pipeline mirrors this by reading all separately retrieved chunks together in one prompt, binding them into a single integrated answer. This joins separate facts into one connected response, just as interactive imagery requires keyword and referent to interact in a single scene.
Mirrors The LLM behind the engine
Atkinson & Raugh (1975) · Lee & Lan (2023) · Santoro et al. (2017) · Wollen, Weber & Lowry (1972)
The bizarreness effect works because unusual items receive more elaborative processing and suffer less retrieval interference, but only when they are rare in their context. A sparse keyword retriever mirrors this by weighting low-frequency terms, so a rare exact identifier dominates the match and is pulled to the top, just as bizarre cues stand out best when mixed among ordinary material.
Mirrors Keyword (BM25) exact-match retrieval
Atzert et al. (2026) · Lee et al. (2024) · McDaniel & Einstein (1986) · Schaul et al. (2015)
The method of loci works by parasitizing spatial memory—evolutionarily old, high-capacity, with built-in ordering and cues—giving each item a fixed location along a route. In a retrieval-augmented generation pipeline, every chunk sits at a fixed coordinate in a vector space determined by its meaning; retrieval finds material by walking to the nearest locations, navigating back to each stable place.
Mirrors Nearest-neighbour vector space
Dresler et al. (2017) · Graves et al. (2016) · Maguire et al. (2002) · Wulff et al. (2026)
Emotional arousal and humor works because arousing content gets consolidation priority, with the memory advantage growing over a delay. A retriever that uses a relevance score as a salience signal mirrors this: it drops low-salience noise and prioritises only the highest-scoring chunks into the answer, ensuring that the most salient material is consolidated into the page.
Mirrors Relevance scoring and the similarity floor
Balepur et al. (2024) · Kleinsmith & Kaplan (1963) · Lee et al. (2024) · Schaul et al. (2015) · Schmidt (1994)
The shared mechanism is encoding specificity: retrieval succeeds only when the retrieval cue matches the encoded trace. The generation effect ensures learners actively generate these cues, with multimodal LLM scaffolding providing a quality floor. Transfer-appropriate processing aligns practice with the criterion task. Chunking multiplies working-memory capacity by recoding raw material into meaningful units. This same cue-trace match reappears in retrieval-augmented generation, where the query embedding must land near the passage embeddings used to write the store.
How to actually remember this
Reading explains; retrieval retains. The study-aid cards above are retrieval practice — here is the research behind the method, and why this page is structured the way it is.