Architecture Diagram Examples

The 50 architectures worth sketching in a live-coding interview, sized for a 90-second whiteboard drawing. Each card shows the rendered diagram, when to reach for it, and the trade-offs to say out loud while drawing. Expand “Mermaid source” to copy a chart into the whiteboard’s diagram tool. Explaining the take-home itself? Sears assignment diagrams →

Agent loops

The while-loops at the heart of every agent — tool use, ReAct, planning, reflection, approval gates.

Basic tool-use loop

The while-loop every agent is: model decides, tool runs, result feeds back, repeat until a final answer.

Sketch this when: Sketch this first for anything agentic — every framework is sugar over this loop.

  • The message list is the only state; everything else is derived.
  • Loop needs a stop condition: max turns, token budget, or the model answering without a tool call.
  • Tool errors go back into the loop as observations — the model self-corrects.
Deep dive: Tool calling
Mermaid source
graph TD
  user[User request] --> llm[LLM]
  llm --> decide{Tool call requested?}
  decide -->|yes| exec[Execute tool]
  exec --> append[Append result to messages]
  append --> llm
  decide -->|no| answer(Final answer)

ReAct (Reason + Act)

Interleaved Thought → Action → Observation until the model has enough evidence to answer.

Sketch this when: Sketch when asked how the agent decides which tool to call next.

  • Explicit reasoning before each action makes the trace auditable.
  • Costs a reasoning step per action — slower and pricier than a one-shot plan.
  • Modern tool-use models internalize this; you rarely hand-write the Thought scaffold now.
Deep dive: Agents lab
Mermaid source
graph TD
  task[Task] --> thought[Thought: reason about next step]
  thought --> act[Action: call a tool]
  act --> obs[Observation: tool result]
  obs --> thought
  thought -->|enough evidence| answer(Answer)

Plan-and-execute

Plan once up front, execute steps cheaply, replan only when a step fails.

Sketch this when: Sketch when the task has a knowable shape — migrations, multi-file edits, research briefs.

  • One expensive planning call, then cheap execution — better cost profile than ReAct per-step reasoning.
  • The plan is inspectable and approvable before anything runs.
  • Weakness: plans go stale when the environment shifts mid-run — hence the replanner edge.
Deep dive: Agent planning
Mermaid source
graph LR
  task[Task] --> planner[Planner LLM]
  planner --> plan[Step list]
  plan --> exec[Executor]
  exec --> ok{Step succeeded?}
  ok -->|yes| next[Next step]
  next --> exec
  ok -->|no| replan[Replanner]
  replan -.-> plan
  next -->|all done| out(Result)

Reflexion / self-critique

Generate, critique against a rubric, revise — bounded iterations.

Sketch this when: Sketch when asked how to improve output quality without a bigger model.

  • The critic can be the same model with a different prompt — separation of roles beats raw scale.
  • Always bound the loop; unbounded self-critique oscillates and burns tokens.
  • Critique against an explicit rubric, not vibes — otherwise the critic just praises.
Deep dive: Self-healing agents
Mermaid source
graph TD
  task[Task] --> gen[Generator LLM]
  gen --> draft[Draft]
  draft --> critic[Critic scores vs rubric]
  critic --> pass{Pass?}
  pass -->|yes| out(Final output)
  pass -->|no, under 3 rounds| notes[Revision notes]
  notes -.-> gen
  pass -->|round limit| out

Human-in-the-loop approval gate

Reads flow through; writes, sends, and payments pause for a human.

Sketch this when: Sketch when asked how to make an agent safe to deploy.

  • Classify tools by blast radius, not by name — the gate is policy, not a hardcoded list.
  • Rejections feed back as observations so the agent can propose an alternative.
  • Async approval means the run must be resumable — pairs with durable execution.
Deep dive: Agent guardrails
Mermaid source
graph TD
  agent[Agent] --> risk{Tool risk class}
  risk -->|read-only| exec[Execute]
  risk -->|write, send, pay| queue[Approval queue]
  queue --> human[Human review]
  human -->|approve| exec
  human -->|reject| deny[Rejection reason]
  exec --> result[Result]
  result --> agent
  deny -.-> agent

Deep-research loop

Decompose, search, read, accumulate notes, stop on coverage, synthesize with citations.

Sketch this when: Sketch when the prototype involves gathering and synthesizing information.

  • The notes store is what makes it scale past one context window.
  • The stop condition is the hard part — coverage check, not iteration count.
  • Citations must point at the notes, not the model's memory.
Mermaid source
graph TD
  q[Research question] --> decomp[Decompose into sub-questions]
  decomp --> search[Search + fetch]
  search --> read[Read + extract]
  read --> notes[(Notes store)]
  notes --> gap{Coverage enough?}
  gap -->|no, new queries| search
  gap -->|yes| synth[Synthesize with citations]
  synth --> report(Report)

Computer-use loop

Screenshot → pick action → execute → new screenshot, with a safety stop.

Sketch this when: Sketch when the task needs a GUI no API covers.

  • Slowest, most brittle tool — use only when no API exists.
  • Grounding (finding the right pixel) is the main failure mode.
  • Run in a disposable VM; the safety check is non-negotiable for real accounts.
Deep dive: Computer use
Mermaid source
graph TD
  goal[Goal] --> shot[Screenshot]
  shot --> model[Model picks action click, type, scroll]
  model --> safe{Safe action?}
  safe -->|yes| exec[Execute in VM]
  safe -->|no| stop(Stop + ask human)
  exec --> shot
  model -->|task complete| done(Done)

Multi-agent patterns

Orchestrators, supervisors, pipelines, debates — how agents divide and coordinate work.

Orchestrator–workers (parallel subagents)

A lead agent decomposes the task, fans out to parallel workers, aggregates results.

Sketch this when: Sketch when the task splits into independent pieces — the classic parallel-agents answer.

  • Workers get isolated contexts — the orchestrator keeps conclusions, not raw dumps.
  • Only works when subtasks are truly independent; shared state forces coordination.
  • Wall-clock cost is the slowest worker, token cost is all of them.
Deep dive: Multi-agent orchestration
Mermaid source
graph TD
  task[Big task] --> lead[Orchestrator]
  lead --> split[Decompose into independent subtasks]
  split --> w1[Worker 1]
  split --> w2[Worker 2]
  split --> w3[Worker N]
  w1 & w2 & w3 --> agg[Aggregate + reconcile]
  agg --> lead
  lead --> out(Answer)

Supervisor / router handoff

One supervisor routes each turn to a specialist; specialists hand control back.

Sketch this when: Sketch for chat products with distinct domains needing different tools or prompts.

  • One shared conversation, many system prompts — the user never sees the seams.
  • Router quality is the ceiling; misroutes look like incompetence.
  • Specialists stay small and testable in isolation.
Deep dive: Multi-agent orchestration
Mermaid source
graph TD
  user[User turn] --> sup[Supervisor]
  sup -->|billing intent| a1[Billing agent]
  sup -->|tech intent| a2[Support agent]
  sup -->|sales intent| a3[Sales agent]
  a1 & a2 & a3 -.->|hand back| sup
  sup --> reply(Reply)

Sequential agent pipeline

Researcher → writer → editor; each stage's output is the next stage's input.

Sketch this when: Sketch for content or transformation work with clear stage boundaries.

  • Each stage is independently evaluable — pin down which stage regressed.
  • Latency adds up linearly; parallelize only what's independent.
  • Contracts between stages (schemas) prevent drift.
Mermaid source
graph LR
  brief[Brief] --> res[Researcher]
  res --> notes[Notes]
  notes --> writer[Writer]
  writer --> draft[Draft]
  draft --> editor[Editor]
  editor --> final(Final copy)
  editor -.->|revision requests| writer

Evaluator–optimizer pair

A generator and an evaluator loop until the output clears the bar.

Sketch this when: Sketch when quality matters more than latency — code, copy, translations.

  • Works because judging is easier than generating.
  • The evaluator needs concrete criteria; a vague judge converges on nothing.
  • Cap the rounds and keep the best-so-far candidate.
Deep dive: Judge panels
Mermaid source
graph LR
  task[Task] --> gen[Generator]
  gen --> cand[Candidate]
  cand --> eval[Evaluator]
  eval -->|below bar| fb[Feedback]
  fb -.-> gen
  eval -->|pass| out(Accepted output)

Debate + judge

Two agents argue opposing positions; a judge model picks or merges.

Sketch this when: Sketch for high-stakes judgment calls where one model would be sycophantic.

  • Forcing opposing positions surfaces evidence a single pass hides.
  • The judge sees arguments, not just conclusions.
  • 3× the cost — reserve for decisions worth it.
Deep dive: Judge panels
Mermaid source
graph TD
  claim[Hard question] --> a[Agent A argues for]
  claim --> b[Agent B argues against]
  a -.->|rebuts| b
  b -.->|rebuts| a
  a --> judge[Judge model]
  b --> judge
  judge --> verdict(Verdict + rationale)

Hierarchical teams

A supervisor of supervisors; teams own domains, escalation flows up the tree.

Sketch this when: Sketch when a flat orchestrator's context would drown in worker output.

  • Each level compresses: workers report conclusions, leads report summaries.
  • Deep trees add latency and telephone-game distortion — two levels is usually enough.
  • Escalation paths matter more than the happy path.
Deep dive: Multi-agent orchestration
Mermaid source
graph TD
  top[Top supervisor] --> t1[Research lead]
  top --> t2[Build lead]
  t1 --> r1[Searcher]
  t1 --> r2[Reader]
  t2 --> b1[Coder]
  t2 --> b2[Reviewer]
  r1 & r2 -.-> t1
  b1 & b2 -.-> t2
  t1 & t2 -.->|report / escalate| top

Blackboard / shared workspace

Agents coordinate by reading and writing a shared store instead of messaging each other.

Sketch this when: Sketch when agents run at different speeds or restart — state must outlive any one agent.

  • Decouples agents in time; no one blocks on a message.
  • Needs conventions (file names, schemas) or agents clobber each other.
  • This is how coding agents already work — the repo is the blackboard.
Mermaid source
graph TD
  bb[(Shared workspace files, scratchpad, DB)]
  a1[Agent A] -->|writes findings| bb
  a2[Agent B] -->|writes findings| bb
  bb -->|reads context| a1
  bb -->|reads context| a2
  bb --> ctl[Controller watches for completion]
  ctl --> out(Assembled result)

Tools & MCP

Function calling on the wire, MCP topology, tool routing, sandboxes, text-to-SQL.

Function-calling round-trip

The wire-level truth under every framework: schemas in, tool_call out, result back, text out.

Sketch this when: Sketch when asked what actually happens when an agent calls a tool.

  • The model never executes anything — it emits a request; your code runs it.
  • Schema quality is prompt engineering; descriptions steer tool choice.
  • Two LLM calls minimum per tool use — that's the latency floor.
Deep dive: Tool calling
Mermaid source
sequenceDiagram
  participant App
  participant LLM
  participant Tool
  App->>LLM: messages + tool schemas
  LLM-->>App: tool_call get_weather(city)
  App->>Tool: execute get_weather
  Tool-->>App: result JSON
  App->>LLM: messages + tool result
  LLM-->>App: final text answer

MCP topology

One host with an MCP client talking to N servers, each exposing tools, resources, and prompts.

Sketch this when: Sketch when MCP comes up — it's the USB-C analogy drawn out.

  • MCP standardizes the plug, not the tools — any client works with any server.
  • stdio for local processes, HTTP for remote; same protocol.
  • Servers declare capabilities; the client discovers them at connect time.
Deep dive: MCP
Mermaid source
graph LR
  subgraph host[Host application]
    llm[LLM agent]
    client[MCP client]
    llm --> client
  end
  client -->|stdio| files[Files MCP server]
  client -->|HTTP| gmail[Gmail MCP server]
  client -->|HTTP| db[Postgres MCP server]
  files & gmail & db -.->|tools, resources, prompts| client

MCP inside an agent stack

Where MCP sits in a real agent: one registry mixing native tools and MCP servers, auth per server.

Sketch this when: Sketch when asked where credentials live in an MCP setup.

  • The agent sees one flat tool list; MCP is invisible at the loop level.
  • Each server owns its auth — the model never touches credentials.
  • MCP tool results are untrusted input — injection defense applies.
Deep dive: Corti MCP case study
Mermaid source
graph TD
  user[User] --> runtime[Agent runtime tool-use loop]
  runtime --> reg[Tool registry]
  reg --> native[Native tools]
  reg --> client[MCP client]
  subgraph auth[Auth boundary per server]
    s1[Internal API server service token]
    s2[SaaS server user OAuth]
  end
  client --> s1
  client --> s2
  s1 & s2 -.->|results| runtime

Tool router for large toolsets

Retrieve the relevant 10 tools out of 500 before the LLM call — schemas are context too.

Sketch this when: Sketch when the tool count outgrows the context budget or tool choice degrades.

  • Tool descriptions are a retrieval corpus — RAG over your own toolbox.
  • Fewer, relevant schemas beat many: tool-choice accuracy drops as the list grows.
  • Cache the catalog embeddings; only the query embeds at request time.
Mermaid source
graph LR
  q[User query] --> qe[Embed query]
  cat[(Tool catalog 500 descriptions)] --> ce[Embedded descriptions]
  qe --> sim[Similarity search]
  ce --> sim
  sim --> top[Top 10 tool schemas]
  top --> llm[LLM call with only relevant tools]
  llm --> out(Tool call / answer)

Code-execution sandbox

The universal escape-hatch tool: model writes code, an isolated sandbox runs it.

Sketch this when: Sketch when the task needs math, data wrangling, or anything without a dedicated tool.

  • One tool that subsumes hundreds — code is the most general interface.
  • Isolation is the whole design: no network, resource caps, throwaway filesystem.
  • Errors are observations; the fix-and-retry loop is where the value is.
Mermaid source
graph TD
  llm[LLM] -->|writes code| code[Script]
  code --> sb[Sandbox no network, CPU + time caps]
  sb --> res[stdout, files, error]
  res -->|observation| llm
  llm -.->|fix + retry| code
  llm --> out(Answer + artifacts)

Text-to-SQL with repair loop

NL → schema retrieval → SQL → validate → execute read-only → summarize; errors feed back.

Sketch this when: Sketch when the prototype needs to answer questions over a database.

  • Never hand the model the whole schema — retrieve relevant tables like documents.
  • Read-only role and row limits are the guardrail, not prompt instructions.
  • Showing the SQL keeps wrong answers debuggable.
Deep dive: Text-to-SQL
Mermaid source
graph TD
  q[NL question] --> schema[Retrieve relevant tables + columns]
  schema --> gen[LLM writes SQL]
  gen --> check[Validate + EXPLAIN]
  check -->|error| repair[Error message fed back]
  repair -.-> gen
  check -->|ok| run[Execute read-only]
  run --> summ[Summarize rows]
  summ --> out(Answer + SQL shown)

RAG

From naive retrieve-and-stuff to reranking, hybrid search, agentic and graph-based retrieval.

Naive RAG

The baseline: chunk-embed-store offline; embed-retrieve-stuff online. Draw it, then improve it aloud.

Sketch this when: Sketch as the starting point for any knowledge-grounded prototype, then name its failure modes.

  • Two planes with different lifecycles — ingest is batch, query is latency-bound.
  • Chunking strategy usually matters more than embedding model choice.
  • Fails on: multi-hop questions, keyword-exact queries, stale indexes — segue to the variants.
Deep dive: RAG
Mermaid source
graph TD
  subgraph ingest[Ingest — offline]
    docs[Documents] --> chunk[Chunk]
    chunk --> embed[Embed]
    embed --> vs[(Vector store)]
  end
  subgraph query[Query — online]
    q[Question] --> qe[Embed query]
    qe --> topk[Top-k chunks]
    topk --> prompt[Stuff into prompt]
    prompt --> llm[LLM]
    llm --> ans(Grounded answer)
  end
  vs --> topk

RAG + reranking

Over-retrieve with cheap vectors, rerank with a cross-encoder, keep the top 5.

Sketch this when: Sketch as the first upgrade when retrieval quality is the complaint.

  • Bi-encoders are fast but lossy; the cross-encoder reads query and doc together.
  • Cheapest large quality win in RAG — one extra model, no reindexing.
  • Adds ~100ms; rerank depth is the tuning knob.
Deep dive: RAG
Mermaid source
graph LR
  q[Query] --> retr[Vector search top 50]
  retr --> rr[Cross-encoder reranker]
  rr --> top[Top 5]
  top --> llm[LLM]
  llm --> ans(Answer)

Hybrid retrieval (BM25 + dense)

Keyword and vector search in parallel, fused by reciprocal rank.

Sketch this when: Sketch when queries contain identifiers, SKUs, error codes — things embeddings blur.

  • Embeddings miss exact tokens; BM25 misses paraphrase — they fail on disjoint queries.
  • RRF needs no score calibration between the two systems.
  • Most production RAG is hybrid; pure-vector is a demo simplification.
Deep dive: RAG
Mermaid source
graph TD
  q[Query] --> bm[BM25 keyword search]
  q --> dense[Dense vector search]
  bm --> fuse[Reciprocal-rank fusion]
  dense --> fuse
  fuse --> rr[Rerank]
  rr --> llm[LLM]
  llm --> ans(Answer)

Agentic RAG

Retrieval as a tool: the model decides when to retrieve, rewrites queries, loops on weak results.

Sketch this when: Sketch when single-shot retrieval fails on multi-hop or vague questions.

  • The model's query rewrite usually beats the user's phrasing.
  • Grading retrieved context prevents confident answers from junk.
  • Latency is now variable — budget loops explicitly.
Deep dive: RAG
Mermaid source
graph TD
  q[Question] --> agent[Agent LLM]
  agent --> need{Need more context?}
  need -->|yes| rw[Rewrite / split query]
  rw --> retr[Retrieve]
  retr --> grade[Grade results]
  grade -.->|weak, retry| rw
  grade -->|good| agent
  need -->|no| ans(Answer with citations)

Corrective / Self-RAG

Grade each retrieved doc; rewrite the query or fall back to web search when retrieval fails.

Sketch this when: Sketch when asked how RAG should behave when the index doesn't have the answer.

  • The honest failure mode is 'not in the corpus' — this pattern makes it explicit.
  • Doc-level grading catches the retrieval lying before the LLM does.
  • The fallback needs its own trust label — web results aren't your corpus.
Mermaid source
graph TD
  q[Query] --> retr[Retrieve top-k]
  retr --> grade[Grade each doc]
  grade -->|relevant| gen[Generate answer]
  grade -->|ambiguous| rewrite[Rewrite query]
  rewrite -.-> retr
  grade -->|all irrelevant| web[Web search fallback]
  web --> gen
  gen --> ans(Answer)

Multi-index router

Classify the query, then send it to docs, tickets, SQL, or a live API — one interface, many sources.

Sketch this when: Sketch when the knowledge lives in heterogeneous systems, not one corpus.

  • One index per data shape beats one mega-index with mixed chunks.
  • The router is cheap and testable — a classification eval, not a generation eval.
  • Some queries need two sources; allow fan-out, then synthesize.
Mermaid source
graph TD
  q[Query] --> router[Router LLM classify intent]
  router -->|product docs| i1[(Docs index)]
  router -->|history| i2[(Tickets index)]
  router -->|numbers| sql[Text-to-SQL]
  router -->|live data| api[API tool]
  i1 & i2 & sql & api --> synth[Synthesize]
  synth --> ans(Answer)

Ingestion / sync pipeline

Where staleness lives: change detection, parse, chunk with metadata, embed, upsert, tombstone.

Sketch this when: Sketch when asked why the RAG answer is out of date.

  • Deletes are the bug factory — removed docs must tombstone their chunks.
  • Metadata (source, date, ACL) at chunk time enables filtered retrieval later.
  • Change detection makes re-sync cheap enough to run often.
Mermaid source
graph LR
  cron[Scheduler / webhook] -.-> src[Sources wiki, tickets, code]
  src --> diff{Changed since last sync?}
  diff -->|no| skip(Skip)
  diff -->|yes| parse[Parse + clean]
  parse --> chunk[Chunk + attach metadata]
  chunk --> embed[Embed]
  embed --> up[(Upsert + delete stale chunks)]

GraphRAG

Extract entities and relations into a graph; answer entity questions locally, theme questions globally.

Sketch this when: Sketch when questions span the whole corpus — 'what are the main themes' breaks chunk retrieval.

  • Vector RAG answers point questions; graph communities answer corpus-level ones.
  • Extraction is expensive LLM work at ingest — pay once, query cheap.
  • Overkill for FAQ bots; earns its keep on investigative or analytical corpora.
Mermaid source
graph TD
  docs[Documents] --> extract[LLM extracts entities + relations]
  extract --> kg[(Knowledge graph)]
  kg --> comm[Community detection + summaries]
  q[Query] --> mode{Local or global question?}
  mode -->|about an entity| kg
  mode -->|about themes| comm
  kg --> gen[Generate]
  comm --> gen
  gen --> ans(Answer)

Evals & observability

How you know it works: eval harnesses, LLM judges, feedback flywheels, tracing, A/B prompts.

Offline eval harness

Dataset in, system under test, code assertions plus an LLM judge, scores gate the merge.

Sketch this when: Sketch immediately when asked 'how do you know it works?' — the strongest signal you can send.

  • Assert what's checkable in code (schema, facts present); judge only what isn't (tone, quality).
  • The dataset starts tiny — 20 real cases beat 500 synthetic ones.
  • Run it in CI like tests; a prompt change is a code change.
Deep dive: Evals
Mermaid source
graph LR
  ds[(Eval dataset inputs + expected)] --> run[Run system under test]
  run --> outs[Outputs]
  outs --> code[Code assertions]
  outs --> judge[LLM judge]
  code & judge --> scores[Scores]
  scores --> gate{Above baseline?}
  gate -->|yes| ship(Merge / deploy)
  gate -->|no| block(Block + report)

LLM-as-judge with calibration

A judge model scores outputs against a rubric — and is itself checked against human labels.

Sketch this when: Sketch when proposing to grade outputs with a model — pre-empt the 'who judges the judge' question.

  • A judge without calibration is vibes at scale.
  • Rubrics with concrete criteria beat 'rate 1–10' — anchor each score level.
  • Judges drift with model updates; re-calibrate on upgrade.
Deep dive: Judge panels
Mermaid source
graph TD
  out[System output] --> judge[Judge model]
  rubric[Rubric criteria] --> judge
  judge --> score[Score + critique]
  human[(Human-labeled sample)] --> cal[Calibration: judge vs human]
  score --> cal
  cal -->|agreement low| fix[Revise rubric or judge model]
  fix -.-> judge

Online feedback flywheel

Sample production traces, label them, dashboards for trends, failures become new eval cases.

Sketch this when: Sketch when asked how the system improves after launch.

  • Production is the only distribution that matters — synthetic evals drift from it.
  • Every triaged failure becomes a permanent regression test.
  • Sampling keeps judge costs sane; you don't label everything.
Deep dive: Evals
Mermaid source
graph TD
  prod[Production traffic] --> traces[(Trace store)]
  traces --> sample[Sample + label judge or human]
  sample --> dash[Quality dashboards]
  sample --> fails[Failure cases]
  fails --> ds[(Eval dataset grows)]
  ds --> dev[Fix prompts, retrieval, tools]
  dev -.->|deploy| prod

Agent run tracing

A span tree per request: LLM calls, tool calls, retrievals — each with latency, tokens, cost.

Sketch this when: Sketch when asked how you'd debug a bad agent answer.

  • Without traces, 'the agent was wrong' is undebuggable folklore.
  • Log retrieval IDs and scores — most 'LLM bugs' are retrieval bugs.
  • Cost per request falls out of the same spans for free.
Mermaid source
graph TD
  req[Request span] --> agent[Agent run]
  agent --> llm1[LLM call latency, tokens, cost]
  agent --> tool1[Tool call]
  agent --> retr[Retrieval top-k ids + scores]
  tool1 --> llm2[Nested LLM call]
  llm1 & tool1 & retr -.->|spans| store[(Trace store)]
  store --> ui[Trace viewer waterfall]

A/B prompt experimentation

Prompts as deployable artifacts: split traffic, compare per-variant metrics, promote the winner.

Sketch this when: Sketch when asked how to ship a prompt change safely.

  • Offline evals gate the experiment; online metrics decide it.
  • Version prompts like code — rollback must be one click.
  • Watch cost and latency alongside quality; wins on one axis often lose another.
Mermaid source
graph LR
  traffic[Traffic] --> split{Split 90 / 10}
  split -->|90| a[Prompt v1]
  split -->|10| b[Prompt v2]
  a & b --> metrics[Per-variant metrics quality, cost, latency]
  metrics --> decide{v2 wins?}
  decide -->|yes| promote(Promote v2)
  decide -->|no| keep(Keep v1)

Guardrails & reliability

Input/output rails, injection defense, validated outputs, fallbacks, rate limits, durable runs.

Guardrails sandwich

Input rails before the model, output rails after — block, retry, or pass.

Sketch this when: Sketch as the frame whenever safety or compliance comes up.

  • Rails are deterministic code where possible — regex and classifiers before judge models.
  • Output rails catch what prompting can't guarantee.
  • Every block needs telemetry; silent refusals destroy trust.
Deep dive: Agent guardrails
Mermaid source
graph LR
  user[User input] --> inrail[Input rails injection, PII, topic]
  inrail -->|blocked| refuse(Refusal)
  inrail -->|pass| llm[LLM]
  llm --> outrail[Output rails schema, groundedness, tone]
  outrail -->|pass| resp(Response)
  outrail -->|fail| retry[Retry / fallback]
  retry -.-> llm

Prompt-injection defense

The privileged model with tools never reads raw untrusted text — only structured extractions.

Sketch this when: Sketch when the agent reads emails, web pages, or any third-party content.

  • Injection isn't a prompt bug, it's a trust-boundary bug — solve it with architecture.
  • The quarantined model can be fooled but can't act.
  • The extraction schema is the choke point: no free text crosses the boundary.
Deep dive: Agent guardrails
Mermaid source
graph TD
  ext[Untrusted content email, web page] --> qtn[Quarantined LLM no tools]
  qtn --> extract[Structured summary fixed schema]
  extract --> priv[Privileged LLM has tools]
  user[User instruction] --> priv
  priv --> act(Tool actions)
  ext -.->|never reaches| priv

Structured output with validate-retry

Schema-constrained generation, a validator, and a repair loop before anything downstream.

Sketch this when: Sketch whenever LLM output feeds code instead of a human.

  • Constrained decoding gets you syntax; validation gets you semantics — you need both.
  • Feed the exact validation error back; models repair well with specifics.
  • Always bound retries and define the give-up path.
Deep dive: Structured outputs
Mermaid source
graph LR
  req[Request] --> llm[LLM schema-constrained]
  llm --> val{Validate parse + schema}
  val -->|ok| obj(Typed object)
  val -->|error| repair[Repair prompt with error message]
  repair -.-> llm
  val -->|N failures| fb(Fallback / flag)

Fallback & retry chain

Backoff retry on the primary, secondary provider next, degraded canned response last.

Sketch this when: Sketch when asked what happens when the model API goes down.

  • Retry only retryable errors; a 400 retried forever is a self-DoS.
  • Cross-provider fallback means abstracting the prompt format ahead of time.
  • A useful degraded answer beats an error page — decide it per feature.
Deep dive: Durable execution
Mermaid source
graph TD
  call[LLM call] --> p1[Primary provider]
  p1 -->|ok| out(Response)
  p1 -->|timeout / 429| retry[Backoff retry]
  retry -.-> p1
  retry -->|still failing| p2[Secondary provider]
  p2 -->|ok| out
  p2 -->|down| deg(Degraded canned response)

Rate limiting & concurrency control

Per-org token buckets plus a global concurrency semaphore in front of the provider.

Sketch this when: Sketch when asked how the system behaves under load or a runaway client.

  • Two different problems: fairness between tenants (buckets) and protecting the upstream (semaphore).
  • Rate limit in tokens, not requests — one request can be 100× another.
  • Queues need depth limits; unbounded queues just move the outage later.
Mermaid source
graph LR
  reqs[Requests] --> bucket{Per-org token bucket}
  bucket -->|over limit| rej(429 + retry-after)
  bucket -->|ok| sem{Global concurrency semaphore}
  sem -->|slot free| prov[LLM provider]
  sem -->|full| q[Queue]
  q -->|slot frees| prov
  q -->|overflow| shed(Shed load)

Durable execution for long agents

Checkpoint after every step; a crash resumes from the last checkpoint instead of restarting.

Sketch this when: Sketch when the agent run outlives a request — minutes-long jobs, approval waits, deploys.

  • Replay requires determinism: side effects live in steps, orchestration stays pure.
  • This is also how human approval waits work — the run just parks at a checkpoint.
  • Idempotency keys on tool calls prevent double-charges on resume.
Deep dive: Durable execution
Mermaid source
graph TD
  s1[Step 1: plan] --> ck1[(Checkpoint)]
  ck1 --> s2[Step 2: tool call]
  s2 --> ck2[(Checkpoint)]
  ck2 --> s3[Step 3: LLM call]
  s3 --> done(Done)
  crash[Crash / redeploy] -.-> resume[Resume from last checkpoint]
  resume -.-> ck2

Serving, streaming & cost

Gateways, token streaming, semantic and prompt caches, model cascades, batch inference.

LLM gateway

Every app goes through one gateway: authn, rate limits, routing, logging, fallback.

Sketch this when: Sketch when asked how multiple teams or apps share LLM access sanely.

  • One place for keys, budgets, and audit — apps never hold provider credentials.
  • Provider swaps and fallbacks happen behind the gateway, invisible to apps.
  • It's also the single point of failure — make it boring and redundant.
Deep dive: Token economics
Mermaid source
graph LR
  a1[App A] & a2[App B] & a3[App C] --> gw[LLM gateway authn, limits, routing, logging, cache]
  gw --> p1[Anthropic]
  gw --> p2[OpenAI]
  gw --> p3[Self-hosted]
  gw -.-> obs[(Usage + cost logs)]

Streaming pipeline

Token deltas flow provider → server → SSE → client; tool-call chunks interleave with text.

Sketch this when: Sketch when asked why the UI shows text appearing word by word — or why it stopped.

  • Streaming fixes perceived latency; time-to-first-token is the metric that matters.
  • Tool calls arrive as partial JSON chunks — buffer until complete before executing.
  • Usage arrives at the end; mid-stream disconnects need server-side accounting.
Mermaid source
sequenceDiagram
  participant Client
  participant Server
  participant Provider
  Client->>Server: POST /chat
  Server->>Provider: request, stream on
  Provider-->>Server: token deltas
  Provider-->>Server: tool_call chunks
  Server-->>Client: SSE events
  Provider-->>Server: done + usage
  Server-->>Client: close stream

Semantic cache

Exact match first, embedding similarity second, LLM only on a true miss.

Sketch this when: Sketch when repeated or near-duplicate queries dominate traffic — support bots, search.

  • The similarity threshold is a precision dial — too loose serves wrong answers confidently.
  • Personalized or time-sensitive answers must skip the cache; segment before caching.
  • TTL plus event-based invalidation when the underlying docs change.
Mermaid source
graph TD
  q[Query] --> exact{Exact match in cache?}
  exact -->|hit| ret(Cached answer)
  exact -->|miss| sim{Similarity above threshold?}
  sim -->|hit| ret
  sim -->|miss| llm[LLM]
  llm --> store[(Store answer with TTL)]
  store --> fresh(Fresh answer)

Prompt caching / prefix reuse

Order the prompt so the stable prefix (system, tools, docs) is cached at the provider.

Sketch this when: Sketch when asked to cut cost or latency on an agent with big system prompts.

  • Cache hits are ~10× cheaper and faster — ordering the prompt is free money.
  • Anything volatile (timestamps, user data) must live after the stable prefix.
  • Agents benefit most: the same huge prefix replays every loop iteration.
Deep dive: Token economics
Mermaid source
graph LR
  subgraph prefix[Stable prefix — cached]
    sys[System prompt] --> tools[Tool schemas]
    tools --> docs[Reference docs]
  end
  subgraph suffix[Varies per request]
    hist[Recent turns] --> q[User query]
  end
  docs --> hist
  q --> llm[LLM prefix read from cache]

Model cascade

A small model answers first; only low-confidence or invalid cases escalate to the frontier model.

Sketch this when: Sketch when asked to cut inference cost without capping quality.

  • Most traffic is easy; pay frontier prices only for the hard tail.
  • The confidence check is the design problem — validators, logprobs, or a self-check.
  • Escalation rate is your quality dial and your cost forecast.
Deep dive: Token economics
Mermaid source
graph TD
  q[Request] --> small[Small fast model]
  small --> conf{Confident and valid?}
  conf -->|yes, most traffic| out(Respond)
  conf -->|no| big[Frontier model]
  big --> out2(Respond)
  conf -.->|log escalation rate| metrics[(Metrics)]

Batch / async inference

Queue jobs, submit as a batch at half price, collect via poll or webhook.

Sketch this when: Sketch when the workload isn't latency-bound — backfills, nightly evals, enrichment.

  • If nobody is waiting on the answer, you're overpaying for sync inference.
  • Design for partial failure: individual items fail, the batch doesn't.
  • Great fit for eval suites — nightly quality runs at half cost.
Mermaid source
graph LR
  jobs[Jobs summaries, tags, evals] --> q[(Queue)]
  q --> batch[Batch API request half price]
  batch --> wait[Poll / webhook]
  wait --> res[(Results store)]
  res --> consume[Downstream consumers]
  fail[Failed items] -.->|requeue| q

Memory & context

Memory tiers, long-term user memory, context budgets, compaction, file-based scratchpads.

Conversation memory tiers

Recent turns verbatim, older turns as a rolling summary, everything recallable by vector search.

Sketch this when: Sketch when asked how a long conversation fits in a finite context window.

  • Recency gets fidelity, history gets compression, everything stays findable.
  • Summaries lose detail silently — the vector tier is the safety net.
  • Tier budgets are tunable per product: support bots weight recall, chat weights recency.
Deep dive: Short-term memory
Mermaid source
graph TD
  turns[Incoming turns] --> recent[Tier 1: last N turns verbatim]
  turns --> summ[Tier 2: rolling summary of older turns]
  turns --> vec[(Tier 3: vector index over full history)]
  q[Current query] --> vec
  vec --> recall[Relevant old snippets]
  recent & summ & recall --> ctx[Assembled context]
  ctx --> llm[LLM]

Long-term user memory

Extract durable facts after each conversation; retrieve them into the system prompt next session.

Sketch this when: Sketch when personalization across sessions comes up.

  • Extract facts, not transcripts — 'prefers TypeScript' beats 500 stored turns.
  • Contradiction handling is the hard part: new facts must supersede stale ones.
  • Memory is user data — deletion and inspection are product requirements.
Deep dive: Agent memory
Mermaid source
graph LR
  conv[Conversation ends] --> extract[LLM extracts durable facts]
  extract --> dedupe{New, duplicate, or contradiction?}
  dedupe -->|new| store[(Memory store)]
  dedupe -->|update| store
  next[Next session] --> retr[Retrieve relevant memories]
  store --> retr
  retr --> sys[Inject into system prompt]

Context-window budget

The assembly diagram: every context slot has a token budget and an eviction order.

Sketch this when: Sketch when asked what happens as the prompt grows past the window.

  • Context is a zero-sum budget; every slot fights every other slot.
  • Decide eviction order up front — accidental truncation drops the wrong thing.
  • More context isn't free even when it fits: attention degrades and cost scales.
Mermaid source
graph LR
  subgraph window[Context window]
    sys[System ~5%] --> tools[Tools ~10%]
    tools --> mem[Memory ~10%]
    mem --> docs[Retrieved docs ~30%]
    docs --> hist[History ~40%]
    hist --> q[Query ~5%]
  end
  evict[Eviction order: history first, system never] -.-> hist
  q --> llm[LLM]

Context compaction loop

Near the limit, summarize the transcript into goals + state and continue with a fresh context.

Sketch this when: Sketch when asked how an agent works for hours without running out of context.

  • The summary must preserve intent and open threads, not just topics.
  • Compaction loses detail — pair it with a file scratchpad for anything precise.
  • This is how long-running coding agents actually survive.
Mermaid source
graph TD
  run[Agent running] --> check{Context near limit?}
  check -->|no| cont[Continue]
  cont --> run
  check -->|yes| compact[Summarize transcript keep goals, state, files]
  compact --> fresh[Fresh context summary + recent turns]
  fresh --> run

File-based scratchpad memory

The agent writes notes, todos, and artifacts to a workspace and re-reads them across steps.

Sketch this when: Sketch when asked where an agent keeps precise state the context window can't hold.

  • Files survive compaction, crashes, and handoffs — context doesn't.
  • The todo file doubles as the plan; the agent literally checks items off.
  • This is the blackboard pattern applied to a single agent.
Deep dive: Agent memory
Mermaid source
graph TD
  agent[Agent] -->|writes| notes[notes.md findings so far]
  agent -->|writes| todo[todo.md remaining work]
  agent -->|writes| art[Artifacts code, data]
  notes & todo & art --> ws[(Workspace files)]
  ws -->|re-reads next step| agent
  compact[Context compaction] -.->|survives it| ws