The take-home’s actual architecture, drawn the way you’d sketch it in the interview: a voice AI diagnostic agent — Twilio to Pipecat to a LlamaIndex FunctionAgent over Postgres, with a GPT-4o vision tier. Every card names the real files, the design decision to say out loud, and the Mermaid source to paste into the whiteboard.
System context
One FastAPI app between Twilio and the provider fleet; Postgres on Neon underneath.
Sketch this when: Open with this — it frames every later question and takes 30 seconds.
- Single deployable: one FastAPI app owns webhook, media stream, agent, uploads, and replay UI.
- Every provider is an env-swappable factory — DeepSeek and OpenAI trade places with one variable.
- No queues, no separate workers: background work rides FastAPI background tasks, sized to the assignment.
Key files: app/main.py, README.md
Mermaid source
graph LR
caller((Caller on PSTN)) --> twilio[Twilio Programmable Voice]
twilio -->|webhook + media stream| api[FastAPI app :8000]
browser((Caller browser)) -->|photo upload page| api
api --> llm[OpenAI / DeepSeek LLM]
api --> stt[Deepgram streaming STT]
api --> tts[Cartesia TTS]
api --> vision[GPT-4o Vision]
api --> db[(Postgres on Neon)]
Phone call setup
Webhook → signature check → TwiML Connect/Stream → media WebSocket → bot.
Sketch this when: Sketch when asked how a phone call actually reaches the agent.
- Signature validation distinguishes misconfig (500) from forgery (403) — different alerts.
- The greeting is a constant, spoken without an LLM round-trip — first words are instant.
- app/phone/ owns HTTP + TwiML; app/voice/ owns the media pipeline — the old phone STT path is retired.
Key files: app/phone/webhook.py, app/phone/twiml.py, app/voice/routes.py
Mermaid source
sequenceDiagram
participant Twilio
participant Webhook as POST /twilio/voice
participant WS as WS /ws/twilio
participant Bot as run_bot
Twilio->>Webhook: incoming call webhook
Webhook->>Webhook: validate X-Twilio-Signature
Webhook-->>Twilio: TwiML Connect + Stream
Twilio->>WS: open media stream
WS->>WS: read connected + start events (streamSid, callSid)
WS->>Bot: run_bot(ws, streamSid, callSid)
Bot-->>Twilio: greeting audio frames
Pipecat voice pipeline
The frame pipeline a caller's audio traverses, in processor order.
Sketch this when: Sketch when asked what happens between the caller speaking and hearing an answer.
- Each box is a Pipecat frame processor — swapping a provider swaps one factory, not the pipeline.
- The Filler masks LLM time-to-first-token with a cached spoken line; Sanitizer keeps TTS from reading markup.
- Barge-in: VAD interrupt cancels downstream frames so the caller can talk over the bot.
Key files: app/voice/bot.py, app/voice/processors.py
Mermaid source
graph LR
in[transport.input] --> vad[Silero VAD]
vad --> stt[Deepgram STT]
stt --> gate[SafetyGate]
gate --> refresh[SystemPromptRefresh]
refresh --> uagg[User aggregator]
uagg --> llm[LLM gpt-4.1-mini tool loop]
llm --> filler[Filler]
filler --> clean[Sanitizer]
clean --> tts[Cartesia TTS]
tts --> out[transport.output]
out --> aagg[Assistant aggregator]
Web/text agent turn
The hermetic test surface: same agent, same tools, WebSocket text in, audio frames out.
Sketch this when: Sketch when explaining how the system is tested without dialing a phone.
- The agent is rebuilt every turn so the case-file-derived system prompt is always current.
- Sentences stream to TTS as they complete (lookahead 2) — first audio doesn't wait for the full reply.
- Both transports satisfy one SessionBridge protocol; evals drive this path hermetically.
Key files: app/ws/routes.py, app/agent/core.py, app/agent/tts_pipeline.py
Mermaid source
sequenceDiagram
participant Client
participant WS as WS /ws/call
participant Safety
participant Agent as FunctionAgent
participant Tools
participant TTS as SpeechPipeline
Client->>WS: user utterance
WS->>Safety: detect_safety_trigger on raw text
Safety-->>WS: clear, or scripted safety response
WS->>Agent: run_turn(session)
Agent->>Tools: tool calls
Tools-->>Agent: results mutate CaseFile
Agent-->>TTS: SentenceReady events
TTS-->>Client: AudioFrame pcm24k + TranscriptFrame + StateFrame
Tool registry + case-file memory
Auto-discovered tools all write into one structured CaseFile that re-enters the prompt every turn.
Sketch this when: Sketch when asked how the agent remembers what the caller already said.
- Never-re-ask is structural, not prompt-hope: the CaseFile is pydantic, persisted, and re-injected.
- Adding a tool = adding a file — the registry walks app/tools/ for TOOLS lists.
- The same tool functions serve both transports; the phone side re-exposes them as Pipecat FunctionSchemas unmodified.
Key files: app/tools/registry.py, app/contracts.py, app/agent/prompts.py, app/voice/tools.py
Mermaid source
graph TD
reg[Registry auto-discovers TOOLS lists in app tools modules] --> core[core_tools: identify appliance, symptoms, steps]
reg --> sched[scheduling_tools: find technicians, book]
reg --> vis[visual_tools: upload link, poll analysis]
reg --> lib[library_tools: flag-gated RAG search]
core & sched & vis --> cf[(CaseFile contextvar + sessions.case_file jsonb)]
cf --> prompt[System prompt rebuilt each turn]
prompt --> agent[Agent never re-asks answered questions]
Scheduling + booking atomicity
A conditional UPDATE makes double-booking impossible by construction.
Sketch this when: Sketch when asked how you prevent two callers booking the same slot — the headline answer.
- The claim is one conditional UPDATE: zero rows back means someone else won the race — no locks, no advisory tricks.
- Slots are pre-generated rows over a two-week horizon, not recurrence rules — matching is a plain indexed join.
- Confirmation requires read-back plus an explicit yes, enforced in the scheduling contract prompt.
Key files: app/tools/scheduling_tools.py, app/db/matching.py, app/db/models_scheduling.py
Mermaid source
sequenceDiagram
participant Agent
participant Find as find_technicians
participant Book as book_appointment
participant DB as Postgres
Agent->>Find: zip + appliance specialty
Find->>DB: join service_areas, specialties, open slots
DB-->>Find: candidate slots
Find-->>Agent: offered slots as stable refs
Agent->>Agent: read back slot, wait for explicit yes
Agent->>Book: chosen slot ref
Book->>DB: UPDATE slot to booked WHERE status open RETURNING
Book->>DB: INSERT appointment in same transaction
Book-->>Agent: confirmed, or slot_taken re-offer
Safety interrupt
Deterministic regex on the raw utterance, before the LLM ever sees it — in both channels.
Sketch this when: Sketch when asked how the agent handles a caller reporting a gas smell.
- Safety is a structural pre-filter, not prompt engineering — the model can't be argued around because it never gets the choice.
- The same gate runs in the web bridge and as a Pipecat processor on the phone path.
- Knowledge-base steps prefixed safety_ also raise the flag — belt and suspenders.
Key files: app/agent/safety.py, app/voice/processors.py
Mermaid source
graph TD
utt[Raw utterance] --> regex{Deterministic regex gate before the LLM}
regex -->|gas smell| halt[Scripted safety response]
regex -->|smoke or sparking| halt
regex -->|burning smell| halt
regex -->|water near electrics| halt
regex -->|clear| agent[Agent proceeds normally]
halt --> flag[safety_flag set on CaseFile]
flag --> tech[Steer to technician, no DIY steps]Tier-3 visual diagnosis
Email a tokenized upload link mid-call; vision runs in the background and merges into the case file.
Sketch this when: Sketch when explaining how a photo sharpens the diagnosis without blocking the call.
- Vision output is schema-constrained JSON and merged as advisory evidence — it refines, never overrides, the caller's account.
- The token is the auth: single-use row, 24-hour expiry, no login flow.
- If the call already ended, findings go back by email instead of the poll.
Key files: app/tools/visual_tools.py, app/vision/pipeline.py, app/vision/client.py, app/uploads/routes.py
Mermaid source
sequenceDiagram
participant Agent
participant Email
participant Caller
participant API as POST /api/upload
participant Vision as run_vision_pipeline
Agent->>Email: send_image_upload_link(email)
Email-->>Caller: tokenized link, 24h expiry
Caller->>API: photo via upload page
API->>Vision: FastAPI background task
Vision->>Vision: gpt-4o with JSON-schema output
Vision-->>Agent: findings merged into CaseFile as advisory evidence
Agent->>Agent: check_image_analysis poll
Data model
Nine tables; sessions carry the case file, slots carry the booking invariant.
Sketch this when: Sketch when asked what's in the database and where the booking invariant lives.
- The conversation state is one jsonb column — no separate conversation store to drift.
- Unique slot_id on appointments plus the status check on slots encode the invariant in the schema.
- Seed is idempotent (Chicago/Dallas zips) and runs at container boot after migrations.
Key files: app/db/models_core.py, app/db/models_scheduling.py, app/db/models_visual.py, app/db/seed.py
Mermaid source
graph LR
customers[customers] --> sessions[sessions: case_file jsonb, transcript]
sessions --> uploads[image_uploads: token, vision_analysis]
sessions --> appts[appointments: unique slot_id]
tech[technicians] --> junction[technician_specialties]
junction --> specialties[specialties]
tech --> areas[service_areas: zip indexed]
tech --> slots[availability_slots: pre-generated, unique tech + start]
slots -->|conditional UPDATE open to booked| appts
Deployment topology
One Dockerfile serves local Compose and a Cloudflare Durable Object container against Neon.
Sketch this when: Sketch when asked how it's deployed and what happens when nobody calls for an hour.
- Same image everywhere — the only deploy-time difference is env vars passed into the container.
- The cron ping trades a few requests for zero cold-start on a live demo number.
- Hosted disk is ephemeral by choice; state that matters lives in Neon.
Key files: Dockerfile, docker-compose.yml, cloudflare/app-worker.ts, docker-entrypoint.sh
Mermaid source
graph TD
img[One Dockerfile] --> compose[Local docker compose, optional ngrok phone profile]
img --> do[Cloudflare Durable Object container]
worker[Cloudflare Worker] --> do
cron[Cron keep-warm ping to healthz] -.-> worker
do --> boot[Entrypoint: migrations + idempotent seed]
boot --> app[FastAPI app]
app --> neon[(Neon Postgres)]
do -->|sleepAfter 30m| sleep[Scale to zero]