Stop studying. Start shipping.
18 projects, three per stage, that turn the lessons into real systems on real infrastructure. Finish one per stage and you have a six-project portfolio. Bring your own model key from OpenAI, Anthropic, Google, Groq or xAI.
Every capstone is done when it's
- Deployed somewhere real
- Measured: evals, p95 latency and cost
- Observable when something breaks
- Safe with secrets and capped on spend
- Documented so a stranger can run it
- Shown in a two to four minute demo
Stage 1: Foundations
Can you wire a model into real software?
The Provider-Agnostic LLM Gateway
Your startup is locked into one model vendor and one outage took down the whole product. Build the internal gateway every team routes through.
You ship: A real service fronting all five providers behind one interface: streaming, schema-constrained structured output, automatic fallback on outage / rate-limit, retries with backoff + jitter, and per-request token + cost logging with a health/cost dashboard.
A Real Tool-Using Agent (terminal ops copilot)
Build a CLI agent that can actually do things, not just chat.
You ship: A ReAct agent using native function-calling (made to work across OpenAI / Anthropic / Gemini tool APIs) with 4+ real tools, web search, a sandboxed code/calc executor, filesystem read/write, one real API, a full Thought → Action → Observation trace, step + budget caps, and a confirmation guard before any destructive tool.
Semantic Search Over Your Own Corpus
Make your own notes / docs / bookmarks searchable by meaning, and prove it works.
You ship: Ingest a real personal corpus (Obsidian/Notion export, PDFs), real embeddings (OpenAI or Gemini; or self-host e5/bge, Groq/Grok don't embed), a vector store, a search UI, and a hand-labeled eval set (≥20 queries with expected hits) measuring recall@k.
Stage 2: Data & Retrieval
Can you make retrieval work on messy data?
Production RAG Over Ugly Real Documents
A law / insurance / clinical firm hands you 500 genuinely ugly PDFs, scans, multi-column, tables, and demands grounded answers with citations and zero hallucination tolerance.
You ship: Real ingestion of the actual mess (OCR + tables), a real vector DB (pgvector / Qdrant / Weaviate), hybrid search + a reranker, query rewriting, grounded answers with inline citations, a refusal path when evidence is weak, and a RAGAS eval on a labeled set.
The Agent That Remembers You
A companion / assistant that holds memory across weeks of conversations and many users.
You ship: A real memory architecture, short-term window management, long-term vector memory + an episodic/summary store in a real DB, per-turn retrieval, an explicit write/update/forget policy, persistence that is auth-scoped per user (no cross-user leakage).
Knowledge-Graph RAG Over a Domain
Q&A over a connected domain where relationships matter, a codebase, a company wiki, a research field, and plain RAG fumbles multi-hop questions.
You ship: Extract entities/relations into a real graph (Neo4j / Postgres) alongside a vector index, do hybrid graph + vector retrieval (GraphRAG / HippoRAG style), answer multi-hop questions, and eval multi-hop accuracy vs plain RAG.
Stage 3: Architecture & Agentic
Can you ship a multi-agent system on real infrastructure?
A Multi-Agent System on a Real Framework
Automate a real knowledge-work task end-to-end: a competitive-research report, a codebase migration, or a due-diligence memo.
You ship: Built on one production framework gone deep, LangGraph, OpenAI Agents SDK, or the Claude Agent SDK, a supervisor routing to specialists (researcher / analyst / writer / critic), real tools, shared state, a human-in-the-loop checkpoint that truly pauses, and a full trace. Deployed.
Ship a Real MCP Server
Expose one of your real systems, your Postgres, your company API, your files, to any model through the Model Context Protocol.
You ship: A real MCP server (TS or Python SDK) exposing tools + resources over correct JSON-RPC, with auth, used live from a real MCP client (Claude Desktop or an SDK client), plus a written security review of the surface you exposed.
Parallel Agentic Pipeline at Volume
Process 10k–100k real items (documents, tickets, products) with agents, on a budget and a deadline.
You ship: A fan-out / fan-in pipeline with real concurrency, per-item model-tier routing (Groq for the easy 80%, escalate hard items to a frontier model), batching, idempotent retries, a real queue (SQS / Redis / Cloud Tasks), and a cost + throughput dashboard.
Stage 4: Production & Reliability
Can you build an AI system you'd put on call?
Evals-as-a-Discipline: a CI Gate for an LLM Feature
Your team ships prompt and model changes straight to prod with no safety net. Build the gate that stops a regression.
You ship: A real eval suite, golden dataset, LLM-as-judge with bias controls (calibrated against human labels), task metrics, running offline + a canary design, wired into CI (GitHub Actions) so a PR that regresses quality fails the build, with the rigor to not block on noise, plus a regression diff report.
Observability + Guardrails for an Agent in Prod
An agent is live and you are on-call for it.
You ship: Real tracing of every span / token / cost (OTel GenAI, Langfuse, or LangSmith), a guardrail layer (prompt-injection + jailbreak detection, PII redaction, output safety → allow / redact / block), SLO dashboards + alerts, and a runbook, then a red-team / chaos session with documented incidents.
Resilience Engineering: Survive Chaos
Providers go down, rate-limit you, and return malformed JSON. Keep serving.
You ship: An agent service with retries (backoff + jitter), circuit breakers, timeouts/budgets, fallback chains (frontier → Groq → cached → degraded), HITL escalation, and idempotency keys, then a chaos harness injecting outages, latency, and bad JSON, and a report of auto-resolve rate vs escalation.
Stage 5: Scalability & MLOps
Can you serve, tune and post-train models, and know when to?
Deploy & Autoscale an Agent Service
Launch day: traffic goes 0 → 500 concurrent sessions on a bursty curve.
You ship: A containerized service on real infra (Fly.io / Cloud Run / Railway / Modal), autoscaling, a real load test (k6 / Locust) to 500 concurrent, a p95 SLO, graceful degradation under overload (load-shedding / queueing), a cost-per-session number, and a CI→deploy pipeline with rollback.
Post-Train a Small Model and Prove It
A narrow, high-volume task where a fine-tuned small model might beat prompting a frontier model on cost and quality.
You ship: Actually post-train an open model, LoRA/QLoRA (PEFT/TRL) and/or DPO on a real preference set, on real GPU (Modal / Runpod / Colab), then build an eval that fairly compares fine-tuned-small vs prompt+RAG-on-a-frontier-model across quality, latency, and $/1k, and write the verdict (including "tuning didn't win, here's why"). Serve via Groq or self-host.
Self-Host & Optimize Inference
At your volume, hosted-API cost is killing margins. Can self-hosting win, and do you understand why?
You ship: Serve an open model with vLLM / TGI on real GPU, apply quantization (int8/int4, AWQ/GPTQ) and continuous batching, measure throughput / latency / $-per-1M-tokens vs a hosted baseline (Groq is the perfect fast-hosted comparison), and write a roofline / KV-cache analysis of the bottleneck.
Stage 6: Advanced & Leadership
Can you architect, explain and defend a system?
Ship a Multimodal Product Feature
A real feature with a real owner: receipts → expense entries, a voice agent, video-moment search, or screenshot → UI code.
You ship: A real multimodal pipeline using a vision/audio provider (OpenAI, Anthropic, or Gemini for vision; Whisper/Gemini for audio), real input handling, an explicit token/cost budget (image patches, audio seconds, video frames), an eval on real samples, and a deployed demo.
Architect & Defend a Real System Design
The actual staff/architect interview, made real: "Design an AI assistant over 10 years of our wiki + support tickets + code."
You ship: A full design doc (requirements, architecture, data model, retrieval strategy, eval plan, safety/threat model, a cost model at 10x scale, rollout plan, an ADR per major decision), PLUS a working reference-implementation skeleton (the spine runs on real infra at small scale), PLUS a recorded design-review defense.
Online Experimentation / Adaptive System
Two prompt/model/agent variants are live, learn which wins on a real metric, and/or build a system that improves from feedback.
You ship: A real experimentation harness: an A/B test or a contextual bandit in production on a real KPI (with peeking-safe statistics), or a feedback-collection + preference loop that measurably improves the system over time, with guardrails against metric gaming and a writeup of what you learned.
One per stage is a portfolio.
It walks a hiring manager up the ladder: wire a model in, make retrieval real, orchestrate agents, make it reliable, make it scale, then architect and defend it.