// Stage 8 · RAG, Agents & AI Engineering
AI Engineering: Evals, Cost & Shipping
Your own eval sets, LLM judges, observability, model gateways, cost control and product design.
about 8 hours3 algorithms1 paperslesson text in progress
01Goals
- Build eval sets and regression tests for real use cases
- Estimate and optimise token cost
- Design human-in-the-loop and fallback flows
Do these first: Agents: Tool Use, MCP, Planning & Memory
02Algorithms to master
Each card has the original paper, prerequisites and resources.
03Must-read papers
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena ↗2023 ✓ verified
04Courses & videos
- huggingface.co ↗BooksEN
- inspect.aisi.org.uk ↗Official docsEN
- OpenAI Cookbook ↗Official docsEN
- Microsoft：Generative AI for Beginners ↗CoursesMULTI
05Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Write 50 eval cases for your own RAG/agent, wire them into CI and score automatically whenever the prompt or model changes.
- 1
- 2
- 3github.com ↗Reference code
Write regression tests for your app that run on every prompt or model change.
06Self-test
Answer in your head, then open to check.
Common biases of LLM judges?
Position bias, length bias and self-preference.
What does prompt caching save?
Compute and cost for repeated prefixes.
The most important test before launch?
Regression evals on your real traffic distribution.
On these routes: Application Developers · Product Managers & Leaders