1. Andrej Karpathy — Zero to Hero
Build intuition for neural networks, backpropagation, language models and GPT by implementing the ideas.
The learning path
The objective is not just to use AI APIs. It is to understand the models and the engineering systems around them.
| Stage | Weeks | Focus |
|---|---|---|
| 1. Foundations | 1–2 | Math, neural networks and PyTorch |
| 2. Language models | 3–4 | Tokenization, embeddings and next-token prediction |
| 3. Transformers | 5–6 | Attention and transformer internals |
| 4. Pretraining | 7 | Data, optimization and training systems |
| 5. Post-training | 8–9 | SFT, LoRA, RLHF, DPO and GRPO |
| 6. Inference | 10–11 | KV cache, quantization and performance |
| 7. LLM applications | 12–14 | Prompting, embeddings, RAG and context |
| 8. Evaluation | 15–16 | Benchmarks, reliability and safety |
| 9. Advanced topics | 17–18 | Reasoning, agents and research |
| 10. Projects | 19–20 | Build and consolidate your knowledge |
How to use this roadmap: follow the stages in order, but spend more time on topics that are unclear. Move forward when you can explain the concept and demonstrate it with code—not simply when a video ends.
Five core resources
Use these as the backbone of the curriculum rather than jumping between dozens of disconnected tutorials.
2. Sebastian Raschka — Build an LLM from Scratch
A structured implementation path covering tokenization, attention, transformer blocks, pretraining and fine-tuning.
3. Hugging Face — LLM Course
Learn to use pretrained models, tokenizers, datasets, fine-tuning and the open-source model ecosystem.
4. Stanford CS336 — Language Modeling from Scratch
A rigorous, implementation-heavy course on building language models, training, GPUs, scaling and systems.
5. Nathan Lambert — RLHF Book and Course
Understand supervised fine-tuning, reward models, preference optimization and reinforcement learning for language models.
The 20-week curriculum
Each phase includes topics, recommended resources and a practical exercise.
01
Mathematics and machine learning foundations
Weeks 1–2Learn enough of the mathematics and machine learning machinery to understand what happens inside a neural network. You do not need to become a mathematician.
Topics
- Vectors, matrices and dot products
- Matrix multiplication
- Derivatives and gradients
- Chain rule
- Probability distributions
- Neurons and network layers
- Activation and loss functions
- Gradient descent
- Backpropagation
- PyTorch tensors and autograd
Resources
Practical exercise: Build a tiny neural network and implement forward propagation, loss calculation and backpropagation. Try to understand how each parameter changes during learning.
02
How language models work
Weeks 3–4Understand what a language model predicts, how text becomes tokens and how a model generates text one token at a time.
Topics
- Natural language processing
- Statistical language models
- N-gram models
- Tokenization and token IDs
- Vocabulary and BPE
- Embeddings
- Next-token prediction
- Cross-entropy and perplexity
- Autoregressive generation
- Context windows
Resources
Practical exercise: Build a character-level language model and train it on a small text dataset. Experiment with generation and observe how its predictions change.
03
Transformer architecture and internals
Weeks 5–6This is one of the most important phases. Aim to explain each major component and how data flows through a transformer.
Topics
- Token embeddings
- Positional encoding
- Self-attention
- Query, Key and Value
- Multi-head attention
- Causal attention masks
- Feed-forward networks
- Residual connections
- Layer normalization
- Transformer blocks
- Softmax and logits
- Encoder-only, decoder-only and encoder-decoder models
- RoPE, GQA and MQA
- Mixture of Experts
- Sliding-window attention
Resources
The Illustrated Transformer ↗The Transformer Family ↗Karpathy: Let's Build GPT ↗Raschka video course ↗
Practical exercise: Implement a tiny GPT-style transformer in PyTorch. Write the attention mechanism and transformer block yourself before relying on prebuilt model libraries.
04
Training LLMs from scratch
Week 7Learn what it takes to create a foundation model, from collecting training data to optimizing a large distributed training run.
Topics
- Pretraining datasets
- Data collection and filtering
- Deduplication and contamination
- Data quality and synthetic data
- Parameters and training tokens
- Batch size and gradient accumulation
- Adam and AdamW
- Learning-rate schedules
- Mixed-precision training
- Training and validation loss
- Scaling laws
- Distributed training
- Tensor and pipeline parallelism
Resources
Practical exercise: Train a tiny language model on your own text dataset. Monitor training and validation loss, then experiment with learning rates and dataset sizes. Expensive hardware is not required for this experiment.
05
Fine-tuning and post-training
Weeks 8–9Understand how a pretrained model is adapted to follow instructions, reflect preferences and improve at tasks such as reasoning.
| Technique | What to understand |
|---|---|
| SFT | Supervised fine-tuning on examples |
| Instruction tuning | Teaching a model to follow instructions |
| PEFT / LoRA | Efficiently adapting a small number of parameters |
| QLoRA | Memory-efficient fine-tuning with quantization |
| RLHF | Using human preference feedback |
| Reward models | Learning to score preferred outputs |
| PPO | Policy optimization |
| DPO | Preference optimization without a traditional reward-model RL loop |
| RLAIF | AI-generated feedback |
| GRPO and RLVR | Group-relative and verifiable-reward training |
| Distillation | Transferring capabilities to smaller models |
Resources
Practical exercise: Fine-tune a small open-weight model with LoRA using a small instruction dataset. Compare the original and adapted model on the same questions.
06
LLM inference and optimization
Weeks 10–11Learn how trained models are executed efficiently. This is especially useful for understanding local LLMs, hardware requirements and serving systems.
Topics
- Inference flow
- Prefill versus decode
- Autoregressive token generation
- KV cache
- Temperature, top-k and top-p
- Context memory requirements
- FP16, BF16, INT8 and INT4
- FlashAttention
- PagedAttention
- Continuous batching
- Speculative decoding
- GPU memory bandwidth
- Time to first token
- Tokens per second
- Inference servers
Resources
Lilian Weng: Inference Optimization ↗Stanford CS336 ↗LLM Inference Explained ↗Inference optimization video ↗
Practical exercise: Run an open-weight model locally. Measure time to first token, output tokens per second and memory use. Compare different context lengths and quantization levels.
07
Building applications with LLMs
Weeks 12–14Learn how models are integrated into useful applications and how external knowledge, tools and memory extend what a model can do.
Prompting and model interfaces
- System and user prompts
- Few-shot prompting
- Structured outputs and JSON schema
- Function and tool calling
- Prompt caching
Embeddings and retrieval
- Semantic embeddings
- Vector databases
- Similarity search and cosine similarity
- Chunking strategies
- Hybrid search and reranking
RAG
- Basic and advanced RAG
- Query rewriting
- Metadata filtering
- Context management
- Multimodal RAG
Memory and context
- Short-term and long-term memory
- Conversation history
- Context engineering
- Summarization and compression
Resources
Full Stack LLM Bootcamp ↗Chip Huyen: AI Engineering resources ↗Hugging Face RAG cookbook ↗Advanced RAG course ↗
Note: some bootcamp recordings are older. Use them for fundamentals and verify current library APIs against official documentation.
Practical exercise: Build a question-answering system over your own documents. Retrieve relevant passages, generate answers grounded in those passages and display the sources used.
08
Evaluation, reliability and safety
Weeks 15–16Learn how to determine whether a model or application is actually reliable. Fluent answers are not necessarily correct answers.
Topics
- Benchmarking
- Accuracy and task-specific metrics
- Perplexity
- Human evaluation
- LLM-as-a-judge
- Hallucination evaluation
- Groundedness
- Dataset contamination
- Robustness
- Prompt injection and jailbreaks
- Bias and safety
- Evaluation datasets
- Regression testing
Resources
OpenAI: Evals learning resources ↗Hugging Face: RAG evaluation ↗Anthropic: Writing tools for agents ↗
Practical exercise: Create an evaluation set of 50 questions. Compare models and prompts using accuracy, hallucination rate, response time and cost.
09
Reasoning models, agents and advanced topics
Weeks 17–18Reasoning models
- Chain of Thought
- Test-time compute
- Inference-time scaling
- Reasoning traces
- Verifiable rewards
- Process and outcome reward models
- GRPO and reasoning distillation
AI agents
- ReAct
- Tool calling
- Planning and task decomposition
- Reflection
- Agent loops
- Memory and context
- Multi-agent systems
- Orchestration and MCP
- Agent evaluation and observability
Further research areas
Explore mixture-of-experts architectures, multimodal models (vision, audio and video), long-context models, mechanistic interpretability, sparse attention, model merging, synthetic data generation, compression and distillation.
10
Practical projects and consolidation
Weeks 19–20Finish with working implementations rather than only notes and videos.
- Build a neural network from scratch: implement a tiny network and backpropagation.
- Build a GPT-style model: tokenizer, embeddings, transformer blocks, training and generation.
- Fine-tune an open-weight LLM: prepare a small dataset and adapt a model using LoRA.
- Build a RAG application: connect documents, embeddings, vector search, reranking and generation.
- Build an LLM agent harness: implement a model interface, tool execution, context management, memory, evaluation, agent loop and observability.
Suggested order: do Projects 1 and 2 before moving into fine-tuning. Projects 4 and 5 are application-engineering projects and can continue growing after the 20-week curriculum.
Research papers to read
Read these after learning the relevant concepts. Focus on the problem, proposed solution, experiments and limitations.
| Paper | Key contribution / topic | Read |
|---|---|---|
| Attention Is All You Need | Transformer architecture | arXiv ↗ |
| BERT | Bidirectional language-model pretraining | arXiv ↗ |
| GPT-3 | Scaling and few-shot learning | arXiv ↗ |
| Chinchilla | Compute-optimal training | arXiv ↗ |
| InstructGPT | Instruction tuning and RLHF | arXiv ↗ |
| LoRA | Parameter-efficient fine-tuning | arXiv ↗ |
| DPO | Direct preference optimization | arXiv ↗ |
| DeepSeek-R1 | Reasoning-focused reinforcement learning | arXiv ↗ |
A repeatable weekly study routine
A balanced approach to learning, reading, implementation and recall.
Watch2 hours
Watch lectures and understand the high-level concept.
Read1–2 hours
Read technical blogs, papers or documentation.
Implement2–3 hours
Write code to reproduce the concept. Prioritize this time.
Recall and review1 hour
Explain the topic without notes, find gaps and create flashcards.
One important rule: don't move to the next topic just because you finished a video. Move on when you can explain the concept in your own words and demonstrate it with code.
What you should be able to do
Use these outcomes to judge whether the learning is becoming practical knowledge.
- Explain how a transformer processes tokens and generates text.
- Implement a small GPT-style model and train it.
- Explain the difference between pretraining, SFT and preference optimization.
- Fine-tune an open-weight model and evaluate the result.
- Explain prefill, decode, KV cache, quantization and inference speed.
- Build a RAG system and measure its retrieval and answer quality.
- Build an agent loop that calls tools and handles failures safely.
- Create an evaluation dataset and compare model behavior systematically.
Master learning checklist
Track the concepts you have genuinely learned. Your progress is saved in this browser using local storage.
LLM knowledge tracker
0 of 36 concepts completed