ML compilers, from zero

A course where you build a machine learning compiler yourself, one week at a time. You start by interpreting add 2 3 and, months later, generate code fast enough to train a language model.

Start with lesson 0.1 Skip to week 1

Course outline

Week 0 covers what the original course assumes you already know. Skip any lesson whose quiz you can already ace. Lessons 0.6 and 0.7 are one per track: read the one for your language.

Week 0 · Prerequisites

  1. 0.1C and pointersAddresses, arrays as base plus offset, int32 overflow, C division
  2. 0.2Memory and cachesCache lines, and why row order beats column order
  3. 0.3Compilers and IRsSource to tree to DAG to C, and a first rewrite
  4. 0.4Tensors and shapesOne buffer, many views: reshape, permute, broadcast, pad
  5. 0.5Matrix multiplyDot products, the triple loop, and GEMM from reshape and reduce
  6. 0.6OCaml crash courseInstall, dune, variants, pattern matching, the 63-bit int trap
  7. 0.7Python for this courseA package the grader can run, i32 wrapping, floor division, f32 rounding, match

Week 1 · UOps and rewrites

  1. 1.1The UOpUOp(op, src, arg), DAGs, and hash consing
  2. 1.2.uir and interpretingThe format, dtypes, evaluation order, exit codes
  3. 1.3Rewrites and foldingPattern rules, bottom-up to a fixpoint, constant folding
  4. 1.4Symbolic simplificationIdentities that hold for i32 but not f32, and bounds
  5. 1.5CALL, PARAM, STACK, INDEXFunctions as graphs, substitution, closed bodies
  6. 1.6Lab: your compilerThe contract, the test runner, your own tests, a build order

Week 2 · Memory and rendering C

  1. 2.1Buffers and viewsGlobal buffers, local scratch, and shrink as a window onto memory
  2. 2.2Load, store, afterVersions of memory, and what "fully ordered" means
  3. 2.3Proving boundsUsing week 1's intervals to prove every access is in range
  4. 2.4Rendering CScheduling, C without undefined behavior, exact float literals
  5. 2.5The runtimeCompile the C, run it, read the results back
  6. 2.6Lab: memory and CThe week 2 contract, the render grader, a build order

Week 3 · Loops

  1. 3.1Ranges and endLoop counters as UOps, open ranges, and nesting
  2. 3.2Memory in loopsAccumulators, where a load runs, and versions inside a loop
  3. 3.3A matmul by handThree ranges, one accumulator, and index math you can prove
  4. 3.4A convolution by handSliding windows, padding with masks, and 2D convs on flat buffers
  5. 3.5Rendering loopsPlacing every value at the right loop depth, and why that needs rewrites
  6. 3.6Lab: loopsThe week 3 contract, the suite, a build order

Week 4 · Rangeify

  1. 4.1Vectors and elementwise opsALU on whole rank-1 values, and why there's no implicit broadcast
  2. 4.2Reduce and expandFolding an axis away in a fixed order, and broadcasting a scalar back
  3. 4.3RangeifyRemoving shapes: every element gets an index from a range
  4. 4.4Fuse or materializeRecompute a value at each use, or store it once with stage
  5. 4.5Rendering vectorsLoops instead of unrolling, and the 20 KB limit
  6. 4.6Lab: rangeifyThe week 4 contract, the suite, a build order

Week 5 · Movement ops

  1. 5.1Any rank, flat memoryValues of any rank over buffers that stay one-dimensional
  2. 5.2Reshape and permuteSame elements with a new shape or a new axis order
  3. 5.3Pad, shrink, expand, stackGrowing, cutting, broadcasting and joining tensors
  4. 5.4Matmul and conv from movementThe syllabus's GEMM, and a convolution, as movement plus one reduce
  5. 5.5Rangeify movement opsEvery movement op becomes index arithmetic: div, mod, clamps and masks
  6. 5.6Lab: movement opsThe week 5 contract, the suite, a build order
Stuck or curious?

The resources have a study guide and quiz review for every week, a guide to each failing test, a debugging walkthrough, and a glossary.

How the course works

Each lesson explains one idea with diagrams you can step through, then asks you to use it. Quizzes check that the idea landed. Exercises in your language, C, or .uir run on your machine against hidden tests, with hints if you get stuck.

Pick your language

You write your compiler in Python or in OCaml. Python is the default. Choose under Language at the bottom of the sidebar. Exercises and the few instructions that differ, such as build commands, follow your choice. Lesson 0.7 sets up a Python compiler, and lesson 0.6 teaches OCaml from scratch. Pick OCaml if you want its pattern matching and type checker, and Python if you'd rather spend your time on the compiler than on a new language.

At the end of each week you extend your own compiler, in your own repository, until the week's test suite passes. Week 1's suite has 142 tests, and it catches each of 54 deliberately planted bugs, so a green run means something.

Running exercises

From the course folder, run python tools/serve.py and open http://127.0.0.1:4111. The indicator in the top bar shows whether the runner is on. Lessons still read fine without it.

Your compiler is yours

This course was built with AI. Your compiler should not be. Its AGENTS.md lets an AI assistant explain, draw, and ask debugging questions, but never write compiler code. The reference compiler in _reference/ is the oracle the tests compare against. Run it as often as you like, and leave it closed. It's written in Python too, so a glance at it would hand you answers you're meant to find yourself.It's written in Python, so it can't be pasted into your OCaml, but reading it would still spoil the puzzles.

An independent course inspired by the public 10-411 syllabus (technical syllabus). Not affiliated with CMU or George Hotz. Spec: spec/uir-week1.md.