Interactive distributed-training exercises
Build training traces by placing kernels and collectives in dependency order. Start with a single GPU, then add TP, FSDP, EP, and CP.
A quick note on traces
When you profile a training run, you record the work done by the CPU and GPUs and how long it takes. That includes kernel launches, GPU kernels, memory copies, collectives, and synchronization.
A trace lays those events out over time. Each row represents an execution context, such as a CUDA compute stream or communication stream. Reading from left to right shows what ran, what waited, and what overlapped.
A rectangle represents one operation or a small group of operations. Its position shows when it started; its width shows how long it ran.
Operations do not run just because they appear next in the Python code. They wait for their inputs, parameters, and any required communication.
Work on separate streams may run at the same time. If communication continues after useful compute ends, it remains on the critical path.
Place operations in the order they can run. Work out what is ready, what is waiting, and what can overlap.
Challenge library
Build the output of common collectives across four ranks before reasoning about them inside a trace.
Reconstruct execution order from individual operations and collectives.
Compare completed schedules and identify the dependencies that placed each block.
Read an anonymized measured trace, identify the setup, and point to the evidence.
Result chunks
Event palette
Kernel names and stream identities are hidden. Use event types, timing, and dependency structure to identify the parallelism.
Colors preserve operation types; width and placement preserve the measured geometry.