time-to-first-token

Overview

This repository is a structured 10-week learning roadmap for engineers who want to serve LLMs in production. It organizes 50 half-hour sessions around building one OpenAI-compatible inference service, rather than disconnected experiments. Learners deploy it on a rented GPU, instrument and load-test it, then explore serving and optimization choices such as quantization, speculative decoding, and cost-aware routing. The roadmap ends with a reproducible benchmark writeup and is intended for people comfortable with Python, transformer concepts, and the command line.

Key features

• 50 half-hour sessions organized across 10 weeks, with buffer days for catching up.
• Build and deploy one OpenAI-compatible inference service on a rented GPU.
• Instrument the service and load-test it beyond 1,000 concurrent requests, tracking metrics such as TTFT, latency, throughput, and cost.
• Explore quantization, speculative decoding, and cost-aware routing, then publish a reproducible benchmark.

Best for

Engineers comfortable with Python, transformer concepts, and the command line who want practical experience running and optimizing LLM inference. No prior serving, Kubernetes, or CUDA experience is required.