LLMs-from-scratch

Code for building, pretraining and finetuning a GPT-like language model in PyTorch, step by step. It accompanies a book and teaches how large language models work.

Browse open issues

Pick a small issue you can reproduce, then fix it with your AI coding agent.

Share on XCustom license (see repository)

Overview

This is the official code repository for the book Build a Large Language Model (From Scratch). It contains the code for developing, pretraining and finetuning a GPT-like model in PyTorch, going step by step with text, diagrams and examples. The approach is meant for education and mirrors how large foundation models are made, but at a small scale. The book also covers loading weights of larger pretrained models.

Key features

  • Step by step code for building a GPT-like LLM in PyTorch
  • Covers pretraining and finetuning
  • Includes code for loading weights of larger pretrained models
  • Provided as Jupyter Notebooks

Best for

Readers who want to understand how LLMs work by coding one themselves. It is written for learning and produces a small model, and it accompanies the book.

Upstream
rasbt/LLMs-from-scratch
Fork on GitHub
Guo-astro/LLMs-from-scratch
Upstream stars
106k
Category
Learning, interviews and curated lists
Language
Jupyter Notebook
License
Custom license (see repository)
Forked
2026-08-10
Sync status
In syncLast synced 2026-10-20

More in Learning, interviews and curated lists