train-llm-from-scratch
A step-by-step tutorial for training your own transformer language model with PyTorch, from raw text to text generation and alignment. Aimed at learners who want to understand how LLMs are built.
Pick a small issue you can reproduce, then fix it with your AI coding agent.
Share on XLicense: MIT
Overview
This is a step-by-step tutorial and set of scripts for training your own transformer language model in PyTorch, based on the Attention Is All You Need paper. It walks from raw text and tokens to a base model, then supervised fine-tuning, a reward model, PPO, DPO and GRPO, ending with evaluation and chat. The algorithms are written by hand in plain PyTorch without trl, peft or transformers. The author says a million or billion parameter model can be trained on a single GPU.
Key features
- Transformer implemented from scratch in plain PyTorch
- Path from raw text to a pretrained base model
- Covers SFT, reward model, PPO, DPO and GRPO
- No trl, peft or transformers dependency
- Ends with evaluation and chat
Best for
Learners who want to understand how language models are built and aligned by reading and running every step themselves. It is a teaching project, and the sample output shown is from a small 13 million parameter model.
- Upstream
- FareedKhan-dev/train-llm-from-scratch
- Fork on GitHub
- Guo-astro/train-llm-from-scratch
- Upstream stars
- 12k
- Category
- AI agents and LLM tools
- Language
- Python
- License
- MIT
- Forked
- 2026-09-25
- Sync status
- In syncLast synced 2026-10-19
More in AI agents and LLM tools
A collection of AI research skills and guides that helps researchers with idea evaluation, literature research, paper writing, and pre-submission review.
Forked 2026-10-18Last synced 2026-10-19Custom license (see repository)AI agents and LLM toolsGitHub
RLVE supports language model reinforcement learning with adaptive, verifiable environments for generating problems and evaluating model outputs.
Forked 2026-10-17Last synced 2026-10-19License: MITAI agents and LLM toolsGitHub
Provides modular Android development instructions for AI agents, helping them follow specialized workflows. Install individual skills or the full set with Android CLI.
Forked 2026-10-16Last synced 2026-10-19License: Apache-2.0AI agents and LLM toolsGitHub
Pi Agent is a local desktop app for using the Pi AI coding agent on macOS, Windows, and Linux. It helps you manage coding sessions, work with project files, and run saved prompts without setting up a terminal environment.
Forked 2026-10-15Last synced 2026-10-19License: MITAI agents and LLM toolsGitHub