LLMs-from-scratch
Code for building, pretraining and finetuning a GPT-like language model in PyTorch, step by step. It accompanies a book and teaches how large language models work.
Share on XCustom license (see repository)
Overview
This is the official code repository for the book Build a Large Language Model (From Scratch). It contains the code for developing, pretraining and finetuning a GPT-like model in PyTorch, going step by step with text, diagrams and examples. The approach is meant for education and mirrors how large foundation models are made, but at a small scale. The book also covers loading weights of larger pretrained models.
Key features
- Step by step code for building a GPT-like LLM in PyTorch
- Covers pretraining and finetuning
- Includes code for loading weights of larger pretrained models
- Provided as Jupyter Notebooks
Best for
Readers who want to understand how LLMs work by coding one themselves. It is written for learning and produces a small model, and it accompanies the book.
- Upstream
- rasbt/LLMs-from-scratch
- Fork on GitHub
- Guo-astro/LLMs-from-scratch
- Upstream stars
- 106k
- Category
- Learning, interviews and curated lists
- Language
- Jupyter Notebook
- License
- Custom license (see repository)
- Forked
- 2026-08-10
- Sync status
- In syncLast synced 2026-10-10
More in Learning, interviews and curated lists
A Jupyter Notebook book on self-teaching for learning coding and study habits. It gives readers a practical path for learning by reading, practicing, and improving their own methods.
Forked 2026-10-08Last synced 2026-10-10No license declaredLearning, interviews and curated listsGitHub
A learning repository for Machine Learning Systems, bringing textbook concepts, labs, tools, and hardware context together so learners can study and build real AI engineering workflows.
Forked 2026-10-07Last synced 2026-10-10Custom license (see repository)Learning, interviews and curated listsGitHub
ThinkStats materials for the third edition, with notebooks, data, and supplemental files for learning statistics through examples and exercises in a runnable local setup.
Forked 2026-10-07Last synced 2026-10-10License: MITLearning, interviews and curated listsGitHub
A Spanish-language guide for software engineering technical interviews, published as a browsable website. It brings together topics and references for interview preparation and reviewing core engineering concepts.
Forked 2026-10-06Last synced 2026-10-10License: MITLearning, interviews and curated listsGitHub