Andrej Karpathy

Technical

@AndrejKarpathy 800K subscribers15 briefs

First-principles explanations of LLMs, deep learning, and the history of AI.

llms
fundamentals

Latest video briefs

How I use LLMs

The video focuses on practical applications of large language models (LLMs). Andrej showcases various LLMs, with a focus on ChatGPT from OpenAI. He explains how to interact with these models and provides examples of their capabilities. The ecosystem of LLMs has expanded, offering unique experiences across different platforms.

Feb 27, 2025Read brief

Deep Dive into LLMs like ChatGPT

This video introduces large language models (LLMs) like ChatGPT to a general audience. The video outlines the stages involved in building LLMs, emphasizing data collection and processing techniques. Key concepts include the importance of diverse, high-quality datasets and the challenges in filtering out unwanted content.

Feb 5, 2025Read brief

Let's reproduce GPT-2 (124M)

This video focuses on reproducing the GPT-2 model with 124 million parameters. Key differences include using PyTorch instead of TensorFlow for implementation. Training the model today is more efficient, taking about an hour for roughly $10 on cloud services.

Jun 9, 2024Read brief

2:13:00

Let's Build a Tokenizer From Scratch

A two-hour first-principles build of the GPT-style byte-pair-encoding (BPE) tokenizer in Python. Karpathy is direct: tokenization is his least favorite part of LLMs, and most 'model bugs' you encounter are actually tokenizer bugs. Implements the BPE training algorithm, vocabulary building, and the encode/decode round-trip - then compares to Tiktoken (GPT) and SentencePiece (Llama). Walks through real-world failure cases - SolidGoldMagikarp, arithmetic, code, non-English text - and explains exactly which tokenizer choice causes each.

Feb 20, 2024Read brief

[1hr Talk] Intro to Large Language Models

Large language models consist of two main files: parameters and a run file. The Llama 270b model by Meta AI is one of the most capable open weights models available. To train such models, massive computational resources and extensive datasets are required.

Nov 23, 2023Read brief

Let's build GPT: from scratch, in code, spelled out.

ChatGPT showcases AI's capability to generate text based on prompts. The Transformer architecture is fundamental to understanding GPT models. This video introduces creating a simplified character-level language model using Shakespeare's works.

Jan 17, 2023Read brief

Building makemore Part 5: Building a WaveNet

Continuation of the MakeMore project focusing on WaveNet architecture. Transitioning from a simple character prediction model to a deeper architecture. Implementing a hierarchical approach similar to WaveNet for predicting sequences. Using layered modules akin to PyTorch for building neural networks.

Nov 21, 2022Read brief

Building makemore Part 4: Becoming a Backprop Ninja

Transitioning from automatic backpropagation to manual implementation. Understanding backpropagation is crucial for debugging neural networks. Exploring pitfalls like vanishing gradients and incorrect loss clipping in neural network implementations. Writing backwards passes manually can deepen knowledge of neural networks.

Oct 11, 2022Read brief

Building makemore Part 3: Activations & Gradients, BatchNorm

Continues building the 'makemore' implementation focusing on MLP enhancements. Emphasizes understanding activations and gradients in neural networks, vital for RNN optimization. Introduces code refactoring for clearer implementation and efficiency improvements.

Oct 4, 2022Read brief

Building makemore Part 2: MLP

Extended context prediction enhances language model performance. Implementing a multi-layer perceptron (MLP) improves character-level predictions. Word embeddings capture semantic relationships and allow for generalization.

Sep 12, 2022Read brief

The spelled-out intro to language modeling: building makemore

Andrej Karpathy introduces 'make more', a character level language model. The model generates unique names based on a dataset of 32,000 names. The tutorial will implement various language models, starting from bi-gram to modern transformers like GPT-2.

Sep 7, 2022Read brief

Stable diffusion dreams of psychedelic faces

Exploration of AI's creativity in generating art. Psychedelic faces as a focus of the visual outputs. Stable diffusion technology showcases unique artistic capabilities.

Aug 19, 2022Read brief

Stable diffusion dreams of steampunk brains

Exploration of steampunk aesthetics in AI art. Demonstration of stable diffusion in creating brain-like imagery. Discussion on the intersection of technology and creativity.

Aug 17, 2022Read brief

Stable diffusion dreams of tattoos

Andrej Karpathy discusses tattoo design using Stable Diffusion. The video showcases AI's creative capabilities in generating unique tattoo ideas. Real-time feedback is explored through audience interaction.

Aug 16, 2022Read brief

The spelled-out intro to neural networks and backpropagation: building micrograd

Introduction to building micrograd, an autograd engine. Explanation of backpropagation for efficient gradient evaluation. Demonstration of constructing mathematical expressions using micrograd.

Aug 16, 2022Read brief