← Goal templates

Transformers and large language models

How ChatGPT-style models work, down to attention and a GPT you build yourself.

Understand the transformer, the architecture behind every modern language model: attention, embeddings and training, from 3Blue1Brown's visual chapters and The Illustrated Transformer to the original paper, then build a small GPT with Andrej Karpathy.

intermediateabout 3 weeks at 60 min a day7 free sources

What you'll cover

  1. Phase 1

    The big picture

    • What a language model predicts
    • Tokens and embeddings
    • How LLMs are trained and fine-tuned

  2. Phase 2

    Attention

    • Queries, keys and values
    • Multi-head attention
    • Positional information
    • Where facts are stored

  3. Phase 3

    The paper and the code

    • Reading 'Attention Is All You Need'
    • Encoder and decoder
    • Building a GPT from scratch

An outline. Your own plan is made when you start, around your level and deadline.

Sources

Free, and fetched for you

All sources belong to their creators and are free to watch or read; Confu links to them and turns them into study PDFs for you.

More templates