AI Miracle Decade+

A self-paced reading of every landmark AI paper from DBN (2006) to the GPT-4 Technical Report (2023). Forty papers, seven concept threads, no time budget. The roadmap is patient.

Reading the originals — not summaries — in roughly the order the field experienced them, with cross-cutting concept-page stops that connect papers across lineages. Side quests are first-class: when a paper hooks you, follow your curiosity; the tutor logs the detour and the roadmap pulls you back so nothing important gets quietly skipped.

Audience: technical reader (engineer, scientist, DSP person, hobbyist) who wants to read the actual papers. Math is welcomed but explained when load-bearing. No assumed PyTorch fluency — runnable bits are optional sidecars; if you want to build that fluency in parallel, Karpathy's Neural Networks: Zero to Hero is the canonical hands-on companion.

Source & ways to take the course: josmithiii/Courses/ai-miracle-decade-plus · companion paper repo: music423-2023 on CCRMA GitLab.

The route

Twelve phases, in suggested order. Numbers are stops on a roadmap, not days — spend an hour on one phase or three weeks on another. Each phase ends at a concept-page stop — a cross-cutting essay tying the phase's papers together.

Out of order is fine. Skip ahead if a stop is already familiar; the tutor marks it [~] skimmed (known) so you can return later. A few real prerequisites do exist — Transformer (Phase 4) before the LLM phases, U-Net (Phase 2) before diffusion, Deep RL (Phase 3) before alignment — and the tutor will flag them when you jump.

orientation

Phase 0 — On-ramp

The 17-year arc in one sitting; how to read an AI paper without drowning; tools (arXiv HTML view, pdf2txt, optional PyTorch).

Meta-wiki overview · chronological master list

2006–2012

Phase 1 — The restart and the cash-out

Deep learning restarts with Hinton's layer-wise unsupervised pretraining; cashes out six years later when GPUs + ImageNet + ReLU + Dropout converge on a CNN that crushes the competition.

2006 DBN — A Fast Learning Algorithm for Deep Belief Nets (Hinton, Osindero, Teh)
2012 AlexNet — ImageNet Classification with Deep CNNs (Krizhevsky, Sutskever, Hinton)

pretrain-finetune lineage (first half) — DBN's 12-year dormancy before BERT and GPT-1 cashed in on the idea.

2013–2015

Phase 2 — The machinery years

The training tools and generative scaffolding that made deep learning industrially reliable. Order inside the phase is flexible.

2013 word2vec (Mikolov et al.) — geometry encodes meaning
2013 VAE (Kingma, Welling) — variational latent generative models
2014 GANs (Goodfellow et al.) — adversarial two-player training
2014 Seq2Seq (Sutskever, Vinyals, Le) — encoder–decoder pattern
2014 Adam (Kingma, Ba) — adaptive moments
2014 Dropout (Srivastava et al.) — cheap implicit ensembling
2015 Batch Normalization (Ioffe, Szegedy)
2015 U-Net (Ronneberger et al.) — encoder–decoder with skips; returns as the diffusion backbone
2015 ResNet (He et al.) — residual connections, depth unbounded

representation-learning arc (first half) — word2vec's geometry thesis, before ELMo and CLIP take it further.

2013–2017

Phase 3 — Deep RL comes of age

From Atari pixels to Go to chess and shogi from scratch — the same machinery will return in 2022 as the alignment layer for LLMs.

2015 DQN (Mnih et al., Nature) — experience replay + target network + CNN on pixels · 2013 workshop predecessor
2016 AlphaGo (Silver et al., Nature) — policy + value + MCTS
2017 AlphaZero (Silver et al.) — self-play from scratch

RL-and-alignment arc (first half) — the game-RL boom; the alignment half waits for Phase 9.

2016–2017

Phase 4 — Sequences, audio, and the attention turn

Causal dilated convolutions for raw audio; attention reorganizes sequence modeling; discrete latents seed the audio-codec future.

2016 WaveNet (van den Oord et al.)
2017 Transformer — Attention Is All You Need (Vaswani et al.) — read this one carefully
2017 VQ-VAE (van den Oord, Vinyals, Kavukcuoglu) — discrete latents

2018–2019

Phase 5 — Pretrain & fine-tune crystallizes

The hinge of the entire decade. Read GPT-1 before BERT so the framing tension lands. BERT won 2018's leaderboards; GPT's autoregressive bet won the next five years.

2018 ELMo (Peters et al.) — contextual word embeddings via bi-LSTM
2018 GPT-1 (Radford et al., OpenAI) — generative pretraining + fine-tune
2018 BERT (Devlin et al.) — bidirectional masked LM
2019 GPT-2 (Radford et al., OpenAI) — unsupervised multitask via scale

autoregressive vs gap-filling — the 2018 fork and its historical irony. · pretrain-finetune lineage (full).

2020

Phase 6 — Scaling and emergence

Loss as a smooth power law — and the surprise of in-context learning, not predicted by the scaling laws.

2020 Scaling Laws for Neural Language Models (Kaplan et al.)
2020 GPT-3 — Language Models are Few-Shot Learners (Brown et al.)

scaling and emergence (first half) — Kaplan's predictable side; Chinchilla's revision waits for Phase 8.

2020–2022

Phase 7 — Diffusion, ViT, and the text-to-image boom

Denoising diffusion as a tractable generative framework; images become token sequences; CLIP becomes the multimodal substrate; latent diffusion makes it all practical.

2020 DDPM (Ho, Jain, Abbeel)
2020 ViT (Dosovitskiy et al.)
2021 DALL-E (Ramesh et al., OpenAI)
2021 CLIP (Radford et al., OpenAI)
2021 LDM / Stable Diffusion (Rombach et al.)
2022 Imagen (Saharia et al., Google)

generative-modeling frameworks · diffusion vs autoregressive by modality.

2022

Phase 8 — Post-Chinchilla LLM era

Data and parameters scale together; reasoning emerges from prompting; retrieval bolts onto the LLM.

2022 Chinchilla (Hoffmann et al., DeepMind) — revises Kaplan
2022 PaLM (Chowdhery et al., Google) — 540B
2022 Chain-of-Thought Prompting (Wei et al.)
2020 RAG (Lewis et al.) — earlier date but lands here pedagogically

scaling and emergence (full).

2022

Phase 9 — Alignment

The same RL machinery that solved Go now solves "be helpful and harmless." Then: AI feedback replacing some of the human labels.

2022 InstructGPT (Ouyang et al.) — RLHF; the recipe behind ChatGPT
2022 Constitutional AI (Bai et al., Anthropic)

RL-and-alignment arc (full).

2022–2023

Phase 10 — Capabilities and the post-2022 frontier

Weak-supervised ASR at scale; a real science breakout; agents that teach themselves to use tools; the open-weights ecosystem begins; and the new normal of closed frontier reports.

2021 AlphaFold 2 (Jumper et al., Nature) — a science breakout
2022 Whisper (Radford et al., OpenAI) — 680K hours of weak-labeled audio
2023 Toolformer (Schick et al.) — self-supervised tool use
2023 LLaMA (Touvron et al., Meta) — open-weights, Chinchilla-compute-optimal
2023 GPT-4 Technical Report (OpenAI) — read as a report, not a paper

synthesis

Phase 11 — Where to go next

Re-read the overview — it should feel like a memoir now, not a syllabus. Write your own “what's dead / what's alive” note on a landmark of your choice. Pick a post-2023 frontier paper (multimodal foundation models, recent audio/music models) and read it freely. Optional capstone: a replication or a teaching writeup.

Take the course

The course is run as an interactive tutor inside Claude Code. Each stop is one session; the tutor reads your private progress.md, picks up where you left off, and logs the day's reading and any side quests.

git clone https://github.com/josmithiii/Courses
cd Courses/ai-miracle-decade-plus
claude       # launches Claude Code in this folder
/lesson      # begins (or resumes) the next stop

On a fresh machine the /lesson command seeds your private progress.md from the shipped template — nothing personal lives in the repo. Set COURSES_DATA_DIR to put learner state somewhere else (the seam a future web-app version flips per-user).

The full curriculum is the canonical version of this page — longer, with the pace philosophy, the side-quest protocol, and the teaching method spelled out. The page you're reading is the orientation.