Click any tag below to further narrow down your results
Links
Karpathy outlines three software eras: human-written code, trained neural weights, and now natural language prompts. He claims LLMs treat English as code, making traditional coding steps optional. The linked talk will explain why this shift matters in 40 minutes.
- Karpathy frames software history as three eras: 1.0 (hand-written code), 2.0 (neural net weights), and now 3.0 (natural language prompts)
- In Software 3.0, English prompts function as code and LLMs act as the CPU running them
- This makes traditional coding infrastructure (compilers, libraries, frameworks) optional rather than mandatory
- A 40-minute talk/demo is promised to explain why this shift matters
This article breaks down the core concepts behind LLMs—from next-token prediction training to tokens, vectors and attention layers—to show how they generate text. It also covers context windows, parameters and why model scale affects performance.
- LLMs work purely through next-token prediction learned by hiding and guessing words billions of times during training, not through actual understanding of letters or math.
- Context windows now range wildly, from 200K tokens (~150K words) up to Llama 4 Scout's 10 million tokens, but bigger windows don't fix factual errors or logical gaps.
- Because models process text as tokens/vectors rather than raw letters, they inherently struggle with tasks like counting letters or doing arithmetic.