30 Years of Postgres Architecture: Tom Lane Interview

Open original

Snowflake interviews Tom Lane, a core Postgres developer for 25 years, about the architectural decisions that have kept the database stable and reliable. He discusses why the process-per-connection model works, how the write-ahead log handles crash recovery, and the trade-offs the project continues to navigate.

  • Postgres uses isolated OS processes per connection instead of threads, which simplifies code and improves crash resistance—if one session crashes, it doesn't corrupt shared state or require a full system restart.
  • The write-ahead log (WAL) lets transactions commit as soon as log data hits disk, even though table changes may still sit in shared buffers; this batching approach reduces random disk I/O compared to writing scattered changes immediately.
  • Connection pooling tools like PgBouncer solve the overhead problem of managing thousands of connections, though the core project hasn't fully integrated pooling into Postgres itself.

Snowflake published an interview with Tom Lane, a core PostgreSQL developer for 25 of the database's 30 years, discussing the architectural decisions that have defined the project. Lane still spends his time the same way he did in the '90s: about a third reading and writing email, a third reviewing patches, and a third on his own code work. He uses Emacs and relies heavily on code comments rather than documentation, having internalized where things live in the codebase. The conversation centers on why Postgres has maintained its core design philosophy across three decades: don't lose data, crash and recover fast, and meet business demands through extensions. The interview digs into Postgres' process-per-connection model versus the threaded approach used by MySQL, SQL Server, and Oracle. Lane argues the main win from isolated processes is code simplicity—developers don't need to worry about other threads corrupting data structures mid-operation. There's also crash resistance: if one session process crashes, it can't corrupt the postmaster's state, so recovery is straightforward. The tradeoff is memory overhead and complexity. Some developers are exploring a threaded conversion, but Lane notes it's a massive undertaking with uncertain payoff. He's not directly involved in that work but supports pushing forward to see if threading could offer better overall tradeoffs. The memory architecture splits between private per-session buffers and shared memory accessible to all backends. The shared buffer cache holds all table data so every session sees the same page state—critical for correctness. Temporary tables live in session-local buffers, which means no locking overhead but also no background vacuuming. Each session maintains its own catalog cache, which solves a real problem: different sessions can have different views of the schema (like uncommitted DDL changes), and a shared catalog would make that nearly impossible to manage. Lane explains that write-ahead logging (WAL) lets multiple processes write concurrently by requiring every INSERT, UPDATE, and DELETE to emit a WAL record before actually modifying the buffer, so if the system crashes, recovery can replay those records from the log.

Ask about this article