Click any tag below to further narrow down your results
Links
AWS has released AWS Emulator, a free offline copy of AWS that coding agents can test against, and the whole thing was written by agents checking themselves against the real cloud. The piece also argues that the engineer's job shifts toward writing specs, tests, and merge gates that decide when agent-written code is correct.
- The emulator covers more than 200 AWS services and was built from 400+ public API models in Smithy, with every line written and tested by agents.
- Agents work from fidelity tests that record how real AWS behaves, then build the emulator until it passes the same tests; a prototype SQS emulator took about 15 minutes from one prompt.
- The author's own experience: an agent struggled for four nights rebuilding a replication engine with no way to check its results, then finished in about two hours once it had a spec and a test environment.
OpenAI built Pando, a new storage and serving backend for Git that keeps the Git client and protocol but handles many concurrent pushes and very high read volume. It now runs their monorepo and cut median push latency nearly 5x.
- Pando coordinates each branch independently and uses causal consistency instead of repository-wide ordering, so pushes to unrelated branches don't contend on one coordination point.
- It reuses packfiles across fetches rather than building one per client, with local caches hitting over 99%, which lets clones and fetches saturate 10 Gbps links on standard cloud VMs.
- In production it delivers nearly a petabyte of Git data per day, and the team contributed upstream changes to Git such as packfile-uri improvements and "explicit haves."
Sean Goedecke argues that strong system design comes from deliberately dropping requirements like strict consistency or instant delivery, using examples from Unicorn, S3, StatsD, and SQLite. He also warns that coding agents tend to pile on unnecessary requirements that make code needlessly complex.
- Systems like Unicorn, S3, StatsD, and SQLite gain speed, simplicity, or scalability by giving up guarantees such as shared state, atomic multi-key updates, delivery of every packet, or concurrent writes.
- Deciding which requirements to relax needs both technical and business input, and trust gaps between engineers and product managers often block that tradeoff.
- Solo developers and startups can make these tradeoffs more easily because one person holds both the technical and business context.
- LLM coding agents tend to invent edge-case requirements (extra mutexes, TOCTOU checks, consistency guards), so reviewers need to strip them out or the codebase becomes unmanageable.
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.
+ northguard
+ xinfra
+ kafka
+ linkedin
distributed systems
+ metadata management
+ storage architecture
This course covers core concepts of concurrency control—threads, locks, transactions, and crash recovery—in the first half, then shifts to distributed systems topics like network models, clocks, replication, consensus, and fault tolerance. It lists lecture topics, objectives, prerequisites, and recommended readings for a Part IB CST Michaelmas module led by Dr. Martin Kleppmann.
- Kleppmann's 16-lecture Part IB course splits evenly: 8 lectures on concurrency control, 8 on distributed systems.
- The concurrency half moves from basic threads/locks through bakery algorithm, semaphores, monitors, and message-passing (actors, CSP) to ACID transactions, 2PL, optimistic concurrency, and crash recovery.
- The distributed half covers network/failure models, physical and logical clocks, replication and consensus (Raft, FLP impossibility, 2PC, CAP), ending with case studies on CRDTs and Google Spanner's TrueTime.
- Core texts backing the course are Tanenbaum's Modern Operating Systems, Goetz's Java Concurrency in Practice, and Kleppmann's own Designing Data-Intensive Applications.
Martin Kleppmann discusses his journey from startups to academia, and the new edition of his book Designing Data-Intensive Applications. They cover trade-offs in modern infrastructure, cloud scalability, distributed system challenges, and emerging topics like formal verification and local-first software.
- Kleppmann rewrote and reorganized Designing Data-Intensive Applications' second edition to add consensus algorithm case studies and deeper cloud failure-handling coverage
- Cloud infrastructure has shifted the core scaling trade-offs from raw hardware capacity to network behavior and API guarantees
- Kleppmann is now researching local-first software algorithms for offline-first collaboration and cryptographic methods for supply-chain transparency that preserve pricing/manufacturing secrecy
- He predicts formal verification will become more important as AI-assisted coding introduces subtle bugs that are hard to catch otherwise
In this Pragmatic Engineer episode, Martin Kleppmann walks through updates in the second edition of Designing Data-Intensive Applications and shares how his LinkedIn experience shaped the book’s core concepts. He breaks down trade-offs in multi-region and cloud architectures, explains why replication still matters more than sharding, and predicts a rise in formal verification and local-first software.
- Second edition of DDIA cuts MapReduce coverage in favor of Spark and Flink, reflecting how the field has moved on
- Manual sharding has become rare thanks to bigger hardware and managed services, making replication the fault-tolerance skill every team now needs
- Kleppmann predicts LLMs will make formal verification mainstream by automating proof generation, shifting the bottleneck to human review
- Architecture decisions like multi-region or multi-cloud are business trade-offs between risk and cost, not universal best practices
Maintaining consistency in a system comprised of separate databases can be challenging, particularly in the absence of transactions. The article discusses the importance of defining a system of record versus a system of reference and emphasizes the Write Last, Read First principle to ensure safety properties like consistency and traceability in financial transactions.
- Write to the system of record last, but read from it first — this ordering prevents referencing data that hasn't actually been durably committed.
- If a downstream write (e.g., to a search index or cache) fails after the system of record succeeds, that's recoverable via retries; failing in the other order risks phantom reads of uncommitted state.
- Treating one database as the authoritative system of record (vs. other systems of reference) gives you a clear conflict-resolution and recovery strategy when multiple stores disagree.