Microsoft will invest A$25 billion (US$18 billion) by 2029 to expand Australia’s digital infrastructure, AI supercomputing capabilities and cloud capacity. The move aims to boost commercial cloud services and AI/GPU offerings for local customers.

In this Pragmatic Engineer episode, Martin Kleppmann walks through updates in the second edition of Designing Data-Intensive Applications and shares how his LinkedIn experience shaped the book’s core concepts. He breaks down trade-offs in multi-region and cloud architectures, explains why replication still matters more than sharding, and predicts a rise in formal verification and local-first software.

This article explores the evolving role of data engineers over the past 50 years, highlighting their often unnoticed contributions to data infrastructure. It discusses the challenges they face, such as managing dependencies and schema changes, while emphasizing that the core problems remain unchanged despite new tools and technologies.

Data engineering teams are facing soaring infrastructure costs that challenge the initial promises of cloud scalability. With fragmented systems and a lack of financial awareness, organizations struggle to manage expenses effectively, but embracing a platform team model and improved cost visibility can lead to significant savings and optimized operations.