Apache Spark 4.1.0 shipped with over 1,800 fixes and features from 230+ contributors. The release focuses on four main areas: higher-level data engineering through new declarative pipelines, sub-millisecond latency streaming, better PySpark performance, and expanded SQL capabilities. The standout feature is Spark Declarative Pipelines, a new framework that lets you define datasets and queries while Spark automatically handles execution graphs, dependency ordering, parallelism, checkpoints, and retries — essentially abstracting away the orchestration complexity. On the streaming side, Structured Streaming now officially supports Real-Time Mode with continuous processing that can hit single-digit millisecond latencies for stateless tasks. PySpark got meaningful performance improvements: new Arrow-native UDF and UDTF decorators eliminate Pandas conversion overhead, and Python Data Sources now support filter pushdown to reduce unnecessary data movement. Spark ML on Connect reached general availability for Python with better model caching and memory management. The release also stabilizes large workloads through zstd-compressed protobuf plans and chunked Arrow result streaming. SQL Scripting and the VARIANT type both hit GA status and are now enabled by default. VARIANT gained shredding for faster semi-structured data reads, and the SQL layer picked up recursive CTE support plus new approximate sketch functions (KLL and Theta) for probabilistic aggregations. Connector improvements span multiple databases — join pushdown now works with Oracle, Postgres, MySQL, and SQL Server. The release also bumped Java and Scala library versions and added a JDBC driver for Spark Connect, expanding how you can interact with Spark clusters.