1 link tagged with all of: data-engineering + apache-spark + sql + streaming + python
Links
Apache Spark 4.1.0 adds declarative pipelines, real-time streaming mode with sub-second latency, improved PySpark performance, and SQL scripting as a stable feature. The release resolved over 1,800 issues with contributions from more than 230 developers.
- Spark Declarative Pipelines (SDP) lets you define datasets and queries while Spark handles execution graphs, parallelism, checkpoints, and retries automatically.
- Structured Streaming Real-Time Mode enables continuous processing with sub-second latency for stateless tasks, dropping to single-digit milliseconds in some cases.
- Arrow-native PySpark UDFs and UDTFs eliminate Pandas conversion overhead, and Python Data Sources now support filter pushdown to reduce data movement.
- SQL Scripting is now GA and enabled by default, while the VARIANT type (for semi-structured data) is GA with shredding support for faster reads.
apache-spark
data-engineering
streaming
sql
python