This article explains Spark Declarative Pipelines (SDP), a framework for creating data pipelines in Spark. It covers key concepts like flows, datasets, and pipelines, along with how to implement them in Python and SQL. The guide also includes installation instructions and usage of the command line interface.

Bruin is a data pipeline tool that integrates data ingestion, transformation, and quality checks into one framework. It supports SQL, Python, and R while working across major data platforms, whether on a local machine or cloud services like EC2. The tool offers built-in features like Jinja templating and data validation for streamlined workflows.

Daft is a distributed query engine designed for large-scale data processing using Python or SQL, built with Rust. It offers a familiar interactive API, powerful query optimization, and seamless integration with data catalogs and multimodal types, making it suitable for complex data operations in cloud environments. Daft supports interactive and distributed computing, allowing users to efficiently handle diverse data types and perform operations across large clusters.