Click any tag below to further narrow down your results
Links
SeaTunnel automatically detects upstream schema changes and propagates them downstream in real time, allowing data pipelines to continue without manual intervention. When a column is added or modified, SeaTunnel updates the schema on the fly, eliminating pipeline pauses.
- SeaTunnel auto-detects upstream schema changes (added/dropped/renamed columns) and propagates them downstream in real time without pausing jobs
- It works across sources like Kafka, relational databases, and cloud storage, updating pipeline schema definitions on the fly to prevent ETL breakage
- Multiple sink connectors (ClickHouse, Elasticsearch, HDFS, S3) automatically inherit schema updates, keeping analytics and warehouses in sync
- Eliminates manual intervention, custom migration scripts, and "field not found" errors while providing audit trails of schema changes
SQLBuild plugs into an existing dbt project to fingerprint models, skip unchanged work, and clone production tables in development instead of rebuilding them. It stores all state in append-only warehouse tables, adds ingestion and Python nodes, and offers built-in testing, audits, and incremental processing with no external services.
- Wraps your existing dbt project (no file edits, no new models) by fingerprinting models/seeds/functions/Python nodes and skipping anything unchanged, even cloning unchanged prod tables instead of rebuilding them.
- All state stored as append-only tables in the warehouse itself—no external service, manifest files, or state machine required.
- Runs audits before data lands: full builds stage until checks pass, incremental models validate each microbatch before merging.
- One pip install gets you a runnable sample project, and it already supports DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, Postgres, and SQL Server.
Amazon MWAA runs Airflow pipelines on AWS as a managed service with serverless or provisioned deployment options. It handles infrastructure provisioning, scaling, security, and integrates with SageMaker for end-to-end ML and data workflows.
- MWAA now offers two deployment modes: serverless (pay-per-execution, auto-scaling) and provisioned (full config control with AWS-managed infrastructure)
- It integrates directly with the next-gen SageMaker Unified Studio, letting Airflow run alongside Jupyter notebooks for unified data prep, training, and inference pipelines
- Both modes run inside your VPC with end-to-end encryption and IAM controls, with serverless adding workflow-level permissions and provisioned offering environment-wide team separation