Reddit - The heart of the internet
A startup tech lead who’s only used cron seeks an orchestration tool for batch ETL jobs with dependency logic. They ask whether to pick Dagster or Airflow and question their relative resource usage. The community is invited to share experiences on reliability, scheduling flexibility, and system overhead.
- Airflow's scheduler/webserver can spike RAM, especially with hundreds of DAGs, and needs a metadata DB plus a message broker for Celery, adding operational overhead
- Dagster's code-first Python pipelines offer lighter UI, built-in lineage tracking, and lower resource use, particularly on Kubernetes or on-demand workers
- Airflow has a more mature ecosystem, larger community, and proven stability at scale, while Dagster's plugin/community support is less extensive but integrates well with dbt, Spark, and Snowflake
- The choice comes down to ecosystem maturity and infrastructure tolerance (Airflow) versus faster setup and operational simplicity (Dagster)
You’re running a growing startup that’s relied on cron for all your batch ETL work so far. Every job’s dependencies live in your scripts, and you invoke them on fixed schedules. Now that you need more reliability and visibility, you’re weighing Dagster versus Airflow. Both can handle jobs at varying intervals, build dependency graphs, and retry failed tasks. You’ve heard Airflow can be memory-hungry, but you’re not certain it’ll actually strain your infrastructure more than Dagster.
Airflow gives you a mature UI, a large ecosystem of operators and community plugins, and battle-tested stability at scale. Its scheduler and webserver can spike RAM usage, especially when you have hundreds of DAGs defined. You’ll also need a metadata database (often Postgres) and a message broker (if you run the Celery executor). Installing and configuring all those components adds operational overhead.
Dagster takes a code-first approach. You define pipelines in Python, and its type system and asset catalog help you reason about data flows. The UI is lighter out of the box. You get built-in lineage tracking and testing hooks for each step. It’s newer, so community support and third-party plugins aren’t as extensive as Airflow’s—but it integrates smoothly with modern data tooling (dbt, Spark, Snowflake). Resource use tends to be lower, especially if you run tasks on Kubernetes or spin up workers on demand.
If you need a proven, widely supported platform and don’t mind extra infrastructure, Airflow remains the industry standard. If you want faster setup, clear code-based pipelines, and less background bloat, Dagster is worth a closer look. You’ll make trade-offs around ecosystem size versus developer ergonomics and operational simplicity.