More on the topic…
The core problem: AI is making data work harder, not easier. Companies expect their data teams to handle more complexity—better governance, observability, quality monitoring, self-service analytics—while budgets stay flat. According to dbt Labs' 2025 survey, only 30% of companies got bigger data budgets and 40% got more headcount. Meanwhile, data volumes keep exploding from IoT, generative AI, and digital activity. A single data engineer at a $10 billion startup working 60+ hours can't scale. The author argues this isn't just a people problem; it's architectural. Teams get buried maintaining tools and fighting fires instead of building. Burnout happens when the work scales with data volume but the headcount doesn't.
The real trap is that AI makes automation accessible to everyone—product, finance, operations can now build workflows with n8n without waiting for engineers. Trendyol scaled to 1,000 n8n users and 700 active workflows. Sounds great until you realize every new automation creates another dependency on data, infrastructure, and APIs. Without central visibility and governance, citizen automation doesn't lighten the data team's load; it quietly piles on more work. The solution isn't hiring your way out or picking random tools. You need one control plane that handles orchestration, compute, and agents without fragmenting into an unmaintainable mess. Databricks and Orchestra win here because they reduce the number of systems to manage.
The author lays out four scenarios based on company maturity and AI readiness. Early-stage SaaS companies with limited data readiness should keep it simple—use Python and dbt on Orchestra with Snowflake. Late-stage tech companies often make the opposite mistake: their engineers want to build everything themselves, turning a basic Airflow instance into an ungovernable multi-tenant disaster over 8 months. The fix is drawing a hard line between platform teams and functional teams. Use clusters for compute-heavy work like warehousing and ingestion to save money, but buy SaaS for orchestration and control planes. The real bottleneck isn't tokens or models—it's operations. AI makes building cheaper, which creates more pipelines and more things to operate. Without scaling operations alongside data volume, your team will burn out.
Questions about this article
No questions yet.