Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day
Salesforce's Data 360 team processed eight million daily Spark jobs and discovered that successful execution doesn't mean efficient resource use. They built instrumentation to measure actual CPU, memory, and executor usage without changing application code, then used those metrics to cut compute costs 31% while improving job throughput.
- Successful job completion masks resource waste: the team found workloads requesting 40 executors where most did zero useful work, yet the job still succeeded, making waste invisible without detailed telemetry.
- They injected a DataFlint Spark plugin through a shared submission gateway (DPC) to collect metrics via Kafka into an Iceberg lakehouse, avoiding the need to instrument hundreds of application teams' code.
- Raw metrics mislead without careful interpretation: low executor utilization can signal either oversized allocation or a shuffle bottleneck requiring different fixes; they built a curated analytical layer (Silver) to distinguish waste from legitimate bottlenecks.
- Reducing initialExecutors from 40 to 4 with dynamic allocation and lowering max-executor limits by 20% cut weekly compute costs 31% over a month in production with no runtime, retry, or failure regressions.
Salesforce's Data 360 platform processes eight million Spark jobs daily, but the team couldn't tell whether their resource allocations were actually necessary or just working by accident. A job completing successfully doesn't mean it used its allocated capacity efficiently — you might have 40 executors running when only a handful actually do useful work, creating unnecessary pod churn and wasted compute. The core problem: success masks waste. You need to distinguish between genuinely unused capacity and real bottlenecks like memory pressure or shuffle failures, then prove your optimizations are safe before cutting resources.
To gather evidence without rewriting every application, they built instrumentation at a shared boundary — the Data Processing Controller, the gateway handling all managed Spark submissions. They extended an open-source DataFlint plugin that hooks into Spark's listener system, capturing metrics about which executors received tasks, CPU and memory consumption, and whether execution spilled to disk or retried. This telemetry feeds through Kafka into an Iceberg lakehouse, all without touching customer code. The safety mechanism matters: listeners run on a separate queue, and jobs continue even if instrumentation fails, with alerts catching any gaps.
Raw metrics lie constantly, though. An incomplete run doesn't show what a complete one needs. Retries inflate work counts. Memory spill and disk spill describe the same pressure event, not two separate problems. They built a curated analytical layer called Silver that models the execution hierarchy correctly — applications, jobs, stage attempts, executors — and defines boundaries explicitly: which runs count as complete, which environment they ran in, what's missing. Low executor utilization can mean either you're over-provisioned or you're hitting a shuffle bottleneck that needs reliability work instead. Zero-task executors suggest unnecessary allocation, but high idle CPU among working executors points to different waste. The analysis had to answer concrete questions (which expensive workloads repeatedly waste money?) rather than treating every low-utilization job as equally important.