Finding Spark Waste Beyond CPU Utilization
High CPU utilization doesn't mean your Spark jobs are efficient — it just means they're busy. This article walks through four layers where waste hides: misaligned executor configs, skewed data distribution, slow storage access, and wasteful application logic that processes data you later discard.
- Executor shape matters as much as utilization: a poorly sized executor can max out CPU while leaving machine capacity stranded. Fixing core-to-memory ratios cut one workload's vCore-hours by 54% and runtime by 83%.
- Data skew concentrates work into a few slow tasks while others finish early. Identifying the specific hot key (market_id=2147) and salting it across partitions reduced one stage's runtime by 24%.
- Spark often spends time listing and reading data it doesn't need before it even starts processing. Late filtering in query plans can keep a cluster busy processing billions of intermediate records that get discarded later.
Spark jobs often look expensive because of poor resource utilization, but that's only part of the story. High CPU usage on an executor doesn't mean the machine itself is being used efficiently—you might have orphaned vCores sitting idle because executor sizing and machine shape create packing constraints. The authors walk through four layers where waste hides: infrastructure (executor topology), Spark execution (task distribution), data access (storage efficiency), and query logic (unnecessary computation). In one real example, a team kept adding memory to fix out-of-memory errors without fixing the core-to-memory ratio. Their executors needed 8 cores but had 46GB of memory, leaving CPU stranded. Reshaping executors to 8 cores and 46GB, then using dynamic allocation (10-75 executors based on demand), cut vCore-hours by 54% and runtime by 83%. The point: you can't see these problems by staring at utilization metrics alone.
Task-level skew is another common culprit that averages completely hide. In one workload, data skew ate up 54% of execution time across 148 runs over 79 days—roughly 7 minutes out of every 13-minute run. Average task duration was 27 seconds, but the max was 3 minutes 33 seconds. That gap tells you something's wrong, but you need to dig deeper to find which key is causing the skew and how much it's actually costing. Shuffle, spills, and stragglers create similar patterns where a few slow tasks hold up an entire stage, and overall utilization numbers won't expose them.
Beyond execution, Spark often wastes time on data access—listing and reading files it doesn't actually need—and on computation that generates huge intermediate datasets only to discard them later. One pipeline ran for 47 hours because of a single line of code doing unnecessary work. The core insight is that utilization tells you how busy Spark is, not whether the work makes sense. You need to trace waste across all four layers: infrastructure, execution, storage, and application logic. Each has different failure modes and different signals that top-level metrics will completely miss.