More on the topic…
Spark jobs often look expensive because of poor resource utilization, but that's only part of the story. High CPU usage on an executor doesn't mean the machine itself is being used efficiently—you might have orphaned vCores sitting idle because executor sizing and machine shape create packing constraints. The authors walk through four layers where waste hides: infrastructure (executor topology), Spark execution (task distribution), data access (storage efficiency), and query logic (unnecessary computation). In one real example, a team kept adding memory to fix out-of-memory errors without fixing the core-to-memory ratio. Their executors needed 8 cores but had 46GB of memory, leaving CPU stranded. Reshaping executors to 8 cores and 46GB, then using dynamic allocation (10-75 executors based on demand), cut vCore-hours by 54% and runtime by 83%. The point: you can't see these problems by staring at utilization metrics alone.
Task-level skew is another common culprit that averages completely hide. In one workload, data skew ate up 54% of execution time across 148 runs over 79 days—roughly 7 minutes out of every 13-minute run. Average task duration was 27 seconds, but the max was 3 minutes 33 seconds. That gap tells you something's wrong, but you need to dig deeper to find which key is causing the skew and how much it's actually costing. Shuffle, spills, and stragglers create similar patterns where a few slow tasks hold up an entire stage, and overall utilization numbers won't expose them.
Beyond execution, Spark often wastes time on data access—listing and reading files it doesn't actually need—and on computation that generates huge intermediate datasets only to discard them later. One pipeline ran for 47 hours because of a single line of code doing unnecessary work. The core insight is that utilization tells you how busy Spark is, not whether the work makes sense. You need to trace waste across all four layers: infrastructure, execution, storage, and application logic. Each has different failure modes and different signals that top-level metrics will completely miss.
Questions about this article
No questions yet.