1 link tagged with all of: data-engineering + resource-utilization + spark + query-optimization
Click any tag below to further narrow down your results
Links
High CPU utilization doesn't mean your Spark jobs are efficient — it just means they're busy. This article walks through four layers where waste hides: misaligned executor configs, skewed data distribution, slow storage access, and wasteful application logic that processes data you later discard.
- Executor shape matters as much as utilization: a poorly sized executor can max out CPU while leaving machine capacity stranded. Fixing core-to-memory ratios cut one workload's vCore-hours by 54% and runtime by 83%.
- Data skew concentrates work into a few slow tasks while others finish early. Identifying the specific hot key (market_id=2147) and salting it across partitions reduced one stage's runtime by 24%.
- Spark often spends time listing and reading data it doesn't need before it even starts processing. Late filtering in query plans can keep a cluster busy processing billions of intermediate records that get discarded later.