5 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
This article discusses a new optimization in ClickHouse 25.11 that enhances the performance of aggregations with small GROUP BY keys by parallelizing the merge phase. The author shares insights from the implementation process, including challenges faced and lessons learned about memory management and concurrency.
If you do, here's more
In ClickHouse 25.11, parallel merge for small GROUP BY queries was introduced, significantly enhancing aggregation speed for 8-bit and 16-bit keys. Jianfei Hu, who implemented this optimization, shared insights from his engineering process, which revealed the complexities of ClickHouse's aggregation mechanics. The article highlights how query performance can vary drastically even with minor changes, such as treating a grouping value as UInt16 versus UInt64, which can lead to significant differences in execution time.
The optimization process involved transitioning from a one-dimensional array for hash maps to a method where each merge worker thread handles distinct subsets of group-by keys. This approach avoids race conditions and eliminates the need for complex conversions. Initially, attempts to segment keys by range did not yield expected improvements, prompting a shift in strategy. Memory management issues arose during development, particularly with the DB::Arena technique, which is designed for efficient memory allocation in short-lived tasks. A misapplication of this system led to race conditions, resulting in segmentation faults and incorrect query results.
Further challenges included performance degradation when using trivial aggregation functions like count or sum. The overhead of parallel merging was unjustifiable for simple queries, leading to a decision to disable the optimization in those cases. A solution was found to extract minimum and maximum indices before the parallel merge, which streamlined the process and limited the range of iteration. The article provides a detailed view of the technical hurdles faced and the learning curve involved in optimizing aggregation in ClickHouse.
Questions about this article
No questions yet.