Skip to content

[fix](nereids) scale group-by-less global agg cost by BE number when partitioned - #66354

Open
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/disinct-no-group-by
Open

[fix](nereids) scale group-by-less global agg cost by BE number when partitioned#66354
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/disinct-no-group-by

Conversation

@foxtail463

Copy link
Copy Markdown
Contributor

Problem Summary:
A scalar count(distinct) runs its global aggregate hash-shuffled by the distinct
key (non-empty partitionExpressions), i.e. in parallel across BEs. But the cost
model set the parallelism factor from group-by keys alone, so with no group by it
used factor = 1, costing the aggregate as single-point. This overestimated the
parallel plan and made CBO reject it.

Solution:
Treat a global aggregate as partitioned when it has group by keys or non-empty
partitionExpressions, and scale its cost by beNumber accordingly.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

@foxtail463
foxtail463 force-pushed the fix/disinct-no-group-by branch from dd90308 to 19f7f33 Compare August 1, 2026 17:21
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28674 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19f7f3395e67d37f32cbd95b0b1b3e8050b1f50f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17584	3905	3924	3905
q2	2235	316	199	199
q3	10193	1328	787	787
q4	4686	466	338	338
q5	7600	842	558	558
q6	187	166	137	137
q7	758	815	602	602
q8	9780	1536	1559	1536
q9	6070	4079	3986	3986
q10	7428	1630	1366	1366
q11	566	354	327	327
q12	751	575	456	456
q13	18138	3209	2733	2733
q14	266	249	228	228
q15	q16	731	724	667	667
q17	1116	982	1046	982
q18	6922	5736	5498	5498
q19	1664	1194	1023	1023
q20	801	714	562	562
q21	5907	2540	2483	2483
q22	430	360	301	301
Total cold run time: 103813 ms
Total hot run time: 28674 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4249	4133	4141	4133
q2	283	306	205	205
q3	4514	4884	4357	4357
q4	2152	2253	1386	1386
q5	4187	4093	4073	4073
q6	219	172	128	128
q7	1700	1882	1684	1684
q8	2332	2059	2022	2022
q9	7243	7202	7145	7145
q10	4278	4328	3836	3836
q11	546	402	358	358
q12	698	720	516	516
q13	3280	3515	2923	2923
q14	295	307	280	280
q15	q16	693	712	625	625
q17	1342	1255	1265	1255
q18	7876	7325	7174	7174
q19	1044	1046	1091	1046
q20	2199	2199	1911	1911
q21	5209	4538	4442	4442
q22	526	439	405	405
Total cold run time: 54865 ms
Total hot run time: 49904 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 168752 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19f7f3395e67d37f32cbd95b0b1b3e8050b1f50f, data reload: false

query5	4360	600	469	469
query6	464	217	202	202
query7	4872	577	340	340
query8	334	177	167	167
query9	8777	3982	3975	3975
query10	475	357	306	306
query11	5839	2204	2005	2005
query12	149	98	96	96
query13	1322	616	455	455
query14	6064	4637	4330	4330
query14_1	3774	3784	3731	3731
query15	212	214	176	176
query16	3025	506	461	461
query17	1102	691	559	559
query18	2209	474	372	372
query19	207	182	141	141
query20	104	99	96	96
query21	232	154	134	134
query22	13134	13057	12811	12811
query23	17167	16335	15988	15988
query23_1	16106	16013	16026	16013
query24	7496	1702	1204	1204
query24_1	1229	1280	1224	1224
query25	535	420	341	341
query26	1331	344	198	198
query27	2575	563	384	384
query28	4990	2034	1994	1994
query29	1043	597	475	475
query30	337	267	225	225
query31	1116	1066	961	961
query32	104	59	58	58
query33	494	303	248	248
query34	1169	1112	643	643
query35	726	733	621	621
query36	780	774	720	720
query37	147	104	95	95
query38	1840	1644	1604	1604
query39	829	814	801	801
query39_1	785	773	785	773
query40	248	153	140	140
query41	70	63	63	63
query42	92	90	91	90
query43	313	321	276	276
query44	1395	774	751	751
query45	185	175	165	165
query46	1023	1206	706	706
query47	1580	1567	1455	1455
query48	435	402	311	311
query49	596	400	320	320
query50	1084	427	349	349
query51	10849	10547	10503	10503
query52	84	86	72	72
query53	250	276	201	201
query54	294	233	210	210
query55	73	69	65	65
query56	291	299	285	285
query57	1030	1013	922	922
query58	291	264	250	250
query59	1541	1565	1342	1342
query60	309	269	241	241
query61	150	142	145	142
query62	398	320	259	259
query63	226	190	203	190
query64	2466	974	803	803
query65	3889	3817	3833	3817
query66	1731	443	366	366
query67	28207	28157	27998	27998
query68	3163	1436	1028	1028
query69	399	331	257	257
query70	898	763	790	763
query71	385	351	319	319
query72	3162	2799	2560	2560
query73	840	771	433	433
query74	4597	4484	4273	4273
query75	2356	2339	2002	2002
query76	1876	1166	753	753
query77	333	367	269	269
query78	11026	11242	10480	10480
query79	1355	1052	743	743
query80	1232	560	447	447
query81	520	329	291	291
query82	656	147	109	109
query83	359	326	298	298
query84	289	154	130	130
query85	969	614	526	526
query86	401	237	216	216
query87	1790	1792	1685	1685
query88	3708	2798	2779	2779
query89	398	321	284	284
query90	1937	195	198	195
query91	200	186	165	165
query92	59	61	56	56
query93	1619	1579	987	987
query94	713	357	311	311
query95	818	529	547	529
query96	1080	768	361	361
query97	2442	2485	2314	2314
query98	200	194	195	194
query99	719	720	619	619
Total cold run time: 257481 ms
Total hot run time: 168752 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19f7f3395e67d37f32cbd95b0b1b3e8050b1f50f, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.05	0.05
query3	0.25	0.13	0.14
query4	1.60	0.13	0.14
query5	0.23	0.22	0.22
query6	1.16	0.83	0.81
query7	0.03	0.01	0.00
query8	0.06	0.03	0.03
query9	0.37	0.31	0.30
query10	0.54	0.54	0.54
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.47	0.46	0.46
query14	0.98	0.99	0.98
query15	0.60	0.58	0.58
query16	0.31	0.32	0.32
query17	1.07	1.10	1.10
query18	0.20	0.19	0.20
query19	1.98	1.96	1.96
query20	0.02	0.01	0.01
query21	15.44	0.20	0.15
query22	4.96	0.05	0.04
query23	16.14	0.30	0.13
query24	2.92	0.39	0.30
query25	0.12	0.05	0.04
query26	0.75	0.21	0.14
query27	0.04	0.04	0.04
query28	3.51	0.73	0.34
query29	12.47	4.03	3.28
query30	0.28	0.16	0.15
query31	2.77	0.57	0.32
query32	3.22	0.59	0.48
query33	3.20	3.20	3.17
query34	15.65	3.94	3.26
query35	3.24	3.22	3.24
query36	0.54	0.44	0.40
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.08 s
Total hot run time: 23.9 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants