Skip to content

[Exec](colocate) disable colocate group execution in bucket shuffle join#63200

Open
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:pr-63062
Open

[Exec](colocate) disable colocate group execution in bucket shuffle join#63200
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:pr-63062

Conversation

@HappenLee
Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None
Related PR: #63062
Problem Summary:
Disable colocate-style parallelization for bucket shuffle join fragments. When exchange nodes exist, UnassignedScanBucketOlapTableJob now falls back to the base degree-of-parallelism logic instead of using the tablet-based colocate strategy,
which avoids over-parallelizing join fragments. The colocate path is still kept for pure colocate scans without exchange nodes.

Release note

None

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:
    • Yes. Disable colocate-style execution for bucket shuffle join fragments.
    • No.
  • Does this need documentation?
    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 30942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9961ce2ae926d044971c422b2bb8347c119af6e, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17620	3864	3815	3815
q2	q3	10760	1331	799	799
q4	4682	467	343	343
q5	7555	2261	2079	2079
q6	295	171	138	138
q7	935	767	628	628
q8	9424	1709	1603	1603
q9	6717	4914	4915	4914
q10	6416	2142	1805	1805
q11	427	278	244	244
q12	693	423	289	289
q13	18179	3405	2762	2762
q14	262	264	229	229
q15	q16	810	778	709	709
q17	1015	1067	897	897
q18	6883	5589	5615	5589
q19	1163	1303	1030	1030
q20	514	403	275	275
q21	5646	2582	2496	2496
q22	424	355	298	298
Total cold run time: 100420 ms
Total hot run time: 30942 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4180	4110	4096	4096
q2	q3	4497	4929	4401	4401
q4	2091	2196	1380	1380
q5	4399	4255	4274	4255
q6	226	171	129	129
q7	2126	1869	1585	1585
q8	2609	2134	2044	2044
q9	7781	7733	7771	7733
q10	4512	4468	4039	4039
q11	577	400	374	374
q12	831	781	516	516
q13	3318	3560	2995	2995
q14	304	317	297	297
q15	q16	733	752	642	642
q17	1317	1300	1281	1281
q18	8017	7441	7136	7136
q19	1158	1080	1083	1080
q20	2212	2209	1943	1943
q21	5297	4607	4450	4450
q22	521	448	420	420
Total cold run time: 56706 ms
Total hot run time: 50796 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169461 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9961ce2ae926d044971c422b2bb8347c119af6e, data reload: false

query5	4338	696	511	511
query6	343	231	219	219
query7	4258	557	300	300
query8	325	240	218	218
query9	8845	3996	4006	3996
query10	457	348	300	300
query11	5823	2487	2198	2198
query12	196	133	127	127
query13	1275	611	449	449
query14	6204	5389	5093	5093
query14_1	4375	4338	4340	4338
query15	216	211	182	182
query16	1002	489	464	464
query17	1163	770	622	622
query18	2498	497	369	369
query19	258	207	176	176
query20	145	139	133	133
query21	216	163	119	119
query22	13635	13695	13370	13370
query23	17371	16418	15996	15996
query23_1	16220	16176	16223	16176
query24	7460	1810	1318	1318
query24_1	1323	1334	1344	1334
query25	574	520	443	443
query26	1311	351	178	178
query27	2667	587	342	342
query28	4459	1949	1985	1949
query29	1002	661	503	503
query30	300	254	201	201
query31	1151	1087	942	942
query32	95	79	74	74
query33	563	369	298	298
query34	1163	1151	645	645
query35	806	804	713	713
query36	1285	1334	1161	1161
query37	158	107	86	86
query38	3215	3085	3006	3006
query39	919	913	895	895
query39_1	877	861	868	861
query40	240	147	126	126
query41	64	63	62	62
query42	111	107	112	107
query43	335	326	285	285
query44	
query45	210	202	190	190
query46	1068	1226	722	722
query47	2288	2321	2173	2173
query48	401	394	297	297
query49	630	515	381	381
query50	998	345	253	253
query51	4497	4269	4162	4162
query52	107	109	94	94
query53	252	289	213	213
query54	304	282	266	266
query55	92	89	101	89
query56	304	303	295	295
query57	1383	1365	1318	1318
query58	306	270	289	270
query59	1575	1704	1411	1411
query60	323	324	313	313
query61	150	151	156	151
query62	669	644	578	578
query63	253	202	206	202
query64	2455	812	682	682
query65	
query66	1719	517	377	377
query67	29599	30004	29804	29804
query68	
query69	451	342	312	312
query70	1031	967	990	967
query71	318	274	270	270
query72	3095	2754	2407	2407
query73	830	735	430	430
query74	5055	4922	4719	4719
query75	2814	2696	2237	2237
query76	2266	1228	769	769
query77	409	412	337	337
query78	12202	12286	11566	11566
query79	1452	1082	732	732
query80	685	564	457	457
query81	470	287	238	238
query82	1355	163	122	122
query83	341	284	254	254
query84	258	148	112	112
query85	877	540	449	449
query86	399	338	298	298
query87	3476	3355	3241	3241
query88	3597	2607	2632	2607
query89	462	386	340	340
query90	1901	180	182	180
query91	175	178	140	140
query92	80	76	76	76
query93	1494	1369	952	952
query94	543	365	326	326
query95	687	477	369	369
query96	1003	748	362	362
query97	2723	2708	2581	2581
query98	242	233	230	230
query99	1127	1117	1020	1020
Total cold run time: 254009 ms
Total hot run time: 169461 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants