[Fix](pyudf) clear stale UDAF state cache on drop

[linrrzqqq]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Fix Python UDAF stale cache reuse after dropping and recreating an inline UDAF with the same name/signature.

The Python server previously keyed UDAF state managers by function name and argument types, so a recreated inline UDAF could reuse the old loaded Python class. This fix includes the FE function id in the Python UDAF metadata/cache key and clears UDAF state manager cache during DROP FUNCTION cleanup.

set enable_sql_cache = 0;
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
drop database if exists db001;
create database db001;
use db001;

-- 0. Prepare test data
DROP TABLE IF EXISTS t_udaf_cache_bug_test;
CREATE TABLE t_udaf_cache_bug_test (
    id INT,
    val INT
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num"="1");
INSERT INTO t_udaf_cache_bug_test VALUES (1, 10), (2, 20), (3, 30);
-- At this moment, the total of the entire table val is 60.

-- 1. Create V1 version of UDAF (Logic: Accumulate and multiply by 10)
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
select sleep(10);
CREATE AGGREGATE FUNCTION py_udaf_bug_repro(INT)
RETURNS BIGINT
PROPERTIES (
    "type"="PYTHON_UDF",
    "symbol"="RecreateUDAF",
    "runtime_version"="3.12.11", 
    "always_nullable"="true"
)
AS $$
class RecreateUDAF:
    def __init__(self):
        self.total = 0
    @property
    def aggregate_state(self):
        return self.total
    def accumulate(self, val):
        if val is not None:
            self.total += val
    def merge(self, other):
        self.total += other
    def finish(self):
        return self.total * 10  # V1: 乘以 10
$$;

-- 2. Verify V1 Logic
SELECT py_udaf_bug_repro(val) FROM t_udaf_cache_bug_test;
-- Expected Return: 600 (60 * 10)
-- Actual Return: 600 (Correct)

-- 3. Drop the old function and create a V2 version of the UDAF with the same name (logic: accumulate and multiply by 100)
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
select sleep(10);
select sleep(10);
CREATE AGGREGATE FUNCTION py_udaf_bug_repro(INT)
RETURNS BIGINT
PROPERTIES (
    "type"="PYTHON_UDF",
    "symbol"="RecreateUDAF",
    "runtime_version"="3.12.11",
    "always_nullable"="true"
)
AS $$
class RecreateUDAF:
    def __init__(self):
        self.total = 0
    @property
    def aggregate_state(self):
        return self.total
    def accumulate(self, val):
        if val is not None:
            self.total += val
    def merge(self, other):
        self.total += other
    def finish(self):
        return self.total * 100  # V2: Logic modified to multiply by 100
$$;

-- 4. Verify V2 Logic
SELECT py_udaf_bug_repro(val) FROM t_udaf_cache_bug_test;
-- Expected Return: 6000 (60 * 100)
-- Actual Return: 600  ([Bug occurs] Still outputs the old cached 600)

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[linrrzqqq]

run buildall

[linrrzqqq]

/review

[github-actions]

I found two blocking issues in the new Python UDAF drop cleanup path.

Critical checkpoint conclusions:

• Goal/test: The PR addresses stale inline UDAF class reuse after DROP/CREATE with the same signature, and adds a targeted regression test. The function-id cache key covers the recreate case, but the cleanup path introduces failure and lifecycle risks.
• Scope: The change is mostly focused, but the cleanup action should preserve existing best-effort cache cleanup semantics and in-flight query lifecycles.
• Concurrency: The new DROP cleanup can run concurrently with active Python Flight UDAF exchanges. Current code can clear states still needed by those exchanges.
• Lifecycle: Function-id keys avoid reuse by new functions, but removing/clearing old managers needs reference-safe lifecycle handling.
• Compatibility/config: No new config or storage/protocol incompatibility identified for the BE/Python process pair built from this PR.
• Parallel paths: Module cache cleanup remains best-effort; the new UDAF cleanup should follow the same non-crashing error handling model.
• Tests: Regression coverage exercises stale reuse, but does not cover cleanup failure or concurrent DROP while an old UDAF query is still running.
• Observability/performance/data correctness: No additional observability requirement found; the main correctness risks are in error propagation and active-state invalidation.
• User focus: No additional user-provided review focus was specified.

[github-actions]

This can crash the BE from a DROP FUNCTION cleanup task. _broadcast_action_to_processes() returns an error whenever an active Python process fails DoAction/Next, and THROW_IF_ERROR converts that into a doris::Exception. The caller is clean_udf_cache_callback(), which is run by TaskWorkerPool via _callback(task) without any catch boundary, so an uncaught exception terminates the worker thread/process instead of reporting a best-effort cache cleanup failure. The existing module-cache cleanup path logs and continues; this should return Status and be handled with a warning (or use the existing cleanup-style WARN_IF_ERROR) rather than throwing out of the task callback.

[github-actions]

Clearing manager.states can invalidate in-flight UDAF queries. The DROP cleanup task is submitted asynchronously after FE removes the function, while an already-started query can still have a Flight exchange using this same UDAFStateManager for the old function id. If this action runs between that query's CREATE/ACCUMULATE and later SERIALIZE/FINALIZE/DESTROY calls, those operations will find their place_id entries removed and fail with KeyError/failed UDAF results. Since adding function_id to the key already prevents a recreated function from reusing the old class, cleanup should detach the manager from udaf_state_managers without clearing a manager that active exchanges may still reference, or add explicit lifecycle/ref-count coordination. Also consider returning the manager from _get_udaf_state_manager() while still under the lock so a concurrent pop cannot occur between lookup and return.

[linrrzqqq]

/review

[github-actions]

Automated review completed for PR head 7bdde75837b919220cc8758638b8bdaf61dd14b7.

No new blocking issues found. I also reviewed the existing inline threads and did not duplicate them; the current head appears to address the previously raised UDAF cleanup exception and in-flight state invalidation concerns by using WARN_IF_ERROR, returning the manager under lock, and detaching managers without clearing active states.

Critical checkpoint conclusions:

• Goal/test: The PR targets stale Python UDAF state/class reuse after DROP/CREATE. The function-id keying plus drop cleanup path matches that goal, with added BE unit tests and regression coverage for inline drop/recreate.
• Scope: The change is small and focused on Python UDF/UDAF metadata, manager cleanup, and related tests.
• Concurrency/lifecycle: Reviewed the async DROP cleanup versus active Flight exchanges. The current implementation avoids clearing in-flight state and prevents lookup/pop races for manager retrieval. No additional concurrency blocker found.
• Configuration: No new configuration items.
• Compatibility/storage: No storage-format or persisted metadata change. The new id field is carried in the internal BE-to-Python descriptor and all reviewed production builders populate it.
• Parallel paths: UDF, UDAF, and UDTF metadata serialization now include the function id; only UDAF manager keys use it, which is appropriate for this bug.
• Error handling: Cleanup failures are best-effort warnings instead of uncaught task-worker exceptions. Status handling in the reviewed changed C++ paths is acceptable.
• Tests: Added BE unit coverage for no-process, failed-action, action broadcast, and JSON id serialization, plus regression coverage for inline UDAF recreation. I did not run tests locally in this review runner.
• Observability: Existing INFO/WARNING logs around cleanup/broadcast are sufficient for this change.
• Transaction/persistence/data writes: Not applicable.
• Performance: Cleanup scans only the per-process UDAF manager registry under a lock; no obvious performance blocker found.
• User focus: No additional user-provided review focus was specified.

[linrrzqqq]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e32186890f425e43b7bdd68a0e28a0509cdeff0f, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17680	4018	3996	3996
q2	q3	10718	881	617	617
q4	4657	462	353	353
q5	7448	1318	1149	1149
q6	195	171	139	139
q7	902	937	750	750
q8	9342	1394	1298	1298
q9	6050	5367	5366	5366
q10	6291	2083	1813	1813
q11	475	274	257	257
q12	697	415	300	300
q13	18212	3352	2721	2721
q14	302	283	262	262
q15	q16	913	879	807	807
q17	968	1049	782	782
q18	6433	5710	5627	5627
q19	1376	1296	1136	1136
q20	530	411	260	260
q21	4645	2349	1925	1925
q22	457	384	330	330
Total cold run time: 98291 ms
Total hot run time: 29888 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4874	4773	4739	4739
q2	q3	4671	4781	4234	4234
q4	2148	2180	1391	1391
q5	5003	4970	5259	4970
q6	205	172	138	138
q7	2076	1831	1648	1648
q8	3316	3078	3073	3073
q9	8416	8599	8561	8561
q10	4536	4516	4254	4254
q11	625	420	419	419
q12	680	754	551	551
q13	3458	3542	2933	2933
q14	312	319	287	287
q15	q16	760	779	766	766
q17	1460	1384	1364	1364
q18	8036	7219	7161	7161
q19	1152	1156	1204	1156
q20	2289	2254	1959	1959
q21	6232	5705	4900	4900
q22	526	484	399	399
Total cold run time: 60775 ms
Total hot run time: 54903 ms

[hello-stephen]

TPC-DS: Total hot run time: 169707 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e32186890f425e43b7bdd68a0e28a0509cdeff0f, data reload: false

query5	4316	660	513	513
query6	336	229	208	208
query7	4240	564	300	300
query8	320	232	217	217
query9	8863	4007	4009	4007
query10	468	346	306	306
query11	5781	2365	2188	2188
query12	186	133	130	130
query13	1342	596	436	436
query14	6414	5366	5064	5064
query14_1	4400	4379	4369	4369
query15	217	208	186	186
query16	1004	456	457	456
query17	1159	771	642	642
query18	2747	499	366	366
query19	233	216	178	178
query20	143	138	132	132
query21	219	142	120	120
query22	13599	13525	13440	13440
query23	17129	16358	16025	16025
query23_1	16201	16125	16206	16125
query24	7453	1788	1355	1355
query24_1	1340	1361	1344	1344
query25	599	520	475	475
query26	1300	317	181	181
query27	2690	597	343	343
query28	4365	1958	1986	1958
query29	1029	650	539	539
query30	300	242	198	198
query31	1138	1065	938	938
query32	89	80	75	75
query33	555	352	307	307
query34	1190	1127	651	651
query35	774	810	682	682
query36	1378	1360	1138	1138
query37	152	106	85	85
query38	3235	3160	3086	3086
query39	931	934	886	886
query39_1	870	857	917	857
query40	234	159	142	142
query41	69	62	60	60
query42	111	115	108	108
query43	330	327	282	282
query44	
query45	216	203	202	202
query46	1126	1202	747	747
query47	2500	2521	2217	2217
query48	402	430	306	306
query49	645	532	424	424
query50	735	295	221	221
query51	4375	4302	4262	4262
query52	107	106	97	97
query53	255	301	211	211
query54	308	291	275	275
query55	96	91	88	88
query56	331	317	305	305
query57	1411	1447	1343	1343
query58	298	276	271	271
query59	1581	1660	1433	1433
query60	344	351	328	328
query61	165	150	152	150
query62	672	632	567	567
query63	241	201	207	201
query64	2358	827	679	679
query65	
query66	1689	520	384	384
query67	30052	30032	29223	29223
query68	
query69	460	337	303	303
query70	1031	1011	970	970
query71	299	273	268	268
query72	2912	2744	2417	2417
query73	880	756	403	403
query74	5081	4900	4751	4751
query75	2772	2669	2354	2354
query76	2333	1134	739	739
query77	420	430	348	348
query78	12988	12955	12484	12484
query79	1481	996	752	752
query80	1385	579	513	513
query81	525	281	243	243
query82	1261	157	118	118
query83	329	282	247	247
query84	274	144	114	114
query85	907	510	446	446
query86	450	351	325	325
query87	3430	3389	3217	3217
query88	3601	2677	2663	2663
query89	452	396	341	341
query90	1944	176	182	176
query91	178	165	135	135
query92	78	82	76	76
query93	980	937	551	551
query94	699	297	315	297
query95	656	380	337	337
query96	1054	783	332	332
query97	2705	2699	2555	2555
query98	236	231	248	231
query99	1155	1148	974	974
Total cold run time: 255203 ms
Total hot run time: 169707 ms

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 70.59% (12/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.79% (27780/37648)
Line Coverage	57.67% (300875/521723)
Region Coverage	54.95% (251028/456811)
Branch Coverage	56.44% (108485/192210)

[linrrzqqq]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29628 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52b51329ac73de8bc1cff1cac56eaf3a6f247668, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17613	3920	3856	3856
q2	q3	10714	883	608	608
q4	4661	467	349	349
q5	7448	1326	1144	1144
q6	184	174	149	149
q7	917	966	750	750
q8	9325	1376	1245	1245
q9	5599	5411	5380	5380
q10	6243	2081	1825	1825
q11	472	260	252	252
q12	636	422	301	301
q13	18117	3275	2728	2728
q14	288	283	263	263
q15	q16	868	857	793	793
q17	965	1036	751	751
q18	6416	5603	5620	5603
q19	1174	1173	1045	1045
q20	503	398	283	283
q21	4854	2434	1968	1968
q22	484	399	335	335
Total cold run time: 97481 ms
Total hot run time: 29628 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4812	4663	4874	4663
q2	q3	4638	4798	4245	4245
q4	2187	2222	1438	1438
q5	4970	5000	5262	5000
q6	201	176	138	138
q7	2154	1827	1621	1621
q8	3349	3150	3106	3106
q9	8459	8485	8480	8480
q10	4479	4510	4247	4247
q11	620	418	401	401
q12	694	743	512	512
q13	3233	3633	2944	2944
q14	297	311	283	283
q15	q16	864	793	693	693
q17	1327	1293	1251	1251
q18	8048	7098	7047	7047
q19	1163	1181	1136	1136
q20	2251	2214	1932	1932
q21	6107	5360	4837	4837
q22	551	510	414	414
Total cold run time: 60404 ms
Total hot run time: 54388 ms

[hello-stephen]

TPC-DS: Total hot run time: 170228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52b51329ac73de8bc1cff1cac56eaf3a6f247668, data reload: false

query5	4348	644	513	513
query6	325	221	202	202
query7	4236	563	298	298
query8	339	224	228	224
query9	8828	4025	3971	3971
query10	445	352	300	300
query11	5831	2427	2208	2208
query12	182	136	129	129
query13	1266	585	427	427
query14	5930	5331	5050	5050
query14_1	4319	4329	4351	4329
query15	211	202	179	179
query16	976	449	436	436
query17	902	748	627	627
query18	2427	482	347	347
query19	213	197	168	168
query20	142	135	135	135
query21	213	137	114	114
query22	13628	13530	13278	13278
query23	17095	16269	16719	16269
query23_1	16410	16445	16406	16406
query24	7546	1851	1414	1414
query24_1	1404	1352	1368	1352
query25	603	529	453	453
query26	1322	313	169	169
query27	2750	615	333	333
query28	4530	1953	1948	1948
query29	1022	650	535	535
query30	304	241	202	202
query31	1103	1057	981	981
query32	87	69	69	69
query33	544	331	286	286
query34	1197	1078	625	625
query35	752	781	679	679
query36	1316	1304	1145	1145
query37	148	101	84	84
query38	3176	3124	3041	3041
query39	922	917	888	888
query39_1	879	878	886	878
query40	233	150	132	132
query41	63	59	58	58
query42	113	105	106	105
query43	319	314	288	288
query44	
query45	213	206	195	195
query46	1100	1238	720	720
query47	2321	2336	2167	2167
query48	397	404	285	285
query49	636	531	420	420
query50	702	282	214	214
query51	4273	4301	4222	4222
query52	105	105	95	95
query53	256	280	203	203
query54	309	277	249	249
query55	91	88	83	83
query56	302	300	303	300
query57	1418	1425	1354	1354
query58	299	277	271	271
query59	1534	1611	1382	1382
query60	356	344	330	330
query61	156	156	159	156
query62	671	626	555	555
query63	244	201	205	201
query64	2463	817	683	683
query65	
query66	1748	518	380	380
query67	30073	29994	29817	29817
query68	
query69	469	357	300	300
query70	1029	1013	897	897
query71	309	274	278	274
query72	2971	2697	2405	2405
query73	827	765	423	423
query74	5051	4929	4722	4722
query75	2781	2662	2339	2339
query76	2257	1121	747	747
query77	407	435	353	353
query78	12955	12940	12316	12316
query79	1463	1024	737	737
query80	695	586	480	480
query81	458	291	236	236
query82	1332	163	138	138
query83	351	279	256	256
query84	310	138	110	110
query85	855	529	434	434
query86	401	334	331	331
query87	3434	3341	3210	3210
query88	3520	2682	2648	2648
query89	439	386	345	345
query90	1904	188	180	180
query91	197	190	159	159
query92	82	103	74	74
query93	958	945	557	557
query94	519	335	285	285
query95	668	372	339	339
query96	1035	768	352	352
query97	2768	2708	2607	2607
query98	240	231	236	231
query99	1149	1136	976	976
Total cold run time: 253067 ms
Total hot run time: 170228 ms

[HappenLee]

/review

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 88.24% (15/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.87% (27811/37651)
Line Coverage	57.76% (301379/521734)
Region Coverage	55.02% (251323/456810)
Branch Coverage	56.55% (108692/192208)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 88.24% (15/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.87% (27811/37651)
Line Coverage	57.76% (301379/521734)
Region Coverage	55.02% (251323/456810)
Branch Coverage	56.55% (108692/192208)

[linrrzqqq]

/review

[github-actions]

I found one additional blocking issue: the newly added BE unit test calls a private PythonServerManager method directly, so the test target will not compile. Existing review threads already cover the async UDAF manager cleanup lifecycle and cleanup-error handling concerns; I did not duplicate those.

Critical checkpoint conclusions:

• Goal/test: The PR targets stale Python UDAF cache reuse after DROP/CREATE and adds regression/unit coverage, but the new unit test has a compile-time access violation.
• Scope: The functional change is focused on Python UDAF metadata/cache keys and drop cleanup; no unrelated broad behavior was necessary for this fix.
• Concurrency/lifecycle: The relevant async DROP cleanup and in-flight UDAF lifecycle risks are already tracked in existing review context; the current distinct finding is test compilation.
• Error handling: The C++ cleanup path now uses warning-style handling rather than throwing from the task callback; no new duplicate comment was added.
• Compatibility/protocol: The new id metadata is produced by BE and consumed by the local Python server; no additional compatibility issue found in this review.
• Tests: Regression coverage was added for drop/recreate, but BE unit coverage must compile before it can validate anything.
• User focus: No additional user-provided review focus was supplied.

[github-actions]

This test will not compile as written. _broadcast_action_to_processes() is declared after private: in PythonServerManager, and PythonServerTest is not a friend and does not use a BE_TEST public wrapper for this method. Please either test through the public clear_udaf_state_cache() path or expose a test-only wrapper under #ifdef BE_TEST instead of calling the private method directly.