[API Compatibility] Change compatibility apis to ChangePrefixMatcher, inject paddle.enable_compat(), fix test environment context pollution -part - #895
Conversation
…to patch_compat
… calls hit the alias The default enable_compat() (level=1) does not alias paddle.*, so prefix-converted calls (torch.X -> paddle.X) would run native and reject torch-style kwargs. level=2 turns on the paddle.* alias; caller-aware dispatch keeps paddle internals native. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…to patch_compat
…to patch_compat
…eset import_transformer.py: - Inject paddle.enable_compat(level=2) via a transform() post-pass so it lands after the docstring, __future__ imports and the import block (fixes a pre-existing SyntaxError where injected imports preceded `from __future__`). - Gate on real torch imports (torch_packages), not paddle_package_list, so a torch-free file importing only os/einops/setuptools no longer gains a needless paddle import or the process-global compat switch. tests: - Rename confest.py -> conftest.py so the autouse compat-reset fixture is loaded by pytest (was dead due to the typo). - Dedupe the disable logic into conftest.disable_paddle_compat(), imported by apibase and called before each torch reference exec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…into patch_compat
…ompat Switch Tensor.allclose and Tensor.sort to ChangePrefixMatcher and keep softmin's softmax call using torch arg names (input/dim), since compat paddle APIs now accept torch-style args and reject paddle-native ones. Update tests to inject enable_compat(level=2); fix torchvision model_apibase proxy leak around the torch reference forward.
| # enable_compat is injected in transform() (needs all imports in the body). | ||
| # Gate on real torch imports, not paddle_package_list: the latter also holds | ||
| # MAY_TORCH packages (os/einops/setuptools) that need no compat switch. | ||
| if self.imports_map[self.file]["torch_packages"]: |
There was a problem hiding this comment.
简化下代码,看有无更简单的写法,是否只需要修改visit_Module就可以?
|
PR冲突了 |
|
注意代码风格简化并贴合已有架构来写,不要让AI随意发挥:
|
| return True | ||
| return False | ||
|
|
||
| def _inject_enable_compat(self): |
There was a problem hiding this comment.
这个使用record_scope不能插入吗?插入这个不需要从0开始重写吧
Bring the latest API compatibility coverage into the branch while preserving the record_scope-based compat injection and test state isolation fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thanks for your contribution! |
Align expected diffs with compat injection and intentional ChangePrefixMatcher behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deduplicate paddle_package_list in ImportTransformer.visit_Module: since imports are now batched into one record_scope call, its cross-call dedup no longer removes duplicates, producing two 'import paddle' lines when a file imports multiple torch-family packages (e.g. torchvision + datasets). Regenerate default-mode goldens for the paddle.enable_compat(level=2) injection and the intentional compat-prefix removals (equal, BatchNorm1d/2d), matching the min-mode baseline update in f7e1c04. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| line_NO += 1 | ||
| import_nodes = [] | ||
| for paddle_package in dict.fromkeys(paddle_package_list): | ||
| import_nodes.extend(ast.parse(f"import {paddle_package}").body) |
There was a problem hiding this comment.
直接在这里判断是否为 paddle_package=='paddle'
如果是,则import_nodes.extend(ast.parse(f"import paddle;paddle.enable_compat(2)").body)
并且该操作仅执行一次(当然执行多次问题也不大,后续会被isort校正)
|
|
||
| has_torch_package = bool(self.imports_map[self.file]["torch_packages"]) | ||
|
|
||
| has_enable_compat = any( |
There was a problem hiding this comment.
这些判断都需要吗?后面也会经过isort、black的格式化处理,移除多余import,这里不要想的太复杂了
| if has_torch_package and not has_enable_compat: | ||
| import_nodes.extend(ast.parse("paddle.enable_compat(level=2)").body) | ||
|
|
||
| import_end = 1 if node.body and ast.get_docstring(node, clean=False) else 0 |
There was a problem hiding this comment.
for循环,self.record_scope((self.root, "body", line_NO), import_nodes) 就行了吧。
原始的node在visit_Import、visit_ImportFrom时就已经被清除,原始node的import内容应该已经被清理掉了。
另外加上log_info,这个是比较重要的调试信息
…a single paddle.enable_compat(level=2) on top of the already-cleaned node, dropping redundant enable_compat/torch-package scans and adding log_info
|
/re-run all-failed |
| "kwargs_change": { | ||
| "input": "x", | ||
| "dim": "axis", | ||
| "input": "input", |
| return "unchange" | ||
|
|
||
| def get_paddle_class_nodes(self, func, args, kwargs): | ||
| if ( |
There was a problem hiding this comment.
这个是什么问题?x.split已经对齐了这里为什么还需要修改
There was a problem hiding this comment.
这是要的,不是为了对齐 Tensor.split 本身,而是避免 PaConvert 将 Python 原生的 str.split 误识别为 torch.Tensor.split
There was a problem hiding this comment.
str.split本身就不会被改动,这里误识别了会有什么问题?
| @@ -40,11 +40,15 @@ echo "Checking code gpu unit test by pytest ..." | |||
| set +e | |||
|
|
|||
| PYTEST_IGNORE=( | |||
There was a problem hiding this comment.
这里分两批跑,第一次跑特定名单,第二次跑其他的
| --ignore=tests/test_set_num_interop_threads.py | ||
| ) | ||
|
|
||
| python -m pytest -v -s -p no:warnings tests/test_set_num_interop_threads.py 2>&1 | tee -a pytest.log |
There was a problem hiding this comment.
这里分两批跑,第一次跑特定名单,第二次跑其他的
不用单独指定某个测试
| python -m pytest -v -s -p no:warnings tests/test_cuda_set_stream.py 2>&1 | tee -a pytest.log | ||
| setstream_exit=${PIPESTATUS[0]} | ||
|
|
||
| python -m pytest -v -s -p no:warnings tests/test_set_num_interop_threads.py 2>&1 | tee -a pytest.log |
| # internal `import torch` / `torch.SymInt` etc. resolve through the | ||
| # paddle proxy and blow up. Disable compat for the torch forward, then | ||
| # restore it so the paddle forward matches how a user runs the output. | ||
| from paddle.compat.proxy import TORCH_PROXY_FINDER |
There was a problem hiding this comment.
torch代码按torch跑
paddle代码按paddle.enable_compat(2)跑,不应该影响到torch代码的运行
这是哪里机制上有问题吗?两者执行应该是互不影响才对
| ) | ||
| assert paddle_code == expect_paddle_code, error_msg | ||
| elif compared_tensor_names: | ||
| disable_paddle_compat() |
There was a problem hiding this comment.
你这里是不是应该从eval执行的地方去修改,适配好torch、paddle交替执行的情况
| ) | ||
| assert paddle_code == expect_paddle_code, error_msg | ||
| elif compared_tensor_names: | ||
| disable_paddle_compat() |
There was a problem hiding this comment.
你这里应该从eval执行的地方去修改,适配torch、paddle交替执行的情况
| self.paddle_api = astor.to_source(func).strip("\n") | ||
| else: | ||
| self.parse_func(func) | ||
| func_str = astor.to_source(func).strip("\n") |
There was a problem hiding this comment.
你为啥不直接修正parse_func呢,修问题根源
| input_list = paddle.unstack(input, axis=0) | ||
| else: | ||
| input_list = paddle.split(input, num_or_sections=world_size, axis=0) | ||
| input_list = paddle.tensor.split( |
| simple_input = np.random.rand(1, 3, 224, 224).astype(np.float32) | ||
|
|
||
| pytorch_output = pytorch_result(torch.tensor(simple_input)) | ||
| with paddle.use_compat_guard(enable=False): |
There was a problem hiding this comment.
这些地方还需要guard吗,目前已经不影响torch的计算了吧
| import os | ||
| import re | ||
| import sys | ||
| from contextlib import nullcontext |
There was a problem hiding this comment.
现在为啥还需要搞这些guard?目前已经不影响torch的计算了吧
|
|
||
| The logic is lazy and best-effort: it only touches a framework that is already in | ||
| ``sys.modules``, so it never forces an import of torch/paddle (which would change | ||
| import ordering) for tests that don't use them. |
| obj.run(pytorch_code, ["result"]) | ||
|
|
||
|
|
||
| def test_case_21(): |
| input_list = paddle.unstack(input, axis=0) | ||
| else: | ||
| input_list = paddle.split(input, num_or_sections=world_size, axis=0) | ||
| input_list = paddle.tensor.split( |
There was a problem hiding this comment.
enable_compat(2)之后,你需要把这个split改成torch的用法,也就是compat.split
|
|
||
| for i in range(len(compared_tensor_names)): | ||
| paddle_ns = {} | ||
| with paddle.use_compat_guard(enable=False): |
There was a problem hiding this comment.
因为转换后的代码第一行就是 paddle.enable_compat(level=2),所以这里先把 proxy 关掉从干净状态开始,compat 由被执行的代码按需开启;退出 guard 之后恢复进入前的全局状态,不影响后续用例
| ["result"], | ||
| check_value=False, | ||
| reason="Pytorch return 'torch.FloatTensor' while Paddle rerurn 'paddle.float32' ", | ||
| reason="Pytorch return tensor type name like 'torch.FloatTensor' while Paddle return dtype name like 'paddle.float32'", |
There was a problem hiding this comment.
这个不是新增的API吗,新增的没做到对齐吗
| @@ -0,0 +1,256 @@ | |||
| # Copyright (c) 2026 PaddlePaddle Authors. All Rights Reserved. | |||
There was a problem hiding this comment.
现在是提出了逻辑到 apibase 里,然后在 tests/test_get_default_device.py,tests/test_set_default_device.py,tests/test_set_grad_enabled.py 里面一些测试 case 拿到 torch 的结果之后就复原
…_compat(level=2) in use
…s that could affect running env
|
/re-run all-failed |
| input_list = paddle.unstack(input, axis=0) | ||
| else: | ||
| input_list = paddle.split(input, num_or_sections=world_size, axis=0) | ||
| input_list = paddle.compat.split(input, world_size, dim=0) |
There was a problem hiding this comment.
enable_compat后,这个可以改成 paddle.split(...) 吧
| a = torch.ones(2, 3, device="cpu") | ||
| # paddle returns paddle.FloatTensor etc, so drop the namespace | ||
| # prefix and compare the remaining segments | ||
| result = a.type(dtype=None, non_blocking=False).split(".", 1)[1] |
| kwargs = {"async": True} | ||
| # paddle returns paddle.FloatTensor etc, so drop the namespace | ||
| # prefix and compare the remaining segments | ||
| result = a.type(**kwargs).split(".", 1)[1] |
| _SENTINEL = object() | ||
|
|
||
|
|
||
| def _snapshot_patches(paddle): |
There was a problem hiding this comment.
- 问题单测单独隔离运行,不修改单测机制
- softmin的配置改成新的配置
…min config - reduce_scatter_tensor: use paddle.split instead of paddle.compat.split - tests/apibase.py: restore the original test mechanism and run the state-polluting tests separately via the CI isolated list - softmin: switch to the new enable_compat-based config - tests/test_Tensor_type.py: compare the converted tensor directly when a dtype is given, without stripping the type-name namespace prefix
|
/re-run all-failed |
1 similar comment
|
/re-run all-failed |
PR Docs
For the 24
torch.*APIs that are aligned throughpaddle.compat.*, switch the converter to the "prefix-only + enable_compat" strategy instead of mapping each topaddle.compat.Xexplicitly:api_mapping.json— change those 24 entries fromChangeAPIMatcher → paddle.compat.Xto{"Matcher": "ChangePrefixMatcher"}. Converted calls now use the correspondingpaddle.*prefix while preserving Torch-style signatures and argument names. For example,torch.sort(x, dim=-1)is converted to:paddle.sort(x, dim=-1). At runtime,enable_compat(level=2)dispatches the externalpaddle.sortcall to its Torch-aligned implementation. Related mappings are adjusted accordingly:torch.nn.Softmax2dusesdiminstead of the native Paddleaxis;torch.nn.functional.softminpreservesinputanddim.transformer/import_transformer.py— for converted files whose mapped packages includepaddle, inject a deduplicatedimport paddleand a singlepaddle.enable_compat(level=2). In default mode, the code is inserted after the existing import block. In min mode, it is inserted after the module docstring and any__future__imports, but before retained Torch imports, ensuring that the compat proxy is enabled before Torch is imported.tests/apibase.py—paddle.enable_compat(level=2)installs process-wide Torch proxy and Paddle namespace dispatch state. Call the shareddisable_paddle_compat()helper immediately before every real PyTorch reference execution, ensuring that the reference always runs against real PyTorch, including when a test invokesrun()multiple times. Cross-test process state is restored separately by the autouse fixture intests/conftest.py.PR APIs
related Paddle PR:
paddle.enable_compat-part Paddle#79391paddle.Tensor.splitis a known failure. It will be fixed inAlso in this PR
The issue of Cross-Test Global State Pollution in
conftest.pyis fixed. Originally PaConvert tests execute the original torch code and converted Paddle code sequentially in the same pytest worker. Some tests modify process-wide state without restoring it, including:paddle.enable_compat(level=2)CPU_NUM;paddle.Tensorandpaddle.nn.*classes.Some generated helpers are defined inside temporary exec namespaces. After those namespaces are cleared, the class-level monkeypatches may remain and reference invalid globals, affecting subsequent tests. Therefore, cleaning up only the compat proxy is insufficient.
This change adds an autouse fixture that snapshots the relevant state before each test and restores it afterward. It disables compat, restores PyTorch/Paddle defaults and environment variables, and reverts modified class attributes. This isolates tests without changing PaConvert’s conversion behavior.
7.30 update
Add five apis to
ChangePrefixMatcherand enhanced the tests:also update Paddle’s public
compatguard to isolate torch and paddle execution, improve CI test isolation.related PR:
Co-authored by Claude and Codex