Repository navigation
Conversation
weich97
requested review from
suteng,
yangzhenzhang02 and
yaoyifanx
as code owners
October 10, 2026 11:34
Contributor
Author
|
/check-pr |
Contributor
Author
|
/retest |
|
|
Add .agent/skills/precision: collect a precision dump (MindStudio msprobe), compare a candidate dump against a baseline to find the first module / operator where precision diverges, and localize nan / loss-spike / non-convergence problems. The acceptance bar is rules/precision-acceptance.md; this skill produces the evidence that rule judges. The scripts are written against a real msprobe run (CANN 9.0.0 / torch_npu 2.9.0.post2 / Ascend 910) rather than an assumed interface: - Dump collection must instrument in process. msprobe reads exactly one MSPROBE_* variable (MSPROBE_LOG_LEVEL) and has no config-path variable, so exporting one and launching a command instruments nothing and leaves an empty dump - which then compares as "no divergence". precision_hook.py is the in-process end that reads the PRECISION_DUMP_* window collect_dump.py exports, mirroring prof_hook / mem_hook. - PrecisionDebugger is a process-wide singleton, so baseline and candidate are separate launches; start() raises on a second call instead of silently producing one dump and one empty directory. - The config schema nests per-task options under a block named after the task, and carries rank; a top-level scope is not part of the schema. - step is a list of steps to dump, not a range: [0, 2] dumps steps 0 and 2 and skips step 1, so a range is expanded to explicit integers. - compare takes -tp / -gp / -o on a step directory; there is no -f framework flag and no -np / -bp, and a dump root is rejected by its path check. - The metric columns differ by dump task, and MaxRelativeErr is a percent string in statistics mode against a bare ratio in tensor mode. Parsing the percent form with a plain float() raises, and treating that as "not comparable" skips every diverging row - reading out as no divergence on a run that diverged. - Judgement is bitwise by default so the row reported is the first cause. A band is opt-in: on a measured single-layer perturbation the perturbed weight sits inside a 0.99 / 0.02 band while its propagated output sits outside, so a banded scan reports the symptom and drops the cause. The acceptance rule's 0.02 is a 1000-step loss-trajectory bound and is not reused as a per-tensor tolerance. - A non-finite metric is reported under any band; msprobe's own Result column is carried for context and never gated on, having been observed as pass on a row whose relative error is four digits. collect_dump.py verifies each configured step produced a step directory with a dump.json, so a run that exits 0 having dumped nothing fails rather than feeding an empty comparison. tests/ut/tools/test_precision_dump.py: 42 cases, CPU only, fixtures carrying the real column names and both relative-error formats.
weich97
force-pushed
the
feat/skill-precision
branch
from
October 12, 2026 08:22
87cceec to
325877e
Compare
Contributor
Author
|
/check-pr |
Contributor
Author
|
/retest |
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paired: GitHub #1084 ↔ GitCode !1731
这次改了什么
本 PR 原先提交的是 precision skill 的骨架。在真机上实跑 msprobe 后发现它
假设的接口大部分不存在：采集靠一个 msprobe 根本不读的环境变量、配置 schema
的字段位置错、
step解析会漏步、compare的命令行旗标全不对、比对表的列名与实际不符。最严重的一条是静默假通过——在确实发散的真实比对表上，骨架的
判定逻辑逐行跳过、输出「无差异」并返回 0。
骨架未合入，故原地重做并强推覆盖（原骨架提交
87cceec2备份在本地）。现在接口、字段、列名、判据全部对着实采数据写，并带自校验。
实采验证
msprobe 随 CANN 9.0.0 / torch_npu 2.9.0.post2 安装，Ascend 910 单卡。
被测场景：3 层
Linear(512,512,bias=False)bf16，固定种子，lr=0（权重不动），三次独立进程：
即 A-vs-B 必须判无差异、A-vs-C 必须定位到 layer 1 的权重。
骨架的八条缺陷（逐条实机坐实）
MSPROBE_CONFIG_PATH是 msprobe 读取的公共环境钩子MSPROBE_LOG_LEVEL。插桩只在进程内构造PrecisionDebugger时发生，导出变量再拉起命令采集不会发生PrecisionDebugger是进程级单例，第二次构造是静默空操作，第二份 dump 目录全空scopetask/dump_path/rank/step/level+ 按 task 命名的嵌套块；顶层scope被忽略，且骨架漏了rank（多卡必需）--step 0-2→ 配置写[0, 2]step0/step2，静默跳过 step1msprobe -f pytorch compare -np … -bp …rc=2usage error。真实为msprobe compare -tp … -gp … -o …，无-f、无-np/-bp-tp可指 dump 根目录FileCheckException），必须指到 step 目录Cosine列task: tensor才有；而骨架的采集默认写task: statistics，该模式下每行都找不到MaxRelativeErr可直接float()'1.1049723756906076%'），tensor 模式才是裸比值第 7+8 两条叠加的后果就是静默假通过：statistics 表里
Cosine不存在、MaxRelativeErr解析抛ValueError被当成「该行不可比较」，于是每一行都被跳过。用骨架代码跑真实表：
#7 linear.1.forward.**output**.0第九条：band 会盖住首因
tensor A-vs-C 里，真正被扰动的那一行与它传播出去的那一行落在 band 两侧：
MaxRelativeErrCosinelinear.1.forward.input.1linear.1.forward.output.0flowchart LR L0["linear.0<br/>Max diff = 0"] --> W1 W1["linear.1.input.1<br/>权重 ×1.01<br/>rel 0.0133 / cos 0.999998"]:::cause --> O1 O1["linear.1.output.0<br/>rel 8.83 / cos 0.999994"]:::sym --> I2 I2["linear.2.input.0<br/>rel 8.83"]:::sym classDef cause fill:#fde2e1,stroke:#a5322e,stroke-width:2px classDef sym fill:#fff6e0,stroke:#b07d2b即「噪声带大到能忽略噪声，就大到能盖住首因」。这与本 skill 自己的硬规则
「报首因不报末端症状」直接冲突，因此判据改为默认逐位（任何非零差异
即发散），band 退为
--max-rel/--cosine-min显式选项。第十条：阈值出处是编的
骨架注释称阈值「与
rules/precision-acceptance.md对齐」。核对该规则：它写的是1000 步 loss 轨迹的平均绝对误差 ≤0.02 或平均相对误差 ≤2%，是端到端
验收口径；规则通篇不提余弦。所以
_DEFAULT_MAX_REL = 0.02把 loss 轨迹的平均绝对误差挪用成单算子张量的最大相对误差，两者不是一回事；
_DEFAULT_COSINE = 0.99在规则里没有任何出处。已据实改写：规则是端到端验收口径，不作逐张量容差；bf16 并列噪声按该规则的
ulp 口径判。
另两条附带发现
Result列不是判据：实测在MaxRelativeErr=9368.47的行上仍写
'pass'。现在报告里带出来供参考，绝不拿它当门槛。（按配置值命名目录很容易踩到）。
collect_dump.py提前拦并给出可操作提示。修后验证（同一批真实比对表）
linear.1.forward.input.1#5 linear.1.forward.input.1（Max diff=4.88e-4）linear.1.forward.input.1#5 linear.1.forward.input.1（MaxAbsErr=4.88e-4）两种 dump task 指向同一行首因，且与注入扰动的那一层逐位吻合
（权重 l2norm 13.06 × 1% ≈ bf16 粒度下 4.88e-4）。
step语义也在各自独立进程里逐形式验过（单例会混淆同进程内的第二次采集，这正是上一轮我自己测错的原因）：
[0, 1, 2]step0, step1, step2[0, 2]← 骨架对--step 0-2的产物step0, step2["0-2"]step0, step1, step2[]脚本把区间展开成显式整数列表，这样写出的配置可被逐条校验；
collect_dump.py校验每个配置步是否产出了含dump.json的 step 目录，run 退出码 0 但没采到东西一律判失败——空 dump 在比对侧正是被读成「无差异」。
代码结构
采集拆成两端，与已过门禁的
profiling（#1085）/memory-analysis（#1103）同形：scripts/collect_dump.pyPRECISION_DUMP_*、拉起 run、逐 step 校验scripts/precision_hook.py（新增）start()/capture()/step()；对重复start()直接抛错scripts/diff_dump.py骨架导出的
MEM/PROF式死接口问题在此一并消除：PRECISION_DUMP_*现在有消费者，采集侧与运行侧共用一份窗口。
SKILL.md 据实收敛
references/align-flow.md按实机重写（真 schema、真 CLI、两种列集、两种相对误差格式、单例约束、路径字符校验）。硬规则新增三条：默认逐位判据、
「没采到不等于通过」、「不拿 msprobe 的
Result列当判据」；并写明验收规则的0.02 是 1000 步 loss 口径、不得当逐张量容差。
测试
tests/ut/tools/test_precision_dump.py，42 条，CPU only。fixture 复刻真实列名与两种相对误差格式。覆盖：区间展开与枚举的区别、启动侧与运行侧对同一
spec 展开一致、真 schema 的嵌套/
rank/无顶层scope、路径字符校验、百分号解析、按
Cosine判表模式、statistics 表能查出发散（即假通过的回归）、两种task 指向同一首因、
unsupported行不算发散、nan 在任何 band 下都报、band 下报出的是下游症状（首因 order 更小）、
Result列只带出不判据、step目录解析的五种情形、逐步校验的缺步/未终结/全齐三种状态。
pytest tests/ut/tools42 条全过；pylint 干净（仅剩C9004 sys-path-mutation，与已过门禁的 #1072 / #1085 / #1103 同形），lizard 零超限（最大 CCN 10），
check_agents_catalog通过。