NVIDIA · jingyu-ml · Aug 5, 2026
@@ -8,6 +8,7 @@ Changelog
 
 *Quantization*
 
+- Store SVDQuant low-rank residual factors as trainable Hugging Face PEFT adapter parameters on exactly the layers calibrated by SVDQuant. The forward remains ``Q(W_residual)x + B(Ax)``. Full ``mto.save`` / ``mto.restore`` checkpoints and the split ``modelopt_state`` + complete ``state_dict`` flow preserve the adapter topology and A/B values.
 - Add the ``nvfp4_act_headroom`` calibration algorithm for NVFP4 **activation** global scales. Plain ``max`` sets a tensor's global scale from the largest per-block amax seen during calibration, leaving no room above it, so any activation larger than the calibration max saturates. ``nvfp4_act_headroom`` instead anchors the global scale to a low percentile of the per-block amax distribution, ``amax = max(rho * anchor, upper)``, placing the calibrated blocks in the lower part of the FP8 block-scale range and leaving the rest as headroom. ``upper`` is the top of the range the scale commits to representing and defaults to the 99.99th percentile rather than the literal maximum: chasing a lone freak block would drag the global scale up until every other block's FP8 block scale falls below subnormal and flushes to zero, so the rarest blocks are clipped instead. Set ``upper_percentile=100`` to use the literal observed max, which guarantees no calibration data is clipped. The calibrator warns when the per-block range is too wide for ``rho`` to clear any headroom. Tunable via ``anchor_percentile`` (default 1), ``upper_percentile`` (default 99.99) and ``rho`` (default 16384). Applies only to NVFP4 dynamic-block input quantizers. Weight scales are an orthogonal axis, selected by a nested ``weight_scale_algorithm`` (``max`` by default, or ``mse`` / ``local_hessian`` with that algorithm's own options), so one recipe can combine a weight calibration with this activation policy in a single pass. ``SequentialQuantizer`` activation quantizers are not supported and raise. Ships ``modelopt_recipes/general/ptq/nvfp4_act_headroom-kv_fp8_cast.yaml``, which mirrors ``nvfp4_default-kv_fp8_cast`` (dynamic NVFP4 W4A4 plus FP8 KV-cache cast) with only the calibration algorithm swapped, so it exports a standard NVFP4 checkpoint.
 
 *Misc*
@@ -17,6 +18,8 @@ Changelog
 
 **Backward Breaking Changes**
 
+- New SVDQuant checkpoints no longer expose low-rank factors through ``weight_quantizer.svdquant_lora_a/b``; Hugging Face PEFT owns them as trainable ``modelopt_svdquant`` adapter parameters. Code that directly inspected or optimized the quantizer buffers must use the PEFT parameters instead.
+
 **Deprecations**
 
 **Bug Fixes**

@@ -114,6 +114,22 @@ Here is an example of performing QAT:
     # Adjust learning rate and training duration
     train(model, train_loader, optimizer, scheduler, ...)
 
+For SVDQuant, the calibrated low-rank residual is represented by a structural
+Hugging Face PEFT adapter on only the layers to which SVDQuant was applied. The
+adapter A/B factors are trainable immediately after calibration or restoration,
+so they are included by a normally constructed optimizer. Construct distributed
+wrappers and the optimizer after quantization or checkpoint restoration:
+
+.. code-block:: python
+
+    model = mtq.quantize(model, quant_config, forward_loop)
+    model = DistributedDataParallel(model)
+    optimizer = create_optimizer(parameter for parameter in model.parameters() if parameter.requires_grad)
+
+The adapter implements the SVDQuant decomposition
+``Q(W_residual)x + B(Ax)`` and requires the ``nvidia-modelopt[hf]`` optional
+dependencies.
+
 .. tip::
 
     We recommend QAT for 10% of the original training epochs. For LLMs, we find that QAT fine-tuning for even
@@ -126,6 +142,11 @@ The model weights and quantizer states need to saved for future use or to resume
 Please see :ref:`saving and restoring of ModelOpt-modified models <save-restore>` to learn
 how to save and restore the quantized model.
 
+For PEFT-backed SVDQuant, ``mto.save`` / ``mto.restore`` preserve the adapter
+topology and factor values. The split checkpoint flow is also supported: restore
+``mto.modelopt_state(model)`` first, then load the complete
+``model.state_dict()`` before constructing the distributed wrapper and optimizer.
+
 
 Optimal Partial Quantization using ``auto_quantize``
 ===================================================================

@@ -119,10 +119,73 @@ Any `DMDConfig` field can be overridden on the CLI (e.g. `--dmd2.guidance_scale=
 
 Checkpoints land under `checkpoint.checkpoint_dir`. Alongside the student, the recipe
 saves the DMD2 sidecars needed to resume exactly: the fake-score model + optimizer, the
-student EMA (`ema_shadow.pt`), and the DMD iteration counter (`dmd_state.pt`). With
+DMD iteration counter (`dmd_state.pt`), and, when EMA is enabled, the student EMA
+(`ema_shadow.pt`). With
 `restore_from: LATEST` a re-launch auto-resumes from the newest checkpoint; pin a
 specific one with `--checkpoint.restore_from=epoch_0_step_500`.
 
+## Quantization-aware training (QAT)
+
+Continue a full-precision DMD2 run with the **student quantized**, so the few-step model
+stays accurate at FP8/NVFP4. QAT here is **restore-only**: the trainer loads a ModelOpt
+quantizer state (recipe + frozen `amax`) from disk and **never calibrates**. Only the
+student is quantized; the frozen teacher and trainable fake-score stay full precision so
+the distribution-matching gradient is exact, and `amax` stays frozen for the whole run.
+
+QAT is driven by a `dmd2.quant` block — there's no dedicated config file. The cleanest
+way to launch is to **reuse the exact config + overrides of the full-precision run you're
+continuing** and add only the three `dmd2.quant.*` keys (plus a reduced LR), so the QAT
+run is provably identical to the FP run except for quantization and learning rate. The
+CLI parser creates the `dmd2.quant` subtree even when it's absent from the YAML.
+
+| Key | Role |
+| --- | --- |
+| `dmd2.quant.enabled` | Turn QAT on (restore-only student quantization). |
+| `dmd2.quant.quant_state_path` | The `transformer.pt` from step 1 below (recipe + frozen `amax`). |
+| `dmd2.quant.init_weights_from` | FP DMD2 checkpoint to warm-start student / fake-score / optimizers from on the first launch (the run `amax` was calibrated against). |
+
+1. **Calibrate once** with the quantization example to produce the quantizer state
+   (`amax`, no weights) for a trained student checkpoint:
+
+   ```bash
+   python examples/diffusers/quantization/quantize.py \
+       --model qwen-image-dmd2 --format fp8 \
+       --extra-param student_path=<.../epoch_4_step_15999/model/consolidated> \
+       --quantized-torch-ckpt-save-path <.../epoch_4_step_15999/quant>
+   # -> writes <.../epoch_4_step_15999/quant/transformer.pt>
+   ```
+
+2. **Launch QAT** by re-running the FP run's command with a new output dir, a reduced
+   student LR, and the three quant keys appended:
+
+   ```bash
+   torchrun --nproc-per-node=<gpus> \
+       examples/diffusers/fastgen/dmd2_finetune.py \
+       --config examples/diffusers/fastgen/configs/<the FP run's config>.yaml \
+       --checkpoint.checkpoint_dir=<NEW output dir> \
+       <... the FP run's other overrides, unchanged ...> \
+       --optim.learning_rate=<FP lr / 10> --lr_scheduler.min_lr=<FP lr / 10> \
+       --dmd2.quant.enabled=true \
+       --dmd2.quant.quant_state_path=<.../epoch_4_step_15999/quant/transformer.pt> \
+       --dmd2.quant.init_weights_from=<.../epoch_4_step_15999>
+   ```
+
+On the first launch (empty `checkpoint_dir`) the student / fake-score / discriminator /
+optimizers warm-start from `init_weights_from`, then the student is quantized from
+`quant_state_path`. `restore_from: LATEST` auto-resumes the new `checkpoint_dir`
+thereafter. Because QAT is restore-only — amax never recalibrates — the recipe re-applies
+`quant_state_path` on every resume rather than persisting a per-checkpoint copy, so keep
+that file accessible for the whole run (it's the only quantization dependency). The saved
+student weights are clean full precision (`model/consolidated` is a normal
+`QwenImageTransformer2DModel`); re-apply `quant_state_path` to deploy or evaluate the
+quantized QAT student via the quantization example.
+
+> The `quant_state_path` `amax` must have been calibrated against the student in
+> `init_weights_from`, with the same few-step schedule (`dmd2.sample_t_cfg.t_list`) the
+> student trains/infers with. Pass `dmd2.quant.enabled=true` on every resume too (it is
+> what tells the recipe to quantize). Reduce only the student LR by keeping
+> `--dmd2.fake_score_lr` / `--dmd2.discriminator_lr` at the FP value.
+
 ## Inference
 
 After training, sample from the distilled student. The pipeline loads your consolidated
@@ -170,7 +233,7 @@ student).
 | `model` | `mode` | `finetune` — loads the pretrained weights. |
 | `step_scheduler` | `global_batch_size`, `local_batch_size`, `max_steps`, `ckpt_every_steps`, `log_every` | Standard AutoModel scheduling knobs. |
 | `dmd2` | `recipe_path` | Built-in fastgen recipe to hydrate `DMDConfig` from (`general/distillation/dmd2_qwen_image`). |
-| `dmd2` | `pipeline_plugin` | `qwen_image` — selects `QwenImageDMDPipeline` (2×2 patch packing / img_shapes). |
+| `dmd2` | `pipeline_plugin` | `qwen_image` — selects `QwenImageDMDPipeline` (2×2 patch packing / `img_shapes`). |
 | `dmd2` | `student_sample_steps` | Number of student sampling steps (e.g. 4). |
 | `dmd2` | `guidance_scale` | CFG strength on the teacher (`null` disables CFG; requires a negative-prompt embedding when set). |
 | `dmd2` | `gan_loss_weight_gen`, `gan_r1_reg_weight`, `gan_feature_indices`, … | GAN branch (set `gan_loss_weight_gen: 0` to disable). |