[Model] Complete port of Transformers v5 heterogeneous config fix - #1086
Draft
amd-callumm wants to merge 1 commit into
Draft
[Model] Complete port of Transformers v5 heterogeneous config fix#1086amd-callumm wants to merge 1 commit into
amd-callumm wants to merge 1 commit into
Conversation
amd-callumm
force-pushed
the
callumm/gemma4-transformers-v5-complete
branch
5 times, most recently
from
August 12, 2026 20:13
b4716b6 to
3280b52
Compare
Fixes Gemma-4 models failing to load with Transformers v5 due to
`AmbiguousGlobalPerLayerAttributeError` when accessing per-layer
attributes like `num_key_value_heads` and `head_dim`.
Transformers v5 introduced heterogeneous configs where certain attributes
vary per layer. Accessing these directly on the config object raises an
exception. The fix involves:
1. Infrastructure changes to ModelArchitectureConfig:
- Add per_layer_overrides field
- Add __getitem__ for layer-specific views
- Add from_layers() classmethod to merge per-layer configs
2. Model arch config convertor updates:
- Add get_per_layer_hf_configs() to generate layer configs
- Refactor convert() to handle per-layer configs via from_layers()
- Override for Gemma4 to handle both Transformers v4 (layer_types)
and v5 (is_heterogeneous) formats
- Add get_total_num_kv_heads() override to return max across layers
3. Helper function for layer config access:
- gemma4_layer_config() handles both v4 and v5 formats
- Returns per-layer config for given layer_idx
4. Config comparison fix:
- Use object identity (is not) instead of equality (!=) in
model.py to avoid triggering __eq__ which accesses all attributes
Based on upstream vllm-project/vllm PR vllm-project#49797 but adapted for ROCm fork.
AIESW-41388
Signed-off-by: Callum McIntyre <callum.mcintyre@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: <callumm@amd.com>
amd-callumm
force-pushed
the
callumm/gemma4-transformers-v5-complete
branch
from
August 12, 2026 22:00
63e9d68 to
104577f
Compare
amd-callumm
marked this pull request as ready for review
August 12, 2026 23:18
Author
|
The pre-commit lint complaints in .buildkite/ and tests/ appear unrelated to these changes. |
amd-callumm
marked this pull request as draft
August 14, 2026 21:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Port remaining infrastructure changes from upstream PR vllm-project#49797 (commit 70b84f0) to fix Gemma-4 models with Transformers v5. Some changes are adapted/added for the ROCm fork/gfx11 branch.
The partial port (c8e2f26) only covered model files, but the error occurs earlier in model_arch_config_convertor.py during config init.
Changes:
Infrastructure changes to ModelArchitectureConfig:
Model arch config convertor updates:
and v5 (is_heterogeneous) formats
Helper function for layer config access:
Config comparison fix:
model.py to avoid triggering eq which accesses all attributes
This enables Gemma-4 to load with both Transformers v4 and v5.
Test Plan
Test Result
All Gemma-3 nightly regression tests are passing both without profiling and with ttft-roofline profiling using the vLLM wheel built against this PR (previously hit
AmbiguousGlobalPerLayerAttributeErroron many tests)