Skip to content

[Model] Complete port of Transformers v5 heterogeneous config fix - #1086

Draft
amd-callumm wants to merge 1 commit into
gfx11from
callumm/gemma4-transformers-v5-complete
Draft

[Model] Complete port of Transformers v5 heterogeneous config fix#1086
amd-callumm wants to merge 1 commit into
gfx11from
callumm/gemma4-transformers-v5-complete

Conversation

@amd-callumm

@amd-callumm amd-callumm commented Aug 11, 2026

Copy link
Copy Markdown

Purpose

Port remaining infrastructure changes from upstream PR vllm-project#49797 (commit 70b84f0) to fix Gemma-4 models with Transformers v5. Some changes are adapted/added for the ROCm fork/gfx11 branch.

The partial port (c8e2f26) only covered model files, but the error occurs earlier in model_arch_config_convertor.py during config init.

Changes:

  1. Infrastructure changes to ModelArchitectureConfig:

    • Add per_layer_overrides field
    • Add getitem for layer-specific views
    • Add from_layers() classmethod to merge per-layer configs
  2. Model arch config convertor updates:

    • Add get_per_layer_hf_configs() to generate layer configs
    • Refactor convert() to handle per-layer configs via from_layers()
    • Override for Gemma4 to handle both Transformers v4 (layer_types)
      and v5 (is_heterogeneous) formats
    • Add get_total_num_kv_heads() override to return max across layers
  3. Helper function for layer config access:

    • gemma4_layer_config() handles both v4 and v5 formats
    • Returns per-layer config for given layer_idx
  4. Config comparison fix:

    • Use object identity (is not) instead of equality (!=) in
      model.py to avoid triggering eq which accesses all attributes

This enables Gemma-4 to load with both Transformers v4 and v5.

Test Plan

  • Wait for the "Build RDNA3/3.5 prototype wheels" job to complete
  • Run rocm-scripts nightly regression workflow with all Gemma-4 vLLM tests, using the new vLLM wheel to make sure it passes
  • Repeat with ttft-roofline profiling enabled

Test Result

All Gemma-3 nightly regression tests are passing both without profiling and with ttft-roofline profiling using the vLLM wheel built against this PR (previously hit AmbiguousGlobalPerLayerAttributeError on many tests)

@amd-callumm
amd-callumm force-pushed the callumm/gemma4-transformers-v5-complete branch 5 times, most recently from b4716b6 to 3280b52 Compare August 12, 2026 20:13
Fixes Gemma-4 models failing to load with Transformers v5 due to
`AmbiguousGlobalPerLayerAttributeError` when accessing per-layer
attributes like `num_key_value_heads` and `head_dim`.

Transformers v5 introduced heterogeneous configs where certain attributes
vary per layer. Accessing these directly on the config object raises an
exception. The fix involves:

1. Infrastructure changes to ModelArchitectureConfig:
   - Add per_layer_overrides field
   - Add __getitem__ for layer-specific views
   - Add from_layers() classmethod to merge per-layer configs

2. Model arch config convertor updates:
   - Add get_per_layer_hf_configs() to generate layer configs
   - Refactor convert() to handle per-layer configs via from_layers()
   - Override for Gemma4 to handle both Transformers v4 (layer_types)
     and v5 (is_heterogeneous) formats
   - Add get_total_num_kv_heads() override to return max across layers

3. Helper function for layer config access:
   - gemma4_layer_config() handles both v4 and v5 formats
   - Returns per-layer config for given layer_idx

4. Config comparison fix:
   - Use object identity (is not) instead of equality (!=) in
     model.py to avoid triggering __eq__ which accesses all attributes

Based on upstream vllm-project/vllm PR vllm-project#49797 but adapted for ROCm fork.

AIESW-41388

Signed-off-by: Callum McIntyre <callum.mcintyre@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>

Signed-off-by:  <callumm@amd.com>
@amd-callumm
amd-callumm force-pushed the callumm/gemma4-transformers-v5-complete branch from 63e9d68 to 104577f Compare August 12, 2026 22:00
@amd-callumm
amd-callumm marked this pull request as ready for review August 12, 2026 23:18
@amd-callumm

Copy link
Copy Markdown
Author

The pre-commit lint complaints in .buildkite/ and tests/ appear unrelated to these changes.

@amd-callumm
amd-callumm marked this pull request as draft August 14, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant