Add TurboMind ViT support for InternVL and Qwen VL models - #4719
Open
irexyc wants to merge 20 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds TurboMind-native vision encoder support for Qwen-VL (Qwen2/2.5/3.5 family) and InternVL/InternViT by introducing new unified C++ ViT runtimes/kernels, updating Python bindings/builders, and adjusting the attention RoPE/MRoPE plumbing to consume vision-produced position-id tables.
Changes:
- Replace the old
qwen3_5vitimplementation with a unifiedqwenvitruntime/config/weight tree and consolidated CUDA kernels. - Add a new
internvitTurboMind vision model (weights/configs/kernels) and wire it intoVisionModelfactory + pybind. - Update TurboMind attention RoPE/MRoPE interfaces and LMDeploy Python integration to enable “native vision” serving paths for Qwen2VL and InternVL models.
Reviewed changes
Copilot reviewed 61 out of 61 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/turbomind/python/bind.cpp | Expose new multimodal items/inputs and configs (QwenVit + InternVit) and add NormType binding. |
| src/turbomind/models/vision_model.h | Update documentation to reference QwenVit rather than Qwen3_5Vit. |
| src/turbomind/models/vision_model.cc | Create QwenVit / InternVit instances based on weight type. |
| src/turbomind/models/qwenvit/test_mrope_position_ids.cu | Update standalone MRoPE position-id kernel test to new flat-table layout and add CUDA error checking. |
| src/turbomind/models/qwenvit/qwenvit.h | Add unified Qwen ViT VisionModel interface. |
| src/turbomind/models/qwenvit/qwenvit_weight.h | Add unified Qwen ViT config + weight tree with new feature toggles and NormType. |
| src/turbomind/models/qwenvit/qwenvit_weight.cc | Implement unified Qwen ViT weight prep/verify logic and module registration. |
| src/turbomind/models/qwenvit/qwenvit_kernels.h | Add consolidated CUDA kernel interfaces (QKV prep, mapping, pos-embed, rotary, MRoPE, window ops). |
| src/turbomind/models/qwenvit/qwenvit_kernels.cu | Implement consolidated CUDA kernels for unified Qwen ViT. |
| src/turbomind/models/qwenvit/qwenvit_input.h | Rename/reshape multimodal input types to unified QwenVitItem/Input. |
| src/turbomind/models/qwenvit/qwenvit_block_weight.h | Rename block config/weight to unified QwenVitBlock* and add optional gated MLP child. |
| src/turbomind/models/qwenvit/qwenvit_block_weight.cc | Register unified Qwen ViT block module. |
| src/turbomind/models/qwen3_5vit/qwen3_5vit.h | Remove legacy Qwen3_5Vit header. |
| src/turbomind/models/qwen3_5vit/qwen3_5vit.cc | Remove legacy Qwen3_5Vit runtime implementation. |
| src/turbomind/models/qwen3_5vit/qwen3_5vit_weight.h | Remove legacy Qwen3.5-only vision config/weight definitions. |
| src/turbomind/models/qwen3_5vit/qkv_preprocess.h | Remove legacy QKV preprocess header (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/qkv_preprocess.cu | Remove legacy QKV preprocess kernel implementation (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/mrope_position_ids.h | Remove legacy MRoPE table writer header (moved/reshaped in qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/mrope_position_ids.cu | Remove legacy MRoPE kernel implementation (moved/reshaped in qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fused_embed_merge.h | Remove legacy fused pos-embed merge header (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fused_embed_merge.cu | Remove legacy fused pos-embed merge kernel (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fast_rotary_pos_emb.h | Remove legacy fast rotary header (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fast_rotary_pos_emb.cu | Remove legacy fast rotary kernel (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fast_pos_embed.h | Remove legacy fast pos-embed header (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/fast_pos_embed.cu | Remove legacy fast pos-embed kernel (moved into qwenvit kernels). |
| src/turbomind/models/qwen3_5vit/bias_gelu.h | Remove legacy Qwen3.5 bias+GELU header (replaced by generic kernel). |
| src/turbomind/models/qwen3_5vit/bias_gelu.cu | Remove legacy Qwen3.5 bias+GELU kernel (replaced by generic kernel). |
| src/turbomind/models/llama/unified_attention_layer.h | Refactor MRoPE buffers to new default buffer approach. |
| src/turbomind/models/llama/unified_attention_layer.cc | Update MRoPE tensor borrowing contract to use flat table + offsets, and remove legacy request-input path. |
| src/turbomind/models/llama/llama_rope.h | Update MRoPE rope kernel params to use position_offsets instead of stride. |
| src/turbomind/models/language_model.cc | Produce symm_buf earlier (Prepare) instead of Forward. |
| src/turbomind/models/internvit/internvit.h | Add InternViT VisionModel interface. |
| src/turbomind/models/internvit/internvit.cc | Add InternViT runtime implementation (patchify/blocks/projector). |
| src/turbomind/models/internvit/internvit_weight.h | Add InternViT config + weight tree definitions. |
| src/turbomind/models/internvit/internvit_weight.cc | Implement InternViT weight prep/verify and register module. |
| src/turbomind/models/internvit/internvit_kernels.h | Add InternViT CUDA kernel interfaces. |
| src/turbomind/models/internvit/internvit_kernels.cu | Implement InternViT CUDA kernels (patchify, embeddings, norms, shuffle, etc.). |
| src/turbomind/models/internvit/internvit_input.h | Add InternViT multimodal input structures. |
| src/turbomind/models/internvit/internvit_block_weight.h | Add InternViT block config/weight module declaration. |
| src/turbomind/models/internvit/internvit_block_weight.cc | Implement InternViT block prepare and module registration. |
| src/turbomind/models/CMakeLists.txt | Switch build from qwen3_5vit/* to qwenvit/* + internvit/* and update test target path. |
| src/turbomind/kernels/norm/rms_norm.h | Add overload to apply bias into a separate output tensor. |
| src/turbomind/kernels/norm/rms_norm.cu | Implement out-of-place bias add and refactor bias kernel signature. |
| src/turbomind/kernels/norm/norm.h | Introduce NormType enum. |
| src/turbomind/kernels/norm/CMakeLists.txt | Link rms_norm against core. |
| src/turbomind/kernels/attention/rotary_embedding.h | Rework MRoPE lookup to use position_offsets + token_idx. |
| src/turbomind/kernels/attention/kv_cache_utils_v2.cu | Update RoPE apply calls to pass token_idx separately from absolute timestep. |
| src/turbomind/kernels/attention/attention_universal.h | Update RoPE apply calls to pass token_idx separately from absolute timestep. |
| src/turbomind/kernels/activation.h | Add generic invokeAddBiasActivation API. |
| src/turbomind/kernels/activation.cu | Implement generic add-bias + activation CUDA kernel. |
| lmdeploy/vl/model/qwen2.py | Enable native TurboMind vision path, simplify preprocessing, and remove qwen_vl_utils usage. |
| lmdeploy/vl/model/internvl3_hf.py | Enable native TurboMind vision path and cast MM tensors to configured dtype. |
| lmdeploy/vl/model/internvl.py | Enable native TurboMind vision path and cast MM tensors to configured dtype. |
| lmdeploy/vl/engine.py | Route TurboMind-native vision models through the PyTorch-style wrapping path. |
| lmdeploy/turbomind/supported_models.py | Register HF Qwen2-VL architectures for TurboMind model mapping. |
| lmdeploy/turbomind/models/qwen3_5.py | Update to new QwenVit* bindings/configs and adjust TP handling. |
| lmdeploy/turbomind/models/qwen2_vl.py | Add TurboMind source-model builder for Qwen2-VL / Qwen2.5-VL vision subtree and MM conversion. |
| lmdeploy/turbomind/models/internvl.py | Add TurboMind source-model builder + key mapping for InternVL/InternS1/legacy InternVLChatModel with native InternVit vision. |
| lmdeploy/turbomind/models/init.py | Export Qwen2VLModel TurboMind source model. |
| lmdeploy/serve/processors/multimodal.py | Skip Python vision inference when using TurboMind-native vision models. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
# Conflicts: # lmdeploy/vl/model/qwen3_5.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Please describe the motivation of this PR and the goal you want to achieve through this PR.
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist