Skip to content

[Feat] Add Qwen3.5 SFT example - #1985

Open
jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:qwen35_example
Open

[Feat] Add Qwen3.5 SFT example#1985
jayhenry wants to merge 1 commit into
InternLM:mainfrom
jayhenry:qwen35_example

Conversation

@jayhenry

Copy link
Copy Markdown
Collaborator

Summary

  • add an environment-driven Qwen3.5 SFT example for Alpaca and multimodal datasets
  • default to 128K sequence length, SP=4, and soft packing without hard-coded storage paths
  • add a single-node 8-GPU launcher and disable deterministic backward when FA3 is enabled
  • support OpenAI-style video_url entries in Qwen3.5 multimodal tokenization
  • mark empty text samples as damaged cache entries so JsonlDataset can filter them

Validation

  • pre-commit hooks: passed (Ruff, formatting, Mypy, Pydantic checks)
  • targeted tests: 3 passed
  • config construction: verified both Alpaca and multimodal branches
  • 8x H200 training:
    • 128K, SP=4 multimodal: completed one optimizer step without OOM
    • 16K multimodal: completed 20 steps without OOM or NaN/Inf
    • 16K Alpaca: completed 20 steps without OOM or NaN/Inf

Add text and multimodal Qwen3.5 SFT configuration and an 8-GPU launcher. Support video_url samples and ignore empty text samples discovered during end-to-end training validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant