Skip to content

Qwen3.8-Flash-Next: blocked for ONNX conversion (qwen4_exp architecture not dispatched) #53

New issue
New issue
Closed as not planned
Closed as not planned
Qwen3.8-Flash-Next: blocked for ONNX conversion (qwen4_exp architecture not dispatched)#53
Labels
enhancementNew feature or request

Description

@elbruno
elbruno
opened on Aug 26, 2026

Model

  • Requested: unsloth/Qwen3.8-Flash-Next-GGUF (GGUF repackaging)
  • Canonical HF repo: Qwen/Qwen3.8-Flash-Next (safetensors, config verified)
  • License: Qwen Community License 1.0 (new class for this catalog — revenue/MAU-gated notice requirement + a separate "Model as a Service"/"AI Work Assistant" commercial licensing carve-out)

Summary

Qwen3.8-Flash-Next is described by Qwen as "an experimental preview of the architecture that will underpin Qwen4" — not a Qwen3 variant despite the name. 125B total parameters, 6B activated per token, plus 51B dedicated N-gram-embedding parameters and a 4B Multi-Token-Prediction (MTP) head. It combines several architectural elements that have no precedent in ONNX Runtime GenAI's model builder:

  • Hybrid attention — Gated DeltaNet (linear attention) + Qwen Sparse Attention (QSA, a micro-block sparse attention variant with its own indexer), in a 12 × (3 × linear → 1 × full/QSA) layer pattern.
  • Gated Residual — a 4-branch gated residual stream with per-branch read/write gates and a 320-dim bottleneck (hc_count, hc_lowrank in config).
  • N-gram Embedding — a 20,000,000-entry embedding table indexed by token bigrams/trigrams (ngram_size, ngram_vocab_size_base, heads_per_ngram), not single-token IDs.
  • MoE (512 experts, 10 routed + 1 shared) — additive, not itself the blocker.
  • Multimodal — ships a vision encoder (vision_config, image_token_id, video_token_id).

Verified blocker (live run, not just architecture reading)

config.json (verified 2026-08-26): architectures: ["Qwen4ExpForConditionalGeneration"], model_type: "qwen4_exp".

Ran the actual conversion entrypoint this repo uses, with the repo's pinned onnxruntime-genai==0.15.1 and the newest available transformers==5.14.1, against the real HF repo (no local weights needed — config_only=true only requires config.json):

python -m onnxruntime_genai.models.builder -m Qwen/Qwen3.8-Flash-Next -o ./out -p int4 -e cpu --extra_options config_only=true

Failed with:

ValueError: The checkpoint you are trying to load has model type `qwen4_exp` but Transformers does not
recognize this architecture. This could be because of an issue with the checkpoint, or because your
version of Transformers is out of date.

Traced to CONFIG_MAPPING["qwen4_exp"] → KeyError inside transformers/models/auto/configuration_auto.py, raised from onnxruntime_genai/models/builder.py's get_hf_details() → AutoConfig.from_pretrained(). Same result with trust_remote_code=True — the repo ships no custom modeling_*.py, so there's no remote-code fallback either.

This means the failure happens before onnxruntime-genai's own architecture dispatch table is even reached — the blocker starts one layer up, in transformers itself not recognizing the config.

A GitHub code search of microsoft/onnxruntime-genai (2026-08-26) confirms zero occurrences anywhere in the repo of: qwen4_exp, qwen4, hc_lowrank, indexer_budget, ngram_vocab_size_base. This is not a partially-supported architecture — it has no footprint in the builder at all. The closest existing precedent is Qwen35TextModel/Qwen35MoeTextModel (dispatched only for Qwen3_5ForConditionalGeneration/Qwen3_5MoeForConditionalGeneration), which already implements a linear+full attention hybrid and an MTP self-speculative head (enable_mtp extra option) — but qwen4_exp is a distinct architecture string with no dispatch entry, and the builder does not fall back to a "closest match."

System specs used for this attempt

  • OS: Windows
  • Python 3.14.4
  • transformers 5.14.1 (newest available at investigation time)
  • onnxruntime-genai 0.15.1 (this repo's pinned version)
  • No local model weights were downloaded — the failure occurs at config-loading time, before any weight download would begin, so this required no unusual RAM/disk resources.

What would unblock it

  1. transformers adding a Qwen4Exp/qwen4_exp config + model class (a prerequisite onnxruntime-genai itself depends on via AutoConfig).
  2. onnxruntime-genai adding a Qwen4ExpForConditionalGeneration builder case with three genuinely new op families: a QSA sparse-attention op (indexer + micro-block routing), a gated-residual op (multi-branch read/write gates), and an n-gram embedding lookup op.
  3. Extending the existing enable_mtp/Qwen3.6 self-speculative-head support to this architecture's MTP head.
  4. Legal review of the Qwen Community License 1.0's revenue/MAU disclosure and MaaS-licensing clauses before any elbruno/*-onnx publication were ever to become possible.

Recommended alternatives (documented in docs/blocked-models.md)

  • GGUF via llama.cpp/Unsloth — unsloth/Qwen3.8-Flash-Next-GGUF ships day-0 quantized GGUF. This repo has a costed proposal for a general llama.cpp/GGUF sibling package (docs/plans/gguf-sibling-package-proposal.md); the community LLamaSharp binding is also an option for ad-hoc C# use outside this library.
  • Qwen3-14B-Instruct (14.77B) — ✅ native ONNX, already in this catalog with the Qwen3 chat template.
  • Qwen2.5-32B-Instruct (32B) — ✅ native ONNX, largest dense Qwen model currently supported.

Documentation

Findings documented in docs/blocked-models.md under "Next-Gen Models" — see PR #52.

Activity

  1. elbruno
    added
    enhancementNew feature or request
    on Aug 26, 2026

  2. elbruno commented on Aug 26, 2026

    @elbruno
    elbruno
    on Aug 26, 2026
    OwnerAuthor

    Follow-up: GGUF viability check

    Verified whether GGUF format is a usable fallback since ONNX is blocked.

    Short answer: not yet, in C#.

    • unsloth/Qwen3.8-Flash-Next-GGUF ships day-0 quantized GGUF files (GGUF metadata reports internal architecture tag qwen4exp, ~165GiB total for the listed quant).
    • However, the GGUF repo card itself says to run it you need llama.cpp PR #27742 or Unsloth's standalone Desktop app — not mainline llama.cpp.
    • Verified directly via the GitHub API: PR #27742 ("model: add Qwen3.8-Flash-Next (qwen4exp)") is state: open, merged: false, opened the same day, 21 files changed, still under review (32 comments).
    • This means it does not work today with LLamaSharp or any prebuilt/mainline llama.cpp binary — those won't recognize the qwen4exp architecture until the PR merges and a new release is cut.
    • This repo has no generic GGUF-consuming package yet either (ElBruno.LocalLLMs.BitNet is narrowly scoped to the BitNet ternary architecture, unrelated to this model). The costed-but-unimplemented docs/plans/gguf-sibling-package-proposal.md is the only path that could add support, and would need to pin to PR #27742's branch specifically (same pattern BitNet uses to pin a llama.cpp fork).

    Recommendation: revisit both the ONNX and GGUF paths once ggml-org/llama.cpp#27742 merges. Until then, this model is not usable in ElBruno.LocalLLMs via either format. Updated docs/blocked-models.md accordingly.

  3. elbruno commented on Aug 26, 2026

    @elbruno
    elbruno
    on Aug 26, 2026
    OwnerAuthor

    Status: docs merged + upstream monitor active

    When that PR merges, the monitor will automatically:

    1. Comment here with the merge date and the llama.cpp release/tag that first contains it.
    2. Check whether LLamaSharp has shipped a backend build pinning a llama.cpp commit at-or-after the merge — a bare upstream merge is not sufficient for C# consumability, since LLamaSharp pins specific commits and cuts its own native binaries.
    3. Open a PR updating the GGUF Status subsection and Quick Summary row in docs/blocked-models.md.
    4. Flag docs/plans/gguf-sibling-package-proposal.md as unblocked, if applicable.

    If the PR is closed without merging, the monitor will comment here and stop polling.

    Keeping this issue open as the tracking anchor until Qwen3.8-Flash-Next is either usable in this library or definitively ruled out.

  4. elbruno commented on Aug 26, 2026

    @elbruno
    elbruno
    on Aug 26, 2026
    OwnerAuthor

    Monitor migrated to GitHub Actions — no longer depends on a developer machine.

    The local Copilot CLI schedule has been retired and replaced by a repo-hosted workflow: .github/workflows/monitor-qwen38-flash-next.yml (added in #54).

    Watches: ggml-org/llama.cpp#27742, daily at 09:00 UTC, plus manual workflow_dispatch.

    Upstream state Action
    PR open (current) Silent — run summary only, no comment here
    PR merged Comments here once with merge details, llama.cpp release, LLamaSharp status, and a re-evaluation checklist
    PR closed unmerged Comments here once flagging that a new tracking target is needed

    Design follows the conventions of the retired monitor-gemma4-blocker.yml: config passed via env: (injection-hardened), silent-by-default so it doesn't become a job people mute, and marker-based idempotency so a terminal state is announced exactly once rather than every morning thereafter.

    One deliberate caution baked into it: it will not claim C# usability on merge. LLamaSharp pins a specific llama.cpp commit and ships its own native binaries, so an upstream merge is necessary but not sufficient. The LLamaSharp release date is reported as an explicitly-labelled heuristic, with a checklist item to verify the pinned commit.

    ✅ Verified by a live workflow_dispatch run (32993837795): correctly detected state=open merged=false, took no action, and posted no comment.

    This issue stays open as the canonical tracker until Qwen3.8-Flash-Next is either usable or definitively ruled out.

  5. github-actions commented on Aug 27, 2026

    @github-actions
    github-actionsBot
    on Aug 27, 2026 – with GitHub Actions

    🚀 Upstream llama.cpp support has merged

    ggml-org/llama.cpp#27742 — "model: add Qwen3.8-Flash-Next (qwen4exp)" — was merged at 2026-08-27T19:32:31Z.

    The GGUF path for Qwen3.8-Flash-Next is now potentially viable. ONNX remains blocked separately (qwen4_exp is still not dispatched by onnxruntime-genai).

    Distribution status

    • Latest llama.cpp release v0.3.0 predates the merge (2026-08-25T10:22:58Z) — a newer release is still needed.
    • Latest LLamaSharp release v0.29.0 (2026-08-24T13:34:22Z) predates the merge, so it cannot contain this change. C# use via LLamaSharp is still blocked.

    ⚠️ An upstream merge does not by itself make this model usable from C#.
    LLamaSharp ships its own native binaries pinned to a specific llama.cpp
    commit, so the pinned commit must be at or after this merge.

    Next steps

    • Confirm the llama.cpp release/tag that first contains the merge
    • Confirm the llama.cpp commit pinned by the current LLamaSharp backend
    • Re-evaluate docs/plans/gguf-sibling-package-proposal.md — now potentially unblocked
    • Update the GGUF Status subsection and Quick Summary row in docs/blocked-models.md
    • Decide whether Qwen3.8-Flash-Next enters KnownModels.cs (note: ~165GiB GGUF, and Qwen Community License 1.0 review still applies)
    • Retire this workflow to a manual-only stub once resolved

    Generated by monitor-qwen38-flash-next.

  6. elbruno commented on Aug 28, 2026

    @elbruno
    elbruno
    on Aug 28, 2026
    OwnerAuthor

    Qwen3.8-Flash-Next remains blocked on both supported investigation paths:

    Given that there is no actionable repository-side fix today, the automated daily monitor in .github/workflows/monitor-qwen38-flash-next.yml is being retired to a manual-only stub. That means no more scheduled polling and no more automatic monitor comments on this issue.

    The canonical source of truth for the technical blockers and the "What Would Unblock It" checklist is now docs/blocked-models.md (see the Qwen3.8-Flash-Next subsection).

    This follows the same terminal-state closure pattern already used for Gemma 4 in #39.

  7. elbruno
    closed this as not plannedon Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions