Alibaba Qwen Releases Qwen3.8-Flash-Next, 125B MoE With 6B Active Parameters
This digest was compiled by AI from multiple sources — links to the originals are below.
Alibaba's Qwen team has released Qwen3.8-Flash-Next, an open-weight multimodal mixture-of-experts model with 125 billion total parameters and 6 billion active per token. The model serves as an architecture preview for Qwen4 and achieves roughly one-ninth the training cost of Qwen3.7-Plus. It introduces a hybrid attention mechanism, gated residual, N-gram embedding, and the Muon optimizer.
Key Facts
- Qwen3.8-Flash-Next has 125 billion total parameters, with only 6 billion active per token.
- The model includes a 51 billion parameter N-gram embedding layer that can run in system RAM.
- Training cost is roughly one-ninth that of Qwen3.7-Plus, according to the Qwen team.
- The FP8 checkpoint requires 172.78 GiB and the BF16 checkpoint 335.28 GiB.
- The production version ships as Qwen3.8-Flash through QwenCloud, priced at $0.16 per million input tokens and $0.47 per million output tokens.
Architecture Innovations
The model pairs a 125B backbone with a 51B N-gram embedding table and a 4B multi-token prediction module, totaling 180B on disk. Three of every four layers use Gated DeltaNet, a linear-attention layer that compresses history into a fixed-size recurrent state. The fourth layer runs Qwen Sparse Attention, which selects context at micro-block granularity with a budget of 512 blocks or 2048 tokens. The residual stream widens into 4 parallel branches with an element-wise read gate and a per-branch scalar write gate at bottleneck rank 320. The N-gram embedding layer is a 20,000,000-entry bigram/trigram table at layer 2 that can be offloaded to host memory with asynchronous prefetch.
Benchmark Performance
Qwen reports 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, and 81.0 on SWE-bench Multilingual. On agentic tasks, the model posts 73.9 on CoWorkBench, 55.7 on JobBench, and 73.5 on Toolathlon Verified. Multimodal results include 84.5 on AndroidWorld, 76.6 on LVBench, and 88.5 on RealWorldQA. Claude Opus 4.6 (Max) leads HLE at 40.0 against Qwen's 35.9, and DeepSeek-V4-Flash-0731 leads NL2Repo-Bench at 54.2 versus 48.1.
Deployment Requirements
The FP8 checkpoint is 172.78 GiB and the BF16 checkpoint is 335.28 GiB. Per vLLM recipes, TP2 is the minimum validated FP8 configuration on GB300 and TP4 is recommended. On an 8×H200 node, use TEP8; plain TP8 is incompatible with the checkpoint's 128-wide quantization blocks. The model natively supports a 262,144-token context window and can scale to one million tokens using YaRN. Weights are available on Hugging Face and ModelScope, and the technical report is on GitHub.
2 sources
Alibaba Qwen Releases Qwen3.8-Flash-Next, 125B MoE With 6B Active Parameters
