WeLM-617B

WeLM-617B is the frontier-scale model of the WeLM (WeChat Language Model) family, developed by WeChat AI inside Tencent. With 617 billion total parameters and 23 billion activated per token in a sparse MoE architecture, it was first shown publicly on July 14, 2026 as part of the "Hidden Decoding at Scale" technical blog and arXiv paper — the first demonstrated sequence-length scaling method at the 100B+ MoE scale.

Model Specifications

Hidden Decoding at Scale

Hidden Decoding is a sequence-length scaling method applied during continued pre-training. It expands each token into parallel streams with independent embedding tables, giving every token more internal computation without adding or widening Transformer layers. The WeLM-HD4-617B variant (n=4) improves all nine shared benchmarks over the plain WeLM-617B autoregressive baseline — with the incremental continuation training costing only about 5.3% of the full base-model training budget (0.90T vs 17.06T tokens).

Benchmark Highlights (early SFT-only)

• GPQA Diamond: 89.1 (91.2 with HD4) — ahead of Kimi K2.6's 90.4 on the HD4 variant
• HMMT: 96.0 (96.2 with HD4)
• PHYBench: 75.3 (76.3 with HD4)
• MMMLU: 86.4 (87.5 with HD4)
• FrontierMath: 49.0 (51.0 with HD4)
• SciCode: 51.4 (52.1 with HD4)

WeChat Ecosystem Integration

WeLM powers Xiaowei (小微), WeChat's native AI assistant in gray-scale testing since June 2026. Its core model is WeLM self-developed by the WeChat team, with DeepSeek used as a supplement for complex reasoning. The integration gives it native access to chat records, group chats, Official Account articles, Video Accounts, PDFs, and Office documents — positioning it to become one of the highest-call-volume language models in China.

Strengths & Limitations

• Efficiency-first design: Highly sparse MoE serving billions of WeChat users at low latency.
• Strong Chinese + multilingual: Optimized for Chinese internet context with code-switching support.
• Closed ecosystem: Not open-weight like Qwen or DeepSeek, and primarily serves the WeChat product.
• Limited public benchmarks: 617B results are early SFT-only; no RL post-training scores have been published.

Conclusion

WeLM-617B represents a distinct scaling philosophy: instead of simply adding parameters, WeChat AI folds additional computation into the sequence dimension. While it remains closed and WeChat-centric, its Hidden Decoding results mark a genuine technical milestone for efficient frontier-scale scaling — and through Xiaowei, it is already reaching hundreds of millions of WeChat users.

Related: Claude 3.5 Sonnet and Claude 3 Haiku. Hub: models.