Baidu Releases ERNIE 5.5 Flash Base: A 280B MoE Base Model With 14B Active Parameters

This is the available feed excerpt, which may be shortened or in the publisher’s original language.

Baidu has released ERNIE 5.5 Flash Base, the raw pre-training checkpoint of its new sparse Mixture-of-Experts language model. ERNIE 5.5 Flash Base is a 280B-parameter text model from Baidu that activates 14B parameters per token. The weights stop at the end of pre-training, with no mid- or post-training applied. This as a clean starting point for teams that want to compare fine-tuning, instruction tuning or RL methods from one checkpoint. TL;DR Size: 280B total, 14B active per token; 128K training context, 1M for fine-tuning and inference. Runs on: Multi-GPU servers. Baidu’s reference configs use 4-way parallelism in SGLang and vLLM. GPU model and VRAM not disclosed. FP8 and BF16 weights. Performance: Best score on 24 of 30 base-model benchmarks in Baidu’s unified pipeline (self-reported). Best: MMLU-Pro 78.4, ahead of DeepSeek-V4-Flash-Base at 69.2. Worst: HELMET-Rerank 56.2, behind DeepSeek-V4-Flash-Base at 60.0. What is ERNIE 5.5 Flash Base? It is a text-only MoE base model with 43 layers and a 4096 hidden size. It was pre-trained from scratch on 33.5T tokens with the Muon optimizer. The tokenizer has 201,216 entries and groups numbers right-to-left in chunks of up to 3 digits. Activated compute, KV cache and long-context cost were treated as one budget. Pre-training data leaned on repository-scale code and long-context material, since those are hard to add later. How does the ERNIE 5.5 Flash architecture work? Hybrid attention: 37 layers use Heavily Compressed Attention (HCA), which caches 1 entry per 128 tokens. The other 6 layers use Multi-head Latent Attention with sparse attention (MLA + DSA). That means 1 in 7 layers keeps full-context access. HCA layers use 512-dim heads; MLA layers use 256-dim heads. Gated and mixed heads: The attention output gate comes from the low-rank query latent, cutting gate-projection parameters by 75%. Virtual Head Attention adds a lightweight low-rank mix across heads. Multi-Router Latent MoE. Each MoE layer has 512