Baidu presenta ERNIE 5.5 Flash Base: un modelo base MoE de 280B con 14B de parámetros activos

Este es el extracto disponible del feed. Puede estar abreviado o en el idioma original.

Baidu has released ERNIE 5.5 Flash Base, the raw pre-training checkpoint of its new sparse Mixture-of-Experts language model. ERNIE 5.5 Flash Base is a 280B-parameter text model from Baidu that activates 14B parameters per token. The weights stop at the end of pre-training, with no mid- or post-training applied. This as a clean starting point for teams that want to compare fine-tuning, instruction tuning or RL methods from one checkpoint. TL;DR Size: 280B total, 14B active per token; 128K training context, 1M for fine-tuning and inference. Runs on: Multi-GPU servers. Baidu’s reference configs use 4-way parallelism in SGLang and vLLM. GPU model and VRAM not disclosed. FP8 and BF16 weights. Performance: Best score on 24 of 30 base-model benchmarks in Baidu’s unified pipeline (self-reported). Best: MMLU-Pro 78.4, ahead of DeepSeek-V4-Flash-Base at 69.2. Worst: HELMET-Rerank 56.2, behind DeepSeek-V4-Flash-Base at 60.0. What is ERNIE 5.5 Flash Base? It is a text-only MoE base model with 43 layers and a 4096 hidden size. It was pre-trained from scratch on 33.5T tokens with the Muon optimizer. The tokenizer has 201,216 entries and groups numbers right-to-left in chunks of up to 3 digits. Activated compute, KV cache and long-context cost were treated as one budget. Pre-training data leaned on repository-scale code and long-context material, since those are hard to add later. How does the ERNIE 5.5 Flash architecture work? Hybrid attention: 37 layers use Heavily Compressed Attention (HCA), which caches 1 entry per 128 tokens. The other 6 layers use Multi-head Latent Attention with sparse attention (MLA + DSA). That means 1 in 7 layers keeps full-context access. HCA layers use 512-dim heads; MLA layers use 256-dim heads. Gated and mixed heads: The attention output gate comes from the low-rank query latent, cutting gate-projection parameters by 75%. Virtual Head Attention adds a lightweight low-rank mix across heads. Multi-Router Latent MoE. Each MoE layer has 512