DeepSeek open-sourcing its production inference framework gives developers rare access to a frontier lab's real serving stack, accelerating AI deployment and transparency.
AI-generated summary. Check the original reporting for context and accuracy.
This is the available feed excerpt, which may be shortened or in the publisher’s original language.
DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services. The DeepSeek DeepInfer inference framework is the engine behind DeepSeek-V4.1-Flash. It ships under the Apache 2.0 license. DeepSeek also released deepseek-recipe, its MIT-licensed token encoding and decoding layer. For developers, this is a rare look at a frontier lab’s real serving stack. TL;DR Size / scope: 1 supported model (DeepSeek-V4.1-Flash), 1M-token context window per the model card. Runs on: NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. Benchmarks used 4 GPUs for prefill and 64 for decode. Performance: Production-grade throughput; DeepSeek says 1 cluster serves 155 trillion tokens per day. Best: Above 25,000 output tokens/s/GPU at 256 requests per rank (EP64, 131,072 context). Worst: Offline inference only. KV cache system, P/D disaggregation and load balancing are not included. What is DeepInfer? DeepInfer is a high-performance inference engine from DeepSeek for its own models. According to it’s GitHub README, it powers DeepSeek’s large-scale production services. This release covers offline inference with the core features needed at scale. It was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. The deployment Q&A gives production context. A single cluster serves 155 trillion tokens per day. DeepSeek reports overall revenue of US$38 per GPU-hour at valley pricing. How does DeepInfer work? The engine stacks DeepSeek’s kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. On top sit 5 system-level ideas. Parallelism: Attention DP plus Wide-EP Attention uses data parallelism. The MoE layers use expert parallelism with an expert-parallel load balancer (EPLB). Decode runs EP64 to EP128 inside a supernode. Prefill uses the smallest EP that memory allows, typically EP2 to EP8. DSpark: confidence-scheduled speculative decoding A draft model proposes 5 tokens per step. A Confidenc
An automated sorting aid based on source trust, keywords, and entities, on a scale of 1–12. High: 9–12; medium: 6–8; low: 1–5. It does not measure urgency, verify reporting, or indicate editorial endorsement.
Mistral’s Voxtral 3 unifies speech-to-text, text-to-speech, and voice agent capabilities into a single API, simplifying the development of real-time voice applications.
Google’s llm-d 1.0 optimizes large-scale AI inference on Kubernetes, solving load-balancing issues for self-hosted agent and reinforcement learning workloads.
NVIDIA's new generative simulation tool transforms clinical surgery training by creating realistic, AI-driven scenarios that enhance medical education and surgical proficiency.