NVIDIA দেখিয়েছে DGX Spark-এ MoE ইনফারেন্স দ্রুত করার ৩টি পদ্ধতি, Ling-3.0-flash থ্রুপুট ৪২.৫% বৃদ্ধি করেছে

এটি ফিডে পাওয়া অংশবিশেষ। এটি সংক্ষিপ্ত বা প্রকাশকের মূল ভাষায় হতে পারে।

NVIDIA’s latest technical blog shows how to speed up mixture-of-experts (MoE) inference on one DGX Spark. The test model is Ant Group inclusionAI’s Ling-3.0-flash, a 124B-parameter MoE with 5.1B active parameters. NVIDIA stacked three optimizations: a new MoE kernel backend, an FP8 output layer and built-in speculative decoding. On all 1,319 GSM8K questions, decode throughput rose 42.5% with no statistically detectable accuracy change. So why is it relevant for Devs? DGX Spark is a desktop system, not a data center node. NVIDIA’s results show a 124B MoE serving single-user requests faster on that desk-side box. The technical post also shows where MoE decode time goes, and how to cut it. TL;DR Size: Ling-3.0-flash-fp4: 124B total, 5.1B active per token; 256K native context. Runs on: One NVIDIA DGX Spark (GB10, 128 GB unified memory, 273 GB/s), batch size one. Performance: Three stacked optimizations lift GSM8K decode throughput from 37.81 to 53.89 tokens/s. Best: +42.5% throughput and -24.5% mean latency, with accuracy flat (96.89% vs 96.82%). Lagging: Synthetic gains (+185.1%) shrink on GSM8K as MTP acceptance drops from 3.90 to 2.68. Why is MoE decoding on DGX Spark memory-bound? DGX Spark is NVIDIA’s desktop system built on the GB10 Grace Blackwell Superchip. It offers 128 GB of unified memory and 273 GB/s of memory bandwidth. Peak compute is up to 1 petaFLOP of sparse FP4. NVIDIA rates it for inference on models up to 200B parameters. In the NVIDIA technical post, the NVIDIA team profiled Ling-3.0-flash-fp4 at batch size 1. Decoding was memory-bound: reading model weights drove execution time. Two operations stood out, the MoE computation and the LM head that scores the vocabulary. That points to two levers: move fewer bytes per decode step, and generate more tokens per verification round. The baseline followed the official W4A8 recipe. It used FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM and an FP32 LM-head path. On a synthetic 32K-input, 2K-outpu