Together AI-এর Together Compiler: এজেন্টগুলো ইনফারেন্সকে ২.৬ গুণ পর্যন্ত দ্রুত করে

এটি ফিডে পাওয়া অংশবিশেষ। এটি সংক্ষিপ্ত বা প্রকাশকের মূল ভাষায় হতে পারে।

Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop. TL;DR System: It is an optimization system, tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B. Runs on: NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2. Each run used at most 2 nodes per model. Performance: Up to 2.6x higher TPS than the best published InferenceX configurations, in under 24 hours per setting. Best: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), 1.8x over the 633 TPS baseline. Worst: GLM 5.3 on MI355X moved only from 463 to 480 TPS (about 4%). What is Together Compiler? Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model’s forward pass. It takes a server launch command, a traffic profile and a hardware configuration. Agents propose faster forward passes, and a persistent engine measures each one. The winning code ships into the serving engine. When traffic drifts, the loop runs again. Why does inference need continual compilation? Together AI team reports that its traffic shape never sits still. Prompt lengths in one country ran about 97K tokens versus 70K elsewhere. One customer’s P90 input length grew from 18K to 33K tokens in 3 days. Kernels tuned on random data can also mislead. Random inputs activated 13 to 38% more MoE experts than real traffic. One agent-written top-k kernel took 20 µs on random data. On real data at 70K context, it took up to 62 ms. How does the persistent engine work? Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The research team measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM. Together Compiler starts the engine once. Agents s