Together Compiler continuously re-optimizes AI inference as traffic shifts, delivering up to 2.6x speed improvements and preventing performance degradation from static tuning.
AI-generated summary. Check the original reporting for context and accuracy.
This is the available feed excerpt, which may be shortened or in the publisher’s original language.
Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop. TL;DR System: It is an optimization system, tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B. Runs on: NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2. Each run used at most 2 nodes per model. Performance: Up to 2.6x higher TPS than the best published InferenceX configurations, in under 24 hours per setting. Best: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), 1.8x over the 633 TPS baseline. Worst: GLM 5.3 on MI355X moved only from 463 to 480 TPS (about 4%). What is Together Compiler? Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model’s forward pass. It takes a server launch command, a traffic profile and a hardware configuration. Agents propose faster forward passes, and a persistent engine measures each one. The winning code ships into the serving engine. When traffic drifts, the loop runs again. Why does inference need continual compilation? Together AI team reports that its traffic shape never sits still. Prompt lengths in one country ran about 97K tokens versus 70K elsewhere. One customer’s P90 input length grew from 18K to 33K tokens in 3 days. Kernels tuned on random data can also mislead. Random inputs activated 13 to 38% more MoE experts than real traffic. One agent-written top-k kernel took 20 µs on random data. On real data at 70K context, it took up to 62 ms. How does the persistent engine work? Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The research team measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM. Together Compiler starts the engine once. Agents s
An automated sorting aid based on source trust, keywords, and entities, on a scale of 1–12. High: 9–12; medium: 6–8; low: 1–5. It does not measure urgency, verify reporting, or indicate editorial endorsement.
OpenAI’s GPT-Image-2.6 speeds up generation by 50% and adds GIF/SVG support, significantly enhancing developer capabilities and creative workflow efficiency.
Google's Agile Deliberation framework helps users define subjective image concepts without fine-tuning, using Gemini APIs in a simple notebook environment.
MIT’s new memory system corrects mislabeled objects in a robot’s database, ensuring it can accurately locate critical items like defibrillators during emergencies.