AI summary
Tsinghua’s open-source AKV system cuts LLM agent costs by 27.7% and boosts performance by intelligently reusing KV cache entries instead of summarizing history.
AI-generated summary. Check the original reporting for context and accuracy.
Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
Source excerpt
This is the available feed excerpt, which may be shortened or in the publisher’s original language.
Tsinghua University and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM agents. Instead of summarizing old history, agents drop selected KV cache entries and keep reusing the rest. Across 3 models and 4 agent benchmarks, average cost falls 27.7%. Average task score rises 1.7 points. The code is Apache-2.0 licensed and builds on SGLang. TL;DR What it is: Serving system plus policy library, no weights. Tested on GPT-OSS-120B (5.1B active), MiniMax-M2.7 (230B total, 10B active) and MiroThinker-1.7-Mini (30B total, 3B active). Runs on: NVIDIA GPUs via SGLang. Paper used 8 A800-80GB GPUs in BF16. Quick start serves GPT-OSS-120B on 2 GPUs (tensor parallel 2). Performance: 27.7% lower average cost and +1.7 points average score versus threshold summarization. Best: GPT-OSS-120B on SWE-bench Verified: cost per task 20.0 to 9.7 cents, score 40.0 to 43.2. Worst: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 points below Text Engineering’s 62.3. What is cache engineering, and how is it different from context engineering? AKV is a Tsinghua serving system that lets LLM agents drop and reposition their own KV cache. Context engineering trims or summarizes the text history. Editing that prefix invalidates every later KV state. The retained suffix must then be re-prefilled. Harnesses therefore summarize late and aggressively, which can lose details from middle rounds. Cache engineering leaves the text trajectory untouched. It only changes which KV entries stay visible to attention. Retained entries are reused without recomputation. They can still carry information from dropped messages. In the research paper’s cost analysis, KV cache reads made up 82.4% of total API cost. How does AKV work? AKV defines 3 cache operations. Append is the standard update for new tokens. Drop removes selected entries from future attention without changing positions. Repos rewrites retained keys to positions 0 through r-1 using a RoPE rotation. Values st
About priority
An automated sorting aid based on source trust, keywords, and entities, on a scale of 1–12. High: 9–12; medium: 6–8; low: 1–5. It does not measure urgency, verify reporting, or indicate editorial endorsement.