Realtime-Bench reveals that current LLMs fail at low-latency tasks like gaming and auctions, highlighting a critical gap in AI's ability to handle real-time decision-making.
AI-generated summary. Check the original reporting for context and accuracy.
This is the available feed excerpt, which may be shortened or in the publisher’s original language.
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.Subscribe nowLLMs are very bad at low-latency tasks:…Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you…Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation).Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks.What it measures: The test is designed to probe for three qualities: temporal state awareness (”RT-Perceive”), where models can use observations to track changes in the world; adaptive deliberation (”RT-Deliberate”), where models alter their reasoning length and depth according to available time; and time-aware action selection (”RT-Act”) where models need to appropriately time their actions for the situation. “For example, an agent purchasing a ticket must track the sale status and assess whether observed availability is still current (RT-Perceive), limit further deliberation when the purchase window is closing (RT-Deliberate), and choose between waiting for sales to open and purchasing while tickets remain available (RT-Act).” They have three different conditions: Wall clock, which is designed to test out performance under real-world latency. Paused clock, which lets you arbitrarily pause things so you can get a sense of baseline capability, and Token clock, which “advances
An automated sorting aid based on source trust, keywords, and entities, on a scale of 1–12. High: 9–12; medium: 6–8; low: 1–5. It does not measure urgency, verify reporting, or indicate editorial endorsement.
MIT energy experts emphasized that the complex energy transition requires an interdisciplinary, international approach to tackle challenges like waste carbon, AI trust, and quantum applications.
Deep Principle’s open-weight Hydrogen-1.0 model democratizes advanced scientific discovery by providing a massive, MIT-licensed tool for chemistry and life sciences research.
NVIDIA’s Auto Ontology ensures AI agents generate accurate SQL by grounding them in certified business definitions, preventing costly errors in enterprise data analysis.