Realtime-Bench revela que los LLMs actuales fallan en tareas de baja latencia, como los juegos y las subastas, lo que destaca una brecha crítica en la capacidad de la IA para tomar decisiones en tiempo real.
Resumen generado con IA. Consulta el original para comprobar el contexto y la precisión.
Este es el extracto disponible del feed. Puede estar abreviado o en el idioma original.
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.Subscribe nowLLMs are very bad at low-latency tasks:…Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you…Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation).Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks.What it measures: The test is designed to probe for three qualities: temporal state awareness (”RT-Perceive”), where models can use observations to track changes in the world; adaptive deliberation (”RT-Deliberate”), where models alter their reasoning length and depth according to available time; and time-aware action selection (”RT-Act”) where models need to appropriately time their actions for the situation. “For example, an agent purchasing a ticket must track the sale status and assess whether observed availability is still current (RT-Perceive), limit further deliberation when the purchase window is closing (RT-Deliberate), and choose between waiting for sales to open and purchasing while tickets remain available (RT-Act).” They have three different conditions: Wall clock, which is designed to test out performance under real-world latency. Paused clock, which lets you arbitrarily pause things so you can get a sense of baseline capability, and Token clock, which “advances
Una ayuda automática para ordenar, basada en fuentes, palabras clave y entidades, en una escala de 1–12. Alta: 9–12; media: 6–8; baja: 1–5. No mide urgencia, verifica noticias ni representa respaldo editorial.
Los expertos en energía del MIT destacaron que la compleja transición energética requiere un enfoque interdisciplinario e internacional para abordar desafíos como el carbono residual, la confianza en la IA y las aplicaciones cuánticas.
El modelo de pesos abiertos Hydrogen-1.0 de Deep Principle democratiza el descubrimiento científico avanzado al ofrecer una herramienta masiva con licencia MIT para la investigación en química y ciencias de la vida.
Auto Ontology de NVIDIA garantiza que los agentes de IA generen SQL preciso al fundamentarlos en definiciones de negocio certificadas, evitando errores costosos en el análisis de datos empresariales.