Kinference simplifica la gestión del servicio de modelos de IA en Kubernetes al enrutar inteligentemente las solicitudes a los trabajadores de vLLM, optimizando el rendimiento y el uso de recursos para los equipos que implementan múltiples modelos.
Resumen generado con IA. Consulta el original para comprobar el contexto y la precisión.
Este es el extracto disponible del feed. Puede estar abreviado o en el idioma original.
Buoyant, the company behind Linkerd, has released Kinference, an open source project for building inference platforms on Kubernetes. Kinference is an inference control plane for Kubernetes and vLLM. It sits in front of a fleet of vLLM workers and decides where each request goes. It applies routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer. For dev teams serving many open-weight models to many internal clients, it targets the layer above the model server. The project is Apache 2.0 licensed on GitHub. The current release is v0.1.0-alpha.7. TL;DR Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs. Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant’s benchmark used 3 L40S GPUs per pool. Performance: large gains on prefix-heavy traffic versus plain Kubernetes Service routing (self-reported). Best: median TTFT fell from 6.88 s to 0.31 s (95%) on a shared-prefix workload. Worst: no median TTFT gain on prompts without shared prefixes (6.92 s vs 6.83 s). What is Kinference? Kinference is a policy-driven control plane that turns Kubernetes plus vLLM into a multi-tenant inference platform. Buoyant’s FAQ puts it simply: vLLM is the data plane and Kinference is the control plane. How does Kinference work? The system has 2 parts: a traffic gateway and an inference handler. The GitHub README states that the handler exposes a gRPC interface. It takes a request and returns the chosen endpoint plus a possibly mutated body. It pairs with a proxy such as the Buoyant Gateway. Kinference watches vLLM KV-cache events and engine metrics. It currently uses NVIDIA’s dynamo-kv-router crate to process cache events. Configuration lives in 5 CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet. What are inference policies? Inference policies are declarative rules for how each request is routed or rejected. Rules can match on tenant, reque
Una ayuda automática para ordenar, basada en fuentes, palabras clave y entidades, en una escala de 1–12. Alta: 9–12; media: 6–8; baja: 1–5. No mide urgencia, verifica noticias ni representa respaldo editorial.
Predecir el comportamiento molecular a partir de datos limitados acelera el descubrimiento de fármacos y materiales, sustituyendo el lento método de prueba y error por una innovación eficiente y basada en datos en diversas industrias.
Los núcleos de código abierto ZYFA de Zyphra aumentan la velocidad de prefill en AMD MI355X hasta 1,76 veces, reduciendo significativamente los errores FP8 en comparación con la biblioteca predeterminada de AMD.
ToyBlocks ofrece un entorno estandarizado y controlado para entrenar y evaluar algoritmos de aprendizaje de robots, permitiendo comparaciones de rendimiento consistentes y reproducibles.