Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers

This is the available feed excerpt, which may be shortened or in the publisher’s original language.

Google Research has released Agile Deliberation, a human-in-the-loop framework for building image classifiers around subjective concepts. Think ‘healthy food’, ‘inspiring artwork’ or clickbait in content moderation. Most tooling assumes users already know the concept boundary. Agile Deliberation assumes they do not, and helps them find it. TL;DR Framework: It prompts Gemini 2.5 Pro and Gemini 2.5 Flash through the API, without fine-tuning. Runs on: A Jupyter or Colab notebook. Retrieval uses CLIP embeddings and a FAISS index on CPU or GPU. The demo index starts at about 260 MB. Performance: 10.5 F1 points above zero-shot Gemini 2.5 Flash on average, across 2 concepts and 18 live sessions. Best: Paid to play F1 of 0.59 vs 0.48 zero-shot, with precision up 22 points (0.35 to 0.57). Worst: Recall drops 17 points on paid to play (0.80 to 0.63) and 15 points on healthy food. What is Agile Deliberation? Agile Deliberation is a Google Research’s framework that turns a vague concept into a structured written definition. That definition does two jobs. It is a human-readable policy, and it is the text prompt for a VLM classifier. Each round keeps the candidate definition with the best F1 on the user’s labels. The design came from interviews with a group of content moderation experts. Experts scoped a concept first, then refined it on borderline images. Finding those images was the slow part. How does concept scoping work? Stage 1 breaks the concept into at most 3 unit concepts, using prompt chaining with Gemini 2.5 Pro. Each unit gets candidate positive and negative subconcepts, shown with retrieved images. For healthy food, positives might be salads or grain bowls. Negatives might be processed food or raw, unprepared ingredients. The user keeps, flips or drops each one. How does concept iteration find borderline images? Stage 2 skips classic uncertainty sampling. The classifier is a prompted generative VLM, so its confidence scores are often uncalibrated. Inste