Google’s Android Bench update exposes Gemini’s gap

Google’s updated Android Bench puts Gemini 3.1 Pro in fifth place behind newer rival models.

Google has refreshed Android Bench with eight new models, and the new board is not flattering for Gemini. In the updated results, Gemini 3.1 Pro lands in fifth place while OpenAI and Anthropic models sit ahead of it.

The benchmark matters because Android app work is one of the places where coding agents can save real time, or waste a lot of it. Google says Android Bench measures performance across 100 Android development tasks, and the company has now added cost and efficiency data alongside accuracy.

Google widened the field, and Gemini lost ground

When Google first launched Android Bench in March, the company’s own models were already behind the leaders. The new update makes that gap easier to see because the leaderboard now includes a wider set of current agents: Claude Sonnet 5, Claude Opus 4.8, GPT 5.4, GPT 5.5, MiniMax, Qwen, Kimi, and GLM.

That matters for developers because benchmark boards often get distorted when they only include a narrow slice of models. Here, Google widened the test and still ended up with Gemini trailing the pack. The strongest result belongs to Claude Fable 5, which reached 84.5 percent accuracy on the Android task set.

• Android Bench covers 100 Android development tasks.
• Google reran earlier tests after moving to Harbor.
• The historical results stay available in an archive.
• The new dataset and instructions live on GitHub.

Cost is now part of the scorecard

Accuracy alone does not tell developers much about whether a model is practical. Google added cost and efficiency metrics, and those numbers change the story in a useful way. Claude Fable 5 and GPT 5.5 may sit near the top, but both burn through more than $130 in tokens just to complete the 100-problem, 10-run benchmark.

Gemini 3.1 Pro scores lower, yet it runs the benchmark for $87. That makes it cheaper than the top performers in this test, even if it cannot match their success rate. Then there is Gemini 3.5 Flash, which was supposed to be the frugal option. On Android Bench, it is the most expensive model on the board at $165 per run, and it needed 28 hours to finish.

“The benchmark is a community effort,” Google wrote in its Android Bench announcement on GitHub.

That line matters because Google is asking outside developers to help shape the test, not just consume it. The company says Android Bench should keep changing as workflows change, and it wants developers to submit tasks and benchmark results for possible inclusion in the official set.

Harbor changes how developers can test models

To make that collaboration easier, Google switched Android Bench to the Harbor framework. Google says Harbor makes it easier to run, evaluate, and share results, which lowers the barrier for developers who want to test their own Android work against AI agents.

The company also reran the old tests under Harbor, so some scores shifted even though the tasks themselves did not change yet. That is an important detail for anyone comparing the old board with the new one. The ranking is not just a fresh snapshot of model quality; it is also a new baseline created with a different testing setup.

• Harbor is meant to simplify local runs and result sharing.
• Google preserved the old data in an archive.
• Developers can submit tasks for possible inclusion.
• The Android Bench GitHub now includes the updated dataset.

What this says about Google’s AI coding push

Google has been pushing harder into agentic development, where models do more of the coding work instead of just answering prompts. Android Bench is useful because it tests that promise in a domain Google knows well: building Android apps.

The uncomfortable part is that Google’s own models are still not leading the field. If Android developers keep seeing better results from OpenAI and Anthropic systems, Google has a product problem, not just a benchmark problem. It also explains why Google would want more real-world Android tasks in the benchmark and more developer participation in the process.

The next thing to watch is whether Google’s updated board changes model behavior or just model marketing. If developers contribute enough new tasks through Harbor, Android Bench could become a more honest test of what these agents can actually ship. If not, the leaderboard will keep telling the same story: Gemini is good enough to compete, but not good enough to lead.