[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-google-android-bench-update-gemini-gap-en":3,"article-related-google-android-bench-update-gemini-gap-en":31,"series-research-abcd6dcf-d3f5-4280-84f7-46439ba1416e":80},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":23,"views":27,"created_at":28,"published_at":29,"topic_cluster_id":30},"abcd6dcf-d3f5-4280-84f7-46439ba1416e","google-android-bench-update-gemini-gap-en","Google’s Android Bench update exposes Gemini’s gap","\u003Cp data-speakable=\"summary\">Google’s updated Android Bench puts Gemini 3.1 Pro in fifth place behind newer rival models.\u003C\u002Fp>\u003Cp>Google has refreshed \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fandroid-bench\" target=\"_blank\" rel=\"noopener\">Android Bench\u003C\u002Fa> with eight new models, and the new board is not flattering for \u003Ca href=\"https:\u002F\u002Fai.google.dev\u002Fgemini\" target=\"_blank\" rel=\"noopener\">Gemini\u003C\u002Fa>. In the updated results, Gemini 3.1 Pro lands in fifth place while \u003Ca href=\"https:\u002F\u002Fopenai.com\" target=\"_blank\" rel=\"noopener\">OpenAI\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\" target=\"_blank\" rel=\"noopener\">Anthropic\u003C\u002Fa> models sit ahead of it.\u003C\u002Fp>\u003Cp>The \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> matters because Android app work is one of the places where coding agents can save real time, or waste a lot of it. Google says Android Bench measures performance across 100 Android development tasks, and the company has now added cost and efficiency data alongside accuracy.\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Model\u003C\u002Fth>\u003Cth>Placement\u003C\u002Fth>\u003Cth>Accuracy\u003C\u002Fth>\u003Cth>Cost for benchmark\u003C\u002Fth>\u003Cth>Runtime\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>Claude Fable 5\u003C\u002Ftd>\u003Ctd>1st\u003C\u002Ftd>\u003Ctd>84.5%\u003C\u002Ftd>\u003Ctd>More than $130\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Gemini 3.1 Pro\u003C\u002Ftd>\u003Ctd>5th\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003Ctd>$87\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Gemini 3.5 Flash\u003C\u002Ftd>\u003Ctd>Lower on board\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003Ctd>$165\u003C\u002Ftd>\u003Ctd>28 hours\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>GPT 5.5\u003C\u002Ftd>\u003Ctd>Near top\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003Ctd>More than $130\u003C\u002Ftd>\u003Ctd>Not stated\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>Google widened the field, and Gemini lost ground\u003C\u002Fh2>\u003Cp>When Google first launched Android Bench in March, the company’s own models were already behind the leaders. The new update makes that gap easier to see because the leaderboard now includes a wider set of current agents: \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-sonnet-5\" target=\"_blank\" rel=\"noopener\">Claude Sonnet 5\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-opus-4\" target=\"_blank\" rel=\"noopener\">Claude Opus 4.8\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fopenai.com\u002Findex\u002Fgpt-5-4\" target=\"_blank\" rel=\"noopener\">GPT 5.4\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fopenai.com\u002Findex\u002Fgpt-5-5\" target=\"_blank\" rel=\"noopener\">GPT 5.5\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fwww.minimax.io\u002F\" target=\"_blank\" rel=\"noopener\">MiniMax\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fqwenlm.github.io\u002F\" target=\"_blank\" rel=\"noopener\">Qwen\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fwww.kimi.com\u002F\" target=\"_blank\" rel=\"noopener\">Kimi\u003C\u002Fa>, and \u003Ca href=\"https:\u002F\u002Fglama.ai\u002Fmodel-catalog\u002Fmodels\u002Fglm-5-2\" target=\"_blank\" rel=\"noopener\">GLM\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783906370516-crbs.png\" alt=\"Google’s Android Bench update exposes Gemini’s gap\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>That matters for developers because benchmark boards often get distorted when they only include a narrow slice of models. Here, Google widened the test and still ended up with Gemini trailing the pack. The strongest result belongs to \u003Ca href=\"\u002Ftag\u002Fclaude\">Claude\u003C\u002Fa> Fable 5, which reached 84.5 percent accuracy on the Android task set.\u003C\u002Fp>\u003Cul>\u003Cli>Android Bench covers 100 Android development tasks.\u003C\u002Fli>\u003Cli>Google reran earlier tests after moving to Harbor.\u003C\u002Fli>\u003Cli>The historical results stay available in an archive.\u003C\u002Fli>\u003Cli>The new dataset and instructions live on GitHub.\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>Cost is now part of the scorecard\u003C\u002Fh2>\u003Cp>Accuracy alone does not tell developers much about whether a model is practical. Google added cost and efficiency metrics, and those numbers change the story in a useful way. Claude Fable 5 and GPT 5.5 may sit near the top, but both burn through more than $130 in tokens just to complete the 100-problem, 10-run benchmark.\u003C\u002Fp>\u003Cp>Gemini 3.1 Pro scores lower, yet it runs the benchmark for $87. That makes it cheaper than the top performers in this test, even if it cannot match their success rate. Then there is Gemini 3.5 Flash, which was supposed to be the frugal option. On Android Bench, it is the most expensive model on the board at $165 per run, and it needed 28 hours to finish.\u003C\u002Fp>\u003Cblockquote>“The benchmark is a community effort,” Google wrote in its Android Bench announcement on GitHub.\u003C\u002Fblockquote>\u003Cp>That line matters because Google is asking outside developers to help shape the test, not just consume it. The company says Android Bench should keep changing as workflows change, and it wants developers to submit tasks and benchmark results for possible inclusion in the official set.\u003C\u002Fp>\u003Ch2>Harbor changes how developers can test models\u003C\u002Fh2>\u003Cp>To make that collaboration easier, Google switched Android Bench to the \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fharbor\" target=\"_blank\" rel=\"noopener\">Harbor\u003C\u002Fa> framework. Google says Harbor makes it easier to run, evaluate, and share results, which lowers the barrier for developers who want to test their own Android work against \u003Ca href=\"\u002Ftag\u002Fai-agents\">AI agents\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783906385603-b36c.png\" alt=\"Google’s Android Bench update exposes Gemini’s gap\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The company also reran the old tests under Harbor, so some scores shifted even though the tasks themselves did not change yet. That is an important detail for anyone comparing the old board with the new one. The ranking is not just a fresh snapshot of model quality; it is also a new baseline created with a different testing setup.\u003C\u002Fp>\u003Cul>\u003Cli>Harbor is meant to simplify local runs and result sharing.\u003C\u002Fli>\u003Cli>Google preserved the old data in an archive.\u003C\u002Fli>\u003Cli>Developers can submit tasks for possible inclusion.\u003C\u002Fli>\u003Cli>The Android Bench GitHub now includes the updated dataset.\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>What this says about Google’s AI coding push\u003C\u002Fh2>\u003Cp>Google has been pushing harder into agentic development, where models do more of the coding work instead of just answering prompts. Android Bench is useful because it tests that promise in a domain Google knows well: building Android apps.\u003C\u002Fp>\u003Cp>The uncomfortable part is that Google’s own models are still not leading the field. If Android developers keep seeing better results from \u003Ca href=\"\u002Ftag\u002Fopenai\">OpenAI\u003C\u002Fa> and \u003Ca href=\"\u002Ftag\u002Fanthropic\">Anthropic\u003C\u002Fa> systems, Google has a product problem, not just a benchmark problem. It also explains why Google would want more real-world Android tasks in the benchmark and more developer participation in the process.\u003C\u002Fp>\u003Cp>The next thing to watch is whether Google’s updated board changes model behavior or just model marketing. If developers contribute enough new tasks through Harbor, Android Bench could become a more honest test of what these agents can actually ship. If not, the leaderboard will keep telling the same story: Gemini is good enough to compete, but not good enough to lead.\u003C\u002Fp>","Google added new models to Android Bench, and Gemini 3.1 Pro fell to fifth behind OpenAI and Anthropic rivals.","arstechnica.com","https:\u002F\u002Farstechnica.com\u002Fgoogle\u002F2026\u002F07\u002Fgoogle-revamps-android-ai-dev-benchmark-adds-fable-5-and-other-agents\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783906370516-crbs.png","research","en","c4597538-217d-4b81-83d0-9b3cc4153861",[17,18,19,20,21,22],"Android Bench","Gemini","Claude Fable 5","OpenAI","Anthropic","Harbor",[24,25,26],"Google updated Android Bench with eight new models and a Harbor-based workflow.","Gemini 3.1 Pro fell to fifth place behind OpenAI and Anthropic rivals.","Cost now matters as much as accuracy, and some top models are expensive to run.",1,"2026-07-13T01:32:25.716691+00:00","2026-07-13T01:32:25.709+00:00","31b59e63-edfa-4db7-9433-7aeb234a074e",{"tags":32,"relatedLang":39,"relatedPosts":43},[33,35,37],{"name":20,"slug":34},"openai",{"name":21,"slug":36},"anthropic",{"name":18,"slug":38},"gemini",{"id":15,"slug":40,"title":41,"language":42},"google-android-bench-update-gemini-gap-zh","Android Bench 更新，Gemini 掉到第五","zh",[44,50,56,62,68,74],{"id":45,"slug":46,"title":47,"cover_image":48,"image_url":48,"created_at":49,"category":13},"154edd47-cb74-4c14-b845-23cd4672b323","vlm-accuracy-visual-cognitive-errors-decade-en","How VLMs Learned Complex Scene Descriptions","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783926189650-jkdx.png","2026-07-13T07:02:37.258905+00:00",{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"ea29ba1b-1436-4f05-9809-f1108d957877","visual-pretraining-language-models-en","Visual Pretraining Beats Text-Only in Language Models","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783924380306-981l.png","2026-07-13T06:32:36.107914+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"82b70e73-d94e-4a11-9d1c-8bf09e74f798","phinn-eeg-topology-dream-state-eeg-en","PHINN-EEG brings topology to dream-state EEG","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783922586181-14w4.png","2026-07-13T06:02:35.072233+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"f15d1c6d-fdb6-4fe0-a671-f0450c038250","llm-benchmarks-not-enough-2026-en","Benchmarks should not pick your LLM in 2026","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783818163394-1kka.png","2026-07-12T01:02:19.841725+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"9386c57c-c41c-4682-afb2-56a661db7f23","rust-breaks-into-tiobe-top-10-en","Rust Breaks Into TIOBE’s Top 10","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783816366396-apgi.png","2026-07-12T00:32:24.472896+00:00",{"id":75,"slug":76,"title":77,"cover_image":78,"image_url":78,"created_at":79,"category":13},"dff7c89f-12cc-4eda-a605-67f4cd0d8036","ai-ransomware-still-needs-a-human-bottleneck-en","AI ransomware still needs a human bottleneck","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783800194209-o22p.png","2026-07-11T20:02:52.648504+00:00",[81,86,91,96,101,106,111,116,121,126],{"id":82,"slug":83,"title":84,"created_at":85},"a2715e72-1fe8-41b3-abb1-d0cf1f710189","ai-predictions-2026-big-changes-en","AI Predictions for 2026: Brace for Big Changes","2026-03-26T01:25:07.788356+00:00",{"id":87,"slug":88,"title":89,"created_at":90},"8404bd7b-4c2f-4109-9ec4-baf29d88af2b","ml-papers-of-the-week-github-research-desk-en","ML Papers of the Week Turns GitHub Into a Research Desk","2026-03-27T01:11:39.480259+00:00",{"id":92,"slug":93,"title":94,"created_at":95},"87897a94-8065-4464-a016-1f23e89e17cc","ai-ml-conferences-to-watch-in-2026-en","AI\u002FML Conferences to Watch in 2026","2026-03-27T01:51:54.184108+00:00",{"id":97,"slug":98,"title":99,"created_at":100},"6f1987cf-25f3-47a4-b3e6-db0997695be8","openclaw-agents-manipulated-self-sabotage-en","OpenClaw Agents Can Be Manipulated Into Failure","2026-03-28T03:03:18.899465+00:00",{"id":102,"slug":103,"title":104,"created_at":105},"a53571ad-735a-4178-9f93-cb09b699d99c","vega-driving-language-instructions-en","Vega: Driving with Natural Language Instructions","2026-03-28T14:54:04.698882+00:00",{"id":107,"slug":108,"title":109,"created_at":110},"a34581d6-f36e-46da-88bb-582fb3e7425c","personalizing-autonomous-driving-styles-en","Drive My Way: Personalizing Autonomous Driving Styles","2026-03-28T14:54:26.148181+00:00",{"id":112,"slug":113,"title":114,"created_at":115},"2bc1ad7f-26ce-4f02-9885-803b35fd229d","training-knowledge-bases-writeback-rag-en","Training Knowledge Bases with WriteBack-RAG","2026-03-28T14:54:45.643433+00:00",{"id":117,"slug":118,"title":119,"created_at":120},"71adc507-3c54-4605-bbe2-c966acd6187e","packforcing-long-video-generation-en","PackForcing: Efficient Long-Video Generation Method","2026-03-28T14:55:02.646943+00:00",{"id":122,"slug":123,"title":124,"created_at":125},"675942ef-b9ec-4c5f-a997-381250b6eacb","pixelsmile-facial-expression-editing-en","PixelSmile Framework Enhances Facial Expression Editing","2026-03-28T14:55:20.633463+00:00",{"id":127,"slug":128,"title":129,"created_at":130},"6954fa2b-8b66-4839-884b-e46f89fa1bc3","adaptive-block-scaled-data-types-en","IF4: Smarter 4-Bit Quantization That Adapts to Your Data","2026-03-31T06:00:36.65963+00:00"]