[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-marginlab-claude-code-opus-46-tracker-zh":3,"tags-marginlab-claude-code-opus-46-tracker-zh":33,"related-lang-marginlab-claude-code-opus-46-tracker-zh":49,"related-posts-marginlab-claude-code-opus-46-tracker-zh":53,"series-ai-agent-62a923b4-173d-465e-93f0-071226ba6119":90},{"id":4,"title":5,"content":6,"summary":7,"source":8,"source_url":9,"author":10,"image_url":11,"keywords":12,"language":21,"translated_content":10,"views":22,"is_premium":23,"created_at":24,"updated_at":24,"cover_image":11,"published_at":25,"rewrite_status":26,"rewrite_error":10,"rewritten_from_id":27,"slug":28,"category":29,"related_article_id":30,"status":31,"google_indexed_at":32,"x_posted_at":10,"tweet_text":10,"title_rewritten_at":10,"title_original":10,"key_takeaways":10,"topic_cluster_id":10,"embedding":10,"is_canonical_seed":23},"62a923b4-173d-465e-93f0-071226ba6119","Marginlab 盯上 Claude Code 漂移","\u003Cp>\u003Ca href=\"https:\u002F\u002Fmarginlab.ai\u002Ftrackers\u002Fclaude-code\u002F\" target=\"_blank\" rel=\"noopener\">Marginlab\u003C\u002Fa> 做了一個每天跑的追蹤器。\u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fclaude-code\" target=\"_blank\" rel=\"noopener\">Claude Code\u003C\u002Fa> 搭配 \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-4-6\" target=\"_blank\" rel=\"noopener\">Opus 4.6\u003C\u002Fa>，每天固定測 50 題。它還會做週報和月報。講白了，就是盯著模型有沒有慢慢歪掉。\u003C\u002Fp>\u003Cp>這件事很實際。很多 coding agent 在 demo 看起來很猛。真的上線後，表現卻會飄。Marginlab 直接抓 \u003Ca href=\"https:\u002F\u002Fwww.swebench.com\u002F\" target=\"_blank\" rel=\"noopener\">SWE-Bench-Pro\u003C\u002Fa> 的子集合來跑。它不用花俏包裝。它想看的，就是使用者真正在 CLI 裡碰到的結果。\u003C\u002Fp>\u003Ch2>每天到底在看什麼\u003C\u002Fh2>\u003Cp>這個 tracker 的核心是 pass rate。也就是 50 題裡，Claude Code 解對幾題。這個數字最直觀。你一眼就知道今天有沒有掉。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147267370-bbwz.png\" alt=\"Marginlab 盯上 Claude Code 漂移\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>但 Marginlab 沒有只看分數。它還顯示 input tokens、output tokens、API cost、平均 runtime，還有總 tool calls。這很重要。因為 agent 出問題時，常常不是只掉準確率。\u003C\u002Fp>\u003Cp>有時候它會一直試。工具呼叫暴增。runtime 也變長。這種情況下，模型可能還能解題，但效率已經變差。對開發者來說，這就是成本開始失控的前兆。\u003C\u002Fp>\u003Cul>\u003Cli>每天固定跑 50 題\u003C\u002Fli>\u003Cli>每週與每月做彙總\u003C\u002Fli>\u003Cli>用 Bernoulli trials 看 pass rate\u003C\u002Fli>\u003Cli>顯示 95% 信賴區間\u003C\u002Fli>\u003Cli>直接跑 Claude Code CLI，不加自製 harness\u003C\u002Fli>\u003C\u002Ful>\u003Cp>最後一點最關鍵。很多 benchmark 一包再包，最後量到的是 wrapper，不是模型。Marginlab 直接走 Claude Code CLI。這比較接近真實開發流程。說真的，這種做法比較不會自嗨。\u003C\u002Fp>\u003Cp>它也有一個 degradation status pan\u003Ca href=\"\u002Fnews\u002Fzellij-0440-windows-remote-sessions-cli-zh\">el\u003C\u002Fa>。這裡會把樣本數和統計門檻一起放出來。不是每次掉分都算數。樣本太少時，波動本來就很大。這點比很多只會貼分數圖的頁面誠實多了。\u003C\u002Fp>\u003Ch2>為什麼現在要做這個\u003C\u002Fh2>\u003Cp>Marginlab 說，這個 tracker 是回應 \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-degradations\" target=\"_blank\" rel=\"noopener\">Anthropic\u003C\u002Fa> 在 2025 年 9 月的 degradation postmortem。那份說明很直接。模型上線後，表現還是可能往下掉。不是每次更新都只會變好。\u003C\u002Fp>\u003Cp>這也是很多團隊最怕的事。你昨天還覺得 agent 很穩。今天突然開始多跑幾輪。或者同一題要花更多 token。這種變化不會立刻炸掉服務，但會慢慢吃掉你的成本和信心。\u003C\u002Fp>\u003Cp>Marginlab 也強調自己是獨立第三方。它沒有跟 frontier model provider 綁在一起。這點很重要。因為很多 d\u003Ca href=\"\u002Fnews\u002Fgemini-live-gets-major-upgrade-with-3-1-flash-live-zh\">ash\u003C\u002Fa>board 很像行銷頁。分數一高就大吹特吹。Marginlab 比較像在做監控，不是在做宣傳。\u003C\u002Fp>\u003Cblockquote>“We want to offer a resource to detect such degradations in the future.” — Marginlab\u003C\u002Fblockquote>\u003Cp>這句話很直白。它不是在做排行榜。它是在做預警器。模型如果在 reasoning、tool use，或長任務表現上開始漂移，日更追蹤就有機會先抓到。\u003C\u002Fp>\u003Cp>我覺得這種工具會越來越重要。因為 agent 的問題，常常不是一次壞掉。是慢慢變鈍。等使用者抱怨時，很多跡象早就出現了。\u003C\u002Fp>\u003Ch2>數字怎麼看，才不會被騙\u003C\u002Fh2>\u003Cp>Marginlab 最有意思的地方，在於它把樣本數和統計門檻講清楚。50 題的日測，本來就很吵。它沒有假裝每天的波動都很有意義。這種誠實很少見。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147268472-6a4w.png\" alt=\"Marginlab 盯上 Claude Code 漂移\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>它的估算是這樣。50 題時，大約要 ±13.8% 的變化，才比較能過 p < 0.05。350 題時，門檻縮到 ±4.8%。1,400 題時，門檻只剩 ±2.3%。樣本越多，越能分辨噪音和真問題。\u003C\u002Fp>\u003Cp>這對 coding agent 特別重要。因為 agent 的行為很不穩。今天可能多試幾次。明天可能少試幾次。runtime、tool calls、token 數都會晃。只看單日分數，很容易誤判。\u003C\u002Fp>\u003Cul>\u003Cli>50 題：約 ±13.8% 才容易有統計意義\u003C\u002Fli>\u003Cli>350 題：約 ±4.8%\u003C\u002Fli>\u003Cli>1,400 題：約 ±2.3%\u003C\u002Fli>\u003Cli>日、週、月都會做聚合\u003C\u002Fli>\u003Cli>同時看 pass rate、runtime、tool calls、token、成本\u003C\u002Fli>\u003C\u002Ful>\u003Cp>這組數字告訴我們一件事。小樣本很適合快速掃描。大樣本才適合下結論。很多團隊在內部看板只放一條分數線，結果天天被噪音搞心態。Marginlab 至少把這件事講開了。\u003C\u002Fp>\u003Cp>如果拿競品來比，差異也很明顯。像 \u003Ca href=\"https:\u002F\u002Fswebench.com\u002F\" target=\"_blank\" rel=\"noopener\">SWE-Bench\u003C\u002Fa> 本身偏向標準化評測。\u003Ca href=\"https:\u002F\u002Fopenai.com\u002Findex\u002Fopenai-codex\u002F\" target=\"_blank\" rel=\"noopener\">OpenAI Codex\u003C\u002Fa> 這類產品展示時，通常更強調能力與體驗。Marginlab 則把焦點放在持續監控。它問的不是「能不能跑贏一次」，而是「能不能一直維持」。\u003C\u002Fp>\u003Ch2>這種監控，為什麼對台灣團隊有用\u003C\u002Fh2>\u003Cp>現在很多軟體團隊都在把 \u003Ca href=\"\u002Fnews\u002Fai-slop-flooding-streaming-services-musicians-fight-back-zh\">AI\u003C\u002Fa> agent 放進流程。寫 code、改 bug、補測試、查 issue，通通有人想交給 LLM。問題是，模型不是靜態元件。今天的表現，和下週可能不一樣。\u003C\u002Fp>\u003Cp>台灣很多團隊資源不算多。你不可能每次模型更新都人工驗證一輪。這時候就需要固定的監控機制。每天跑一小批題目，配上週報和月報，至少能先知道有沒有異常。\u003C\u002Fp>\u003Cp>這種做法也適合跟自家觀測系統搭配。你可以把 pass rate 當功能指標。把 token 和成本當財務指標。把 runtime 和 tool calls 當效率指標。三個一起看，比只看一個漂亮分數實在多了。\u003C\u002Fp>\u003Cp>再往前想一步，這也會影響採購決策。當你要選 Claude、GPT，或其他 LLM API 時，除了價格和上下文長度，也該看它在你自己的任務上，會不會隔幾天就漂一次。這不是學術問題。這是產品風險。\u003C\u002Fp>\u003Cp>我自己的判斷很簡單。未來做 agent 的團隊，會越來越像在管伺服器。不是只裝好就算了。你還要看健康度、延遲、錯誤率，還有版本變更後的差異。模型監控會變成基本功。\u003C\u002Fp>\u003Ch2>接下來該怎麼看這類 tracker\u003C\u002Fh2>\u003Cp>如果你有在用 Claude Code，我會建議先把這個 tracker 收藏起來。不要只看某一天掉分。要看連續幾天的趨勢。再對照 runtime、tool calls 和 token 數。這樣比較不會被單日噪音騙到。\u003C\u002Fp>\u003Cp>更實際一點，團隊可以自己做一個小版。固定任務集。固定執行路徑。固定記錄成本和延遲。只要你有 30 到 50 題的穩定樣本，就已經能抓到不少異常訊號。\u003C\u002Fp>\u003Cp>我猜接下來會有更多這種獨立追蹤器。原因很簡單。模型更新太快了。只看官方公告不夠。只看 demo 也不夠。你需要的是每天都在跑的數據。\u003C\u002Fp>\u003Cp>所以問題不是「模型某次考幾分」。問題是「它下週還能不能維持」。如果你的產品真的靠 coding agent 吃飯，那這種監控最好現在就做起來。\u003C\u002Fp>","Marginlab 每天跑 50 個 SWE-Bench-Pro 任務，追蹤 Claude Code Opus 4.6 的分數漂移，還會看 token、成本與工具呼叫。","marginlab.ai","https:\u002F\u002Fmarginlab.ai\u002Ftrackers\u002Fclaude-code\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147267370-bbwz.png",[13,14,15,16,17,18,19,20],"Claude Code","Marginlab","Opus 4.6","SWE-Bench-Pro","AI agent","模型監控","benchmark drift","LLM","zh",2,false,"2026-04-02T16:27:31.179506+00:00","2026-04-02T16:27:31.153+00:00","done","7b9c08cf-92ae-426b-b6ac-5a5bb0fff51c","marginlab-claude-code-opus-46-tracker-zh","ai-agent","1e86831a-5448-4953-b598-edd58f6f58d6","published","2026-04-08T09:00:50.433+00:00",[34,35,37,39,41,42,45,47],{"name":18,"slug":18},{"name":13,"slug":36},"claude-code",{"name":20,"slug":38},"llm",{"name":19,"slug":40},"benchmark-drift",{"name":17,"slug":29},{"name":43,"slug":44},"SWE-bench Pro","swe-bench-pro",{"name":15,"slug":46},"opus-46",{"name":14,"slug":48},"marginlab",{"id":30,"slug":50,"title":51,"language":52},"marginlab-claude-code-opus-46-tracker-en","Marginlab Tracks Claude Code Opus 4.6 Drift","en",[54,60,66,72,78,84],{"id":55,"slug":56,"title":57,"cover_image":58,"image_url":58,"created_at":59,"category":29},"38406a12-f833-4c69-ae22-99c31f03dd52","switch-ai-outputs-markdown-to-html-zh","怎麼把 AI 輸出改成 HTML","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778743243861-8901.png","2026-05-14T07:20:21.545364+00:00",{"id":61,"slug":62,"title":63,"cover_image":64,"image_url":64,"created_at":65,"category":29},"c7c69fe4-97e3-4edf-a9d6-a79d0c4495b4","anthropic-cat-wu-proactive-ai-assistants-zh","Cat Wu 談 Claude 的主動式 AI","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778735455993-gnw7.png","2026-05-14T05:10:30.453046+00:00",{"id":67,"slug":68,"title":69,"cover_image":70,"image_url":70,"created_at":71,"category":29},"e1d6acda-fa49-4514-aa75-709504be9f93","how-to-run-hermes-agent-on-discord-zh","如何在 Discord 執行 Hermes Agent","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778724655796-cjul.png","2026-05-14T02:10:34.362605+00:00",{"id":73,"slug":74,"title":75,"cover_image":76,"image_url":76,"created_at":77,"category":29},"4104fa5f-d95f-45c5-9032-99416cf0365c","why-ragflow-is-the-right-open-source-rag-engine-to-self-host-zh","為什麼 RAGFlow 是最適合自架的開源 RAG 引擎","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778674262278-1630.png","2026-05-13T12:10:23.762632+00:00",{"id":79,"slug":80,"title":81,"cover_image":82,"image_url":82,"created_at":83,"category":29},"7095f05c-34f5-469f-a044-2525d2010ce9","how-to-add-temporal-rag-in-production-zh","如何在正式環境加入 Temporal RAG","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778667053844-osvs.png","2026-05-13T10:10:30.930982+00:00",{"id":85,"slug":86,"title":87,"cover_image":88,"image_url":88,"created_at":89,"category":29},"10479c95-53c6-4723-9aaa-2fde5fb19ee7","github-agentic-workflows-ai-github-actions-zh","GitHub 把 AI 代理放進 Actions","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778551884342-8io7.png","2026-05-12T02:11:02.069769+00:00",[91,96,101,106,111,116,121,126,131,136],{"id":92,"slug":93,"title":94,"created_at":95},"4ae1e197-1d3d-4233-8733-eafe9cb6438b","claude-now-uses-your-pc-to-finish-tasks-zh","Claude 開始幫你操作電腦","2026-03-26T07:20:48.457387+00:00",{"id":97,"slug":98,"title":99,"created_at":100},"5bede67f-e21c-413d-9ab8-54a3c3d26227","googles-2026-ai-agent-report-decoded-zh","Google 2026 AI Agent 報告解讀","2026-03-26T11:15:22.651956+00:00",{"id":102,"slug":103,"title":104,"created_at":105},"2987d097-563f-46c7-b76f-b558d8ef7c2b","kimi-k25-review-stronger-still-not-legend-zh","Kimi K2.5 評測：更強，但還不是神作","2026-03-27T07:15:55.277513+00:00",{"id":107,"slug":108,"title":109,"created_at":110},"95c9053b-e3f4-4cb5-aace-5c54f4c9e044","claude-code-controls-mac-desktop-zh","Claude Code 也能操控 Mac 了","2026-03-28T03:01:58.58121+00:00",{"id":112,"slug":113,"title":114,"created_at":115},"dc58e153-e3a8-4c06-9b96-1aa64eabbf5f","cloudflare-100x-faster-ai-agent-sandbox-zh","Cloudflare 的 AI 沙箱跑超快","2026-03-28T03:09:44.142236+00:00",{"id":117,"slug":118,"title":119,"created_at":120},"1c8afc56-253f-47a2-979f-1065ff072f2a","openai-backs-isara-agent-swarm-bet-zh","OpenAI 挺 Isara 的 agent swarm …","2026-03-28T03:15:27.513155+00:00",{"id":122,"slug":123,"title":124,"created_at":125},"7379b422-576e-45df-ad5a-d57a0d9dd467","openai-plan-automated-ai-researcher-zh","OpenAI 想做自動化 AI 研究員","2026-03-28T03:17:42.090548+00:00",{"id":127,"slug":128,"title":129,"created_at":130},"48c9889e-86df-450b-a356-e4a4b7c83c5b","harness-engineering-ai-agent-reliability-2026-zh","駕馭工程：從「馬具」到「作業系統」，AI Agent 可靠性的終極密碼","2026-03-31T06:42:53.556721+00:00",{"id":132,"slug":133,"title":134,"created_at":135},"e41546b8-ba9e-455f-9159-88d4614ad711","openai-codex-plugin-claude-code-zh","OpenAI 把 Codex 放進 Claude Code","2026-04-01T09:21:54.687617+00:00",{"id":137,"slug":138,"title":139,"created_at":140},"96d8e8c8-1edd-475d-9145-b1e7a1b02b65","mcp-explained-from-prompts-to-production-zh","MCP 怎麼把提示詞變工作流","2026-04-01T09:24:39.321274+00:00"]