[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-shannon-scaling-law-llm-overtraining-zh":3,"article-related-shannon-scaling-law-llm-overtraining-zh":30,"series-research-04d71e47-d4ad-45bf-b678-5bcbdb1de0ee":81},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":29},"04d71e47-d4ad-45bf-b678-5bcbdb1de0ee","shannon-scaling-law-llm-overtraining-zh","香農尺度律解釋 LLM 過訓練","\u003Cp data-speakable=\"summary\">這篇論文把 \u003Ca href=\"\u002Ftag\u002Fllm\">LLM\u003C\u002Fa> \u003Ca href=\"\u002Fnews\u002Fskillopt-agent-skills-text-space-optimizer-zh\">訓練\u003C\u002Fa>看成帶雜訊的資訊傳輸，說明為何算力增加時，模型在噪聲下反而可能變差。\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>研究機構\u003C\u002Fstrong>：arXiv 摘要未明確標註\u003C\u002Fli>\u003Cli>\u003Cstrong>核心數據\u003C\u002Fstrong>：pooled R² = 0.847\u003C\u002Fli>\u003Cli>\u003Cstrong>突破點\u003C\u002Fstrong>：把訓練建成雜訊通道\u003C\u002Fli>\u003C\u002Ful>\u003Cp>\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.23901\">LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws\u003C\u002Fa> 想處理一個很實際的問題：大家熟悉的 scaling law，通常都默認「模型更大、資料更多、算力更高，表現就會一路變好」。但真實訓練裡，雜訊一旦變強，這條直線就不一定成立。這篇摘要的\u003Ca href=\"\u002Fnews\u002F5-packers-storylines-to-watch-right-now-zh\">重點\u003C\u002Fa>很明確：當訊號雜訊比掉到某個門檻以下，更多 scale 不一定帶來更好結果，甚至可能把模型推進更差的區間。\u003C\u002Fp>\u003Cp>這個切法對開發者很有感。因為失敗模式不只是「邊際效益遞減」而已，而是可能出現非單調的退化。也就是說，訓練還在繼續、算力還在加，但 loss 或表現不一定照著你期待的方向走。這篇論文就是要替這種現象找一個統一的理論框架。\u003C\u002Fp>\u003Ch2>這篇在補哪個洞\u003C\u002Fh2>\u003Cp>傳統 scaling law 多半是單調的 power law。它們假設你只要持續增加 compute、參數或資料，loss 就會穩定往下掉。摘要直接點出，這種觀點無法解釋一些非單調行為，例如 catastrophic overtraining，還有量化之後的性能退化。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1779689757133-oarp.png\" alt=\"香農尺度律解釋 LLM 過訓練\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>這些問題不是理論上的小瑕疵，而是實作裡真的會碰到的坑。模型在前期擴大時看起來正常，後面卻可能因為過訓練、擾動或量化而變差。作者想做的，是把這些看似分散的失敗案例，收進同一套說法裡。\u003C\u002Fp>\u003Cp>摘要裡最重要的轉向，是它不只說「雜訊很重要」，而是說雜訊會改變 scaling curve 的形狀。也就是說，單純把模型、資料或訓練步數拉大，並不保證進步；如果 SNR 沒跟上，曲線可能從單調改善，轉成 U 形退化。\u003C\u002Fp>\u003Ch2>方法怎麼看，才不會太抽象\u003C\u002Fh2>\u003Cp>這篇的核心做法，是把 LLM 訓練映射成通訊理論問題。作者把模型參數視為 channel bandwidth，把訓練 tokens 視為 signal power。這樣一來，訓練不再只是最佳化曲線，而比較像在雜訊通道裡傳遞資訊。\u003C\u002Fp>\u003Cp>這個比喻之所以有用，是因為它把「學習訊號」和「內在雜訊」之間的關係講得更清楚。不是所有 scale 都會幫助模型吸收更多有效訊號。當模型已經超出可用容量，額外的 scale 可能只是在放大雜訊，而不是把訊號榨得更乾淨。\u003C\u002Fp>\u003Cp>摘要提到，這套 formulation 能同時描述單調與非單調的行為。它被設計來解釋為什麼有些訓練曲線平順下降，有些卻會出現 loss basin 或 U-shaped curve。摘要也說，它能處理 Gaussian noise、quantization 這類擾動造成的退化。\u003C\u002Fp>\u003Ch2>論文實際證明了什麼\u003C\u002Fh2>\u003Cp>作者用 Pythia 和 OLMo2 做驗證，涵蓋 Gaussian noise、quantization，以及在 math、QA、code 任務上的 supervised fine-tuning。摘要沒有公開完整 benchmark 細節，所以這裡沒有逐項任務準確率可以完整列出。它能直接提供的是結論層級的訊息：這個新的 law 整體上比傳統 scaling laws 和近期的 perturbation-aware laws 表現更好。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1779689764797-pdai.png\" alt=\"香農尺度律解釋 LLM 過訓練\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>比較具體的一個數字，是作者用 Pythia 在最多 6.9B parameters、180B tokens 的資料上擬合，然後外推到沒看過的 12B Pythia 與 307B tokens，得到 pooled R² = 0.847。對做訓練曲線預測的人來說，這種外推能力很重要，因為它直接關係到你能不能在把完整算力燒完之前，先估出這輪訓練值不值得繼續跑。\u003C\u002Fp>\u003Cp>摘要還說，這個 Shannon Scaling Law 能抓到前人方法漏掉的 loss basin。這代表它不只是把趨勢線擬合得更漂亮，而是更能描述在擾動或過訓練發生時，曲線怎麼彎、怎麼掉。當然，這裡仍然只有摘要層級的說法，真正細節還是要看全文。\u003C\u002Fp>\u003Ch2>對開發者有什麼影響\u003C\u002Fh2>\u003Cp>如果你有在訓練或微調模型，這篇其實是在提醒一件事：不要把「更大」直接等同於「更穩」。摘要的論點是，只要 SNR 沒保住，checkpoint 再大、run 再長，也不保證比較好。這對 overtraining、quantization、以及任何訓練後還要再經過擾動的流程，都很有影響。\u003C\u002Fp>\u003Cp>對做 capacity planning 的團隊來說，這篇的價值在於預測。若一條 law 能用較小模型、較短 \u003Ca href=\"\u002Ftag\u002Ftoken\">token\u003C\u002Fa> budget 去外推更大、未見過的設定，就能幫你估算一輪 run 什麼時候開始不划算。就算你不打算直接採用這套理論，至少也會被提醒：評估 scaling 時，別只看乾淨條件，還要看噪聲條件。\u003C\u002Fp>\u003Cp>在部署端，quantization 的意義也被放大了。量化常被當成壓縮問題，但這篇把它放回同一個 scaling 討論裡，和訓練雜訊、過訓練一起看。這樣比較容易理解，為什麼某個模型在 full precision 看起來正常，一壓縮就開始掉。\u003C\u002Fp>\u003Ch2>還有哪些限制\u003C\u002Fh2>\u003Cp>摘要對理論和 headline 結果講得很清楚，但實務上還是留了不少空白。它沒有給完整 benchmark 表，所以我們無法從摘要知道每個任務、每種擾動到底贏多少。它也沒有說這套理論在 Pythia 和 OLMo2 之外，能不能穩定套到其他模型家族。\u003C\u002Fp>\u003Cp>另一個要保留的態度是：fit 得好，不等於就是萬用規則。作者主張的是一個統一框架，但目前可見的材料，只能證明它在有限的模型、擾動與任務上成立。開發者比較適合把它當成理解失敗模式的鏡頭，而不是保證所有 noisy training run 都會照同一條曲線走的定律。\u003C\u002Fp>\u003Cp>不過，這篇論文最有力的地方還是很清楚：它把 LLM scaling 從「越大越好」改\u003Ca href=\"\u002Fnews\u002Fsaint-marys-softball-turns-one-day-into-history-zh\">寫成\u003C\u002Fa>「這是一個 capacity problem」。一旦噪聲壓過訊號，曲線就可能往錯的方向彎。\u003C\u002Fp>\u003Ch2>總結\u003C\u002Fh2>\u003Cp>這篇的貢獻不是再補一條普通的 scaling law，而是用 Shannon 的語言去解釋，為什麼模型在噪聲下會過訓練、會退化、甚至會出現 U 形表現。對訓練、微調、量化或預估 LLM 成本的人來說，這是一個值得放進工具箱的視角。\u003C\u002Fp>\u003Cp>它也提醒了一個很實際的工程原則：scale 不是唯一答案，noise 才是決定曲線會不會轉彎的關鍵。當你在規劃下一輪訓練時，除了看算力，也要看訊號還剩多少。\u003C\u002Fp>\u003Cul>\u003Cli>把 LLM 訓練改寫成雜訊通道問題。\u003C\u002Fli>\u003Cli>能解釋 overtraining 與量化退化這類非單調失敗。\u003C\u002Fli>\u003Cli>在摘要提供的設定下，對 Pythia 外推得到 pooled R² = 0.847。\u003C\u002Fli>\u003C\u002Ful>","這篇論文把 LLM 訓練看成帶雜訊的資訊傳輸，說明為何算力增加時，模型在噪聲下反而可能變差。","arxiv.org","https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.23901",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1779689757133-oarp.png","research","zh","68b3843b-ea46-49f5-9c1c-7364193d5dc3",[17,18,19,20,21],"scaling law","Shannon theory","LLM training","overtraining","quantization",[23,24,25],"把訓練視為帶雜訊的資訊傳輸，能解釋非單調退化。","摘要未公開完整 benchmark 細節，但提供了 Pythia 外推的 pooled R² = 0.847。","對訓練、微調與量化流程來說，重點不只是 scale，而是 SNR。",5,"2026-05-25T06:15:31.356036+00:00","2026-05-25T06:15:31.34+00:00","0c35a120-52fc-41fc-afa3-d404eb934158",{"tags":31,"relatedLang":40,"relatedPosts":44},[32,34,36,37,39],{"name":18,"slug":33},"shannon-theory",{"name":17,"slug":35},"scaling-law",{"name":21,"slug":21},{"name":19,"slug":38},"llm-training",{"name":20,"slug":20},{"id":15,"slug":41,"title":42,"language":43},"shannon-scaling-law-llm-overtraining-en","Shannon Scaling Law explains LLM overtraining","en",[45,51,57,63,69,75],{"id":46,"slug":47,"title":48,"cover_image":49,"image_url":49,"created_at":50,"category":13},"f374155a-c29e-478c-b7a5-679cad1c51e4","crdts-keep-replicas-in-sync-without-locks-zh","CRDT 讓副本不用鎖也能同步","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781011086259-4p4k.png","2026-06-09T13:17:34.493426+00:00",{"id":52,"slug":53,"title":54,"cover_image":55,"image_url":55,"created_at":56,"category":13},"4b3b5a50-45b7-4238-a38b-160f82e323ff","post-deterministic-systems-autonomous-infra-zh","後決定性分散系：自治基礎設施新框架","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781010194792-5ogb.png","2026-06-09T13:02:32.717551+00:00",{"id":58,"slug":59,"title":60,"cover_image":61,"image_url":61,"created_at":62,"category":13},"04e45398-9814-4907-b416-fcb5b8d69508","causal-learnability-formal-language-tasks-zh","用因果法量化任務可學性","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780987696075-l4g0.png","2026-06-09T06:47:34.438642+00:00",{"id":64,"slug":65,"title":66,"cover_image":67,"image_url":67,"created_at":68,"category":13},"75bcc569-5e89-45c8-b809-6f169e929f4b","rl-training-hands-off-control-gradually-zh","RL 先接管再放手","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780986786312-03yo.png","2026-06-09T06:32:32.849589+00:00",{"id":70,"slug":71,"title":72,"cover_image":73,"image_url":73,"created_at":74,"category":13},"e3ecab4b-7cc7-4246-baf6-e1c170d86ca5","omnigamearena-vlm-game-agent-benchmark-zh","OmniGameArena 讓 VLM 遊戲代理更好比","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780985893022-70pl.png","2026-06-09T06:17:32.189729+00:00",{"id":76,"slug":77,"title":78,"cover_image":79,"image_url":79,"created_at":80,"category":13},"6f25a29c-cbb8-4f53-9af7-1656b394333a","turboquant-cuts-kv-cache-memory-6x-google-tests-zh","TurboQuant 在 Google 測試中省下 6x KV 快取","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780906682236-sqe2.png","2026-06-08T08:17:21.878314+00:00",[82,87,92,97,102,107,112,117,122,127],{"id":83,"slug":84,"title":85,"created_at":86},"f18dbadb-8c59-4723-84a4-6ad22746c77a","deepmind-bets-on-continuous-learning-ai-2026-zh","DeepMind 押注 2026 連續學習 AI","2026-03-26T08:16:02.367355+00:00",{"id":88,"slug":89,"title":90,"created_at":91},"f4a106cb-02a6-4508-8f39-9720a0a93cee","ml-papers-of-the-week-github-research-desk-zh","每週 ML 論文清單，為何紅到 GitHub","2026-03-27T01:11:39.284175+00:00",{"id":93,"slug":94,"title":95,"created_at":96},"c4f807ca-4e5f-47f1-a48c-961cf3fc44dc","ai-ml-conferences-to-watch-in-2026-zh","2026 AI 研討會投稿時程整理","2026-03-27T01:51:53.874432+00:00",{"id":98,"slug":99,"title":100,"created_at":101},"cf046742-efb2-4753-aef9-caed5da5e32e","adaptive-block-scaled-data-types-zh","IF4：神經網路量化的聰明選擇","2026-03-31T06:00:36.990273+00:00",{"id":103,"slug":104,"title":105,"created_at":106},"53a0dc54-0371-4e40-8d5e-74e94a73840c","geometry-aware-similarity-metrics-for-neural-representations-zh","超越距離測量：用微分幾何重新理解神經網路","2026-03-31T06:01:01.241968+00:00",{"id":108,"slug":109,"title":110,"created_at":111},"fee7d472-a775-4b1d-bbc2-1e8bca1bbf8b","on-the-fly-repulsion-in-the-contextual-space-for-rich-divers-zh","讓AI繪圖更有創意：用排斥力提升生成多樣性","2026-03-31T06:01:25.439673+00:00",{"id":113,"slug":114,"title":115,"created_at":116},"a9901203-d69b-447b-8854-15d14eab32b4","vision-aided-beam-prediction-cnn-eca-zh","影像輔助波束預測升級 CNN","2026-04-01T10:00:25.8073+00:00",{"id":118,"slug":119,"title":120,"created_at":121},"b55e7dd4-0a24-4b3d-804d-b0309a03f498","triple-band-fss-mimo-antenna-sub-6-ghz-zh","三頻 FSS MIMO 天線瞄準 sub-6 GHz","2026-04-01T13:18:36.857305+00:00",{"id":123,"slug":124,"title":125,"created_at":126},"f68290bd-e7f3-4b30-ba22-dcd4e0130a66","openclaw-1299-repos-eight-weeks-analysis-zh","OpenClaw 1299 個 Repo 的資料解讀","2026-04-02T05:03:45.208411+00:00",{"id":128,"slug":129,"title":130,"created_at":131},"ed9f80eb-eb02-4d35-8ad4-0ddf428751dd","beam-coherence-aware-combining-mmwave-mimo-zh","毫米波 MIMO 的雙階合併法","2026-04-02T05:27:26.897188+00:00"]