LLM scaling laws

Shannon Scaling Law explains LLM overtraining

A Shannon-based scaling law explains why LLMs can get worse as compute rises under noise.