Tag
AI inference
AI inference is the runtime phase where trained models generate outputs in production, so latency, memory footprint, and compute cost matter most. Topics here include home-based inference nodes, KV-cache compression, and how long contexts keep DRAM demand high.
5 articles
Anthropic’s chip move is a necessary break from GPU dependence
Anthropic should pursue a custom AI chip because control over compute now matters more than vendor convenience.
Why Zyphra Cloud on AMD Matters More Than Another Model Launch
Zyphra Cloud matters because inference, not training, is now the real AI platform battle.
Span, Nvidia, Pulte: Mini AI Data Centers in Homes
Span is testing home-based AI inference nodes with 1.25 MW across 100 homes, cutting build time from years to months.
TurboQuant cuts memory use 6x without accuracy loss
Google Research’s TurboQuant claims 6x less memory and 8x faster inference with no accuracy loss, jolting AI inference economics.
TurboQuant Won’t Fix the Memory Crunch
Google’s TurboQuant can cut KV-cache memory use 6x, but longer contexts may keep DRAM and NAND demand climbing.