The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference - SemiAnalysis

Google's TPU push into inference is a structural assault on the open-ecosystem model, not a product iteration. Technically, the v5p/v6 generation's token-throughput and energy-efficiency edge is eroding CUDA's monopoly on the inference side—JAX and PyTorch now compile natively to TPU ISA, meaning NVIDIA's software moat is leaking from training into inference. Upstream, this reshapes CoWoS and HBM allocation: inference silicon demands lower compute density than training chips, compressing the 2.5D packaging premium and pressuring foundry pricing power in Taiwan, China. On compliance, TPU's proprietary architecture carries a "non-general-purpose" character that may navigate export-control reviews more favorably than x86-based accelerators. Yet HBM supply remains tethered to SK Hynix and Samsung—geopolitical transmission is intact. Strategically, NVIDIA's most probable counter is accelerating Rubin inference variants and deepening NVLink lock-in for enterprise workloads. AMD faces a pincer: outperformed in training by NVIDIA, undercut in inference TCO by TPU. The 12–24 month tail: every 40% drop in per-token cost triggers another wave of AI application democratization. But TPU's fatal constraint is enclosure—it lives exclusively inside GCP. The real inflection point isn't another perf bump; it's whether Google exposes TPU inference as a cross-cloud API. That day, NVIDIA's pricing power finally cracks.