As AI Moves from Training to Inference, Optics Moves Closer to the Chip

As AI transitions from training to large-scale inference, connectivity has emerged as a central bottleneck in AI system design. While training has historically dominated AI infrastructure, inference d... Read original →

The surge in AI inference workloads is forcing optical interconnects to leap from co-packaged optics toward true 3D integration. This shift will trigger upstream reallocation toward barium titanate and III-V materials, while downstream GPU architectures must adapt to the 'wide and slow' data paradigm. Geopolitical export controls on advanced packaging tools have already inflated compliance costs for firms in Taiwan, China and Hong Kong, China, as heterogeneous integration of optical engines relies heavily on U.S., Japanese, and Dutch equipment—amplifying supply chain fragility. Facing Ayar Labs’ and Lightmatter’s CPO lead, NVIDIA may accelerate acquisitions of silicon photonics startups, while TSMC could lock in partners like Alchip via its SoIC platform. Within 18 months, 3D optical I/O will become the decisive factor in AI chip energy efficiency; players failing to breach the 1pJ/bit barrier will be marginalized in the inference market.