AI Infrastructure Is Entering Its Inference Era

Walking through AI Infra Summit 2026 in Santa Clara, I kept hearing what first sounded like separate conversations. NVIDIA was discussing agentic AI. Oracle was presenting large-scale GPU superclusters. Broadcom and Marvell were focused on networking. Across the exhibition floor, companies were talking about memory, storage, advanced packaging, liquid cooling and power management.

Underneath those different technologies was one shared question: how do we turn expensive AI compute into useful intelligence more efficiently?

That was my clearest takeaway from the summit. The first phase of the generative AI infrastructure race was largely about securing GPUs. The next phase is about keeping them productive: feeding them with data, moving information between racks, scheduling workloads and operating within increasingly difficult power and cooling limits.

The race is shifting from raw compute to system-level efficiency.

Training created the boom. Inference will determine the economics.

Training frontier models remains capital-intensive, but it is episodic. Inference is continuous. Every question, generated image, coding task or agent action creates a new request that infrastructure must serve.

Agentic systems make that workload less predictable. A chatbot may produce one answer. An agent may plan, search, call tools, inspect the result, revise its approach and call another model. A single user request can create many model calls and much more internal data movement.

The precise demand varies by application, but the direction is clear: the economic question is no longer simply how many GPUs a company owns. It is how much useful work the complete system can produce from them.

The question is moving from “How many GPUs do you have?” to “How much useful work can your system produce from them?”

The GPU is no longer the right unit of analysis

Another strong impression from the summit was how often speakers looked beyond the individual accelerator. The product is becoming the rack, the cluster and, increasingly, the data center as one coordinated computing system.

A powerful GPU delivers limited value when it is waiting for data, constrained by memory bandwidth, delayed by communication or underused because workloads have been scheduled poorly. Real performance depends on the whole architecture.

• CPUs coordinate workloads, tool calls and serial operations.
• GPUs and other accelerators perform parallel computation.
• High-bandwidth memory keeps data close to accelerators.
• Interconnects move information within and between racks.
• Storage, orchestration software, power and cooling determine how much of the system can stay productive.

Intel's keynote described the opportunity as extending from silicon to systems. NVIDIA's discussion of its Vera CPU made a related point: agentic AI does not make the CPU irrelevant. Orchestration, tool use, security and sequential decision-making still have to run alongside GPU-intensive model execution.

The next infrastructure winner may not be the company with the fastest isolated component. It may be the one that integrates the system best.

Memory and networking are becoming first-class constraints

The industry has spent years discussing compute shortages. At the summit, it was impossible to ignore the equally important problem of moving and retaining data. Longer contexts expand working memory. Distributed inference requires accelerators to communicate at very low latency. Agentic systems must preserve state across many steps.

Performance is therefore limited not only by how quickly a chip can calculate, but by how quickly the system can deliver the right data to it. This explains the attention given to high-bandwidth memory, advanced packaging, storage and network fabrics.

Broadcom describes three layers of AI networking: scale-up within a tightly coupled system, scale-out across racks in a data center, and scale-across between data centers. These are not just networking categories; they show how the boundary of the AI computer is expanding.

Storage is changing as well. It affects utilization, retrieval latency, checkpoint speed and recovery. Western Digital summarized that shift in the title of its summit session: “AI Is Not a Compute System. It's a Data System.”

Power and cooling are becoming compute technologies

Power was another recurring theme. AI workloads fluctuate, but facilities often reserve capacity around theoretical peak demand. When nodes draw less power while capacity elsewhere remains unavailable, the result is stranded power, stranded compute and lost revenue.

That is why power management is moving closer to workload orchestration. Scheduling systems will increasingly have to consider energy availability, thermal conditions, cooling capacity and network congestion alongside accelerator availability.

Cooling is undergoing the same transition. Coolant distribution units, direct-to-chip liquid cooling and high-density rack designs were visible throughout the event. They may look like facilities equipment, but they decide how much computing capacity a site can actually install.

The bottleneck is moving outward: from the chip to the rack, from the rack to the facility and from the facility to the power grid.

Inference will not be one homogeneous market

The inference era does not mean every workload will run in a hyperscale GPU cluster. Summit sessions from Qualcomm, Ambarella, Liquid AI and automotive companies pointed to another direction: intelligence running on vehicles, robots, PCs, phones and industrial devices.

Some workloads need large cloud clusters. Others benefit from a smaller, specialized model close to the user, especially when latency, privacy, memory or energy is the binding constraint.

Inference infrastructure will therefore be optimized across several variables at once: capability, latency, energy use, memory footprint, privacy, deployment location and total cost per completed task.

Better utilization may matter as much as more supply

The industry still needs more compute and power, but existing capacity is not always used well. GPUs wait for data. Fixed power budgets leave capacity idle. Workloads land on hardware that is available but too expensive for the job. Enterprises reserve infrastructure for demand that appears only intermittently.

This creates opportunities across the software and systems layers. Schedulers, inference engines, model routers, caching systems and observability platforms can improve the economics without manufacturing another GPU. SmartNICs and DPUs can move networking, storage and security work away from the host.

Oracle's Acceleron architecture is one example of this systems approach. Its supercluster message also illustrated how cloud providers increasingly sell an integrated combination of accelerators, networking, storage, security and software rather than access to an isolated GPU.

The share of installed capacity that can be converted into reliable, billable AI work may become one of the industry's most important competitive measures.

From compute ownership to intelligence efficiency

My main takeaway is not that the world needs fewer GPUs. Demand will keep growing as models become more capable and agents perform longer, more complex tasks. What is changing is how progress will be measured.

During the training-led phase of generative AI, access to accelerators was itself a major advantage. In the inference era, ownership will not be enough. Companies will have to operate infrastructure efficiently across compute, memory, networking, storage, software, power and cooling.

For investors and enterprise buyers, the question is no longer only who manufactures the leading accelerator. It is also which companies remove the bottlenecks that stop expensive accelerators from doing useful work.

The next phase of AI infrastructure may be less about who owns the most GPUs and more about who can turn compute into useful intelligence most efficiently.