TechVoices

VIDEO PODCAST

AMD's Madhu Rangarajan on Modernizing Infrastructure for AI

James Maguire

September 22, 2025

Enterprise IT leaders face a maze of choices as they modernize infrastructure for AI. In this conversation, Madhu Rangarajan, Corporate VP of Server Products at AMD, lays out a pragmatic roadmap: first reclaim space, power, and cooling by upgrading to modern CPUs; then align compute choices (CPU vs. GPU) to workload scale; and finally embrace open standards across networking and software to avoid lock-in.

Core Takeaways

Start by creating headroom

Refreshing older servers with modern EPYC-class CPUs can cut space and power dramatically, freeing capacity for AI infrastructure (CPU and/or GPU) while improving efficiency.

Match compute to the workload

Classical ML and small-scale LLM/chat workloads often run well on CPUs; large-scale generative AI typically pairs high-performance CPUs with GPUs for maximum throughput.

Open standards reduce lock-in

AMD champions open ecosystems—ROCm, Ultra Ethernet Consortium, and UALink—to enable interoperable AI supercomputers and integrator flexibility as AI adoption broadens.

KEY QUOTES

— The First Step: Create Space, Power, and Cooling Headroom

Before you debate CPU versus GPU, get your data center AI-ready. Many enterprises are running five-year-old servers; moving to the latest generation can deliver the same general-purpose work in far less space and power. That reclaimed capacity becomes your runway for deploying the right mix of CPU and GPU infrastructure.

Efficiency isn't a nice-to-have—it's the enabler. When you free up 70% in space and power, you gain options: scale out CPUs for traditional analytics and smaller models, or slot in GPUs for large-scale generative AI. Modernization is the foundation that makes all those choices possible.

— When to Use CPUs vs. GPUs

Classical machine learning and smaller language model workloads often run great on CPUs—especially with optimized libraries and framework integrations. If you're serving an internal chatbot intermittently, CPU can be the simplest, most cost-effective starting point.

At large scale for LLMs and generative AI, you want CPU + GPU together. And even then, the CPU matters a lot: it handles orchestration, preprocessing, and kernel launch phases. Right-sizing that CPU layer reduces bottlenecks and increases GPU utilization.

— Why High-Performance CPUs Still Matter in GPU Boxes

We built our latest high-frequency, many-core CPUs to 'get out of the way' faster—spiking to very high clocks during CPU-bound phases so GPUs stay fed. In real deployments, that can translate into double-digit percentage gains on an eight-GPU server.

Think of it as performance per dollar: if a GPU server costs into the six figures, a 10–20% uplift from the CPU layer is real money. Tuning the CPU side isn't optional in AI at scale; it's central to achieving the ROI you promised the business.

ABOUT THE AUTHOR

James Maguire

Executive Director

An award-winning journalist, James has held top editorial roles in several leading technology publications, covering enterprise trends in cloud computing, AI, data analytics, cybersecurity and more. He regularly communicates with industry analysts and experts and has interviewed hundreds of technology executives.

Cloudera's Venkat Rajaji: advantages of open source AI

Snowflake's Christian Kleinerman: AI and the speed of data access

David Linthicum: LLM trends

Extreme Networks' Nabil Bukhari: how AI democratizes technology

Komprise's Krishna Subramanian: an AI model for every task

— The First Step: Create Space, Power, and Cooling Headroom

— When to Use CPUs vs. GPUs

— Why High-Performance CPUs Still Matter in GPU Boxes

Ahead of the curve, every week.

Get in touch with us.

Ahead of the curve, 
every week.