Architectural Paradigm Shifts in Data Center Servers: A Review of Workload Offloading, Security Isolation, and the Rise of Infrastructure Processing Units (IPUs)

Abstract

In hyperscale cloud computing, general-purpose central processing units (CPUs) face a fundamental scalability bottleneck driven by the cessation of Dennard scaling, decelerating Moore’s Law, and exponential growth in network I/O bandwidth (scaling from 100 Gbps to 400–800 Gbps). As distributed microservices, disaggregated NVMe storage fabrics, and massive machine learning clusters expand, host CPUs are increasingly burdened by the "datacenter tax" - consuming 20% to 35% or more of total compute cycles on software-defined networking (SDN), storage virtualization, packet cryptography, and hypervisor management. Concurrently, multi-tenant co-location introduces microarchitectural cache contention, tail latency inflation, and side-channel vulnerability surfaces. To address these limitations, modern server architecture has bifurcated into application compute domains and dedicated Infrastructure Processing Units (IPUs) and Data Processing Units (DPUs). This review article provides a comprehensive synthesis of hardware offloading and workload isolation paradigms across five distinct evolutionary generations - spanning legacy NICs, TCP Offload Engines (TOEs), advanced SmartNICs, FPGA accelerators, and sovereign System-on-Chip (SoC) IPUs and DPUs. We examine state-of-the-art hyperscale implementations, focusing on Google’s Titanium architecture, the custom Intel IPU (Mount Evans / E2000) deployed across Google Compute Engine C3 instances, the hardware-accelerated Falcon transport protocol, and Google's open-source Packet Security Protocol (PSP). Furthermore, we provide an extensive comparative analysis of leading commercial architectures - including Intel IPU, NVIDIA BlueField-3, AMD Pensando Elba, and AWS Nitro - and evaluate the emerging transition toward autonomous, "CPU-less" server topologies enabled by Compute Express Link (CXL) coherent fabrics.