In the previous articles, we focused on the “brain” (the CPU) and its local memory. But in modern system design—especially for hyperscale data centers and AI clusters—the bottleneck isn’t how fast a single chip can compute; it’s how fast data can move between chips. This is the domain of the System Fabric.
As a System Architect, you are currently witnessing a generational shift from PCIe (Peripheral Component Interconnect Express) to the transformative world of CXL (Compute Express Link).
1. PCIe: The Foundation of Connectivity
PCIe is the ubiquitous point-to-point serial interconnect. It is a layered protocol:
- Physical Layer: Manages high-speed differential signaling (SerDes).
- Data Link Layer: Ensures reliable packet delivery (ACK/NAK).
- Transaction Layer: Handles memory reads/writes and I/O.
The Limitation: PCIe is “I/O centric.” It treats every device as an external peripheral. This introduces significant latency and overhead because the CPU has to “map” the device’s memory into its own address space, often involving complex driver stacks.
2. CXL: The “Memory-First” Revolution
CXL is a breakthrough because it runs on top of the physical PCIe Gen5/Gen6 wires but introduces Cache Coherency. It allows a CPU to treat an external device (like an FPGA, GPU, or Memory Expander) as if it were local L3 cache or DRAM.
CXL defines three distinct protocols:
- CXL.io: Based on PCIe; used for device discovery and configuration.
- CXL.cache: Allows a device to cache system memory locally with hardware-enforced coherency.
- CXL.mem: Allows the CPU to access memory located on an external device using simple load/store instructions.
3. Memory Pooling and Composable Infrastructure
The “Holy Grail” for data center architects is Memory Pooling. Currently, if a server has 512GB of RAM but only uses 100GB, that extra 412GB is “stranded”—it cannot be used by the server next door.
With CXL and a CXL Fabric Switch, we can create a pool of memory in a separate chassis. Servers can dynamically “borrow” RAM from the pool over the fabric and return it when finished.
- The Benefit: Massive reduction in TCO (Total Cost of Ownership) and increased hardware utilization.
- The Challenge: Managing the “Link Training” and “Hot Plug” events at the fabric level without crashing the host OS.
4. Architecting for Reliability: AER and Hot-Plug
In an embedded or server environment, the fabric must be resilient.
- Advanced Error Reporting (AER): This allows the fabric to log bit-flips or packet drops. As an architect, your firmware must decide if an error is “Correctable” (ignore/log) or “Uncorrectable” (trigger a 0x124 BSOD or a reset).
- Surprise Removal: What happens if a CXL memory module is physically pulled out while the CPU is reading from it? Your architecture must include “Downstream Port Containment” (DPC) to prevent the entire system from hanging.
5. Summary for the System Architect
| Interconnect | Coherency | Primary Use Case |
| PCIe Gen 4/5 | No | Standard NVMe SSDs, NICs, GPUs. |
| CXL 1.1 / 2.0 | Yes | Direct-attached Memory Expansion, AI Accelerators. |
| CXL 3.0+ | Yes | Fabric-wide Memory Pooling and Peer-to-Peer switching. |
| NVLink / Infinity | Yes | Proprietary, ultra-high-speed GPU-to-GPU clusters. |
Closing Thought
The fabric is no longer just a “wire”; it is a distributed memory controller. As we design the next generation of semiconductors, the distinction between “local” and “remote” memory is blurring, making the CXL Controller as important as the CPU core itself.
In the next article, we move from connectivity to protection: Article 7: Security Architecture — TrustZone, Enclaves, and the Hardware Root of Trust.
Ready to lock down the system?
