The Hardware Hierarchy: Why Your CPU Needs Both Memory and Storage

By Sumit Somani | Staff Validation Engineer

As a Validation Engineer working on next-generation NVMe and CXL hardware, I spend my days looking at how data moves.

But before we can talk about advanced storage protocols, we have to start at the very beginning. If you've ever wondered why computers have both "Memory" (RAM) and "Storage" (SSDs), it all comes down to a fundamental law of physics: You can't have something that is both infinitely fast and infinitely large.

To solve this, hardware architects created The Memory Hierarchy.

1. The Analogy: The Master Chef

Imagine a Master Chef working in a restaurant. The Chef is your computer's CPU (The Brain). The Chef is incredibly fast, but they can only chop vegetables if those vegetables are immediately within reach.

Here is how the kitchen is laid out:

• The Chef's Hands (Registers & L1 Cache): This is the memory built directly into the CPU. It is instantaneously fast, but the Chef can only hold a few items at a time.

• The Kitchen Countertop (Memory / RAM): The Chef can chop a lot of food here. It's very quick to reach, but the countertop size is limited. Furthermore, at the end of the night, the countertop is wiped completely clean (RAM is volatile).

• The Walk-in Fridge (Storage / SSD): The fridge is massive. You can store weeks of food here, and the food stays there even when the restaurant closes (Non-volatile). However, walking to the fridge and opening the heavy door takes a very long time compared to just reaching across the countertop.

2. The Theory: Speed vs Capacity

In computer architecture, data must travel from the Walk-in Fridge (SSD), be placed on the Countertop (RAM), so the Chef (CPU) can finally use it.

We separate Memory and Storage because of cost and physics:

• SRAM / DRAM (Memory): It uses electrical capacitors that hold a charge. It is lightning fast (measured in nanoseconds), but it is very expensive to manufacture per gigabyte, and requires constant power to keep the data alive.

• NAND Flash (Storage): SSDs trap electrons inside microscopic floating gates. It is much cheaper to manufacture, allowing us to build Terabytes of storage. It retains data without power indefinitely. However, reading and writing to those gates is much slower (measured in microseconds).

The entire goal of a hardware architect is to hide the slowness of the Fridge from the Chef.

3. Visualizing the Pyramid

If we map these components out, they form a perfect pyramid. As you go up the pyramid, hardware gets faster but smaller and more expensive. As you go down, it gets cheaper and larger, but significantly slower.

The Takeaway

Now that we know the CPU has to fetch everything from the Storage (The Fridge) and put it into Memory (The Countertop), a massive question arises:

How exactly do we build the door to the fridge?

If the door is too small, it doesn't matter how fast the flash memory is inside—the Chef will be left waiting. This specific problem is exactly why the tech industry eventually had to abandon SATA cables and invent PCIe and NVMe.

Stay tuned for Part 2, where we compare the SATA dirt road to the PCIe Superhighway.