S03 — With DRAM, Why Does the CPU Still Need Cache? How Multiple Cache Levels Work Together
Key Insight: The biggest bottleneck in the bucket often determines the ceiling of your effort.
The Speed Chasm Between CPU and Memory
Modern CPUs are extremely powerful at computation, but accessing memory is painfully slow:
Performance gap (typical 2024 desktop CPU):
CPU executes one addition: ~1 cycle (≈ 0.3ns @ 3.5GHz)
Access L1 Cache: ~4 cycles (≈ 1.2ns)
Access L2 Cache: ~12 cycles (≈ 3.5ns)
Access L3 Cache: ~40 cycles (≈ 11ns)
Access DRAM: ~200~300 cycles (≈ 60~90ns)
Memory bandwidth: DDR5-4800 ≈ 76.8 GB/s (dual channel)
CPU compute throughput: 3.5GHz × 8 cores × 4 FLOPs/cycle ≈ 112 GFLOPS
Problem:
- Memory latency ~60~90ns, if the CPU waited at full speed, it would spend over 200 days per year waiting for memory
- Although memory bandwidth is high, the latency of individual memory requests improves very slowlySpeed comparison (scaling CPU compute speed to "1 second"):
CPU core compute ─────────── 1 second ───────────
L1 Cache hit ───────────── 4 seconds ────────────
L2 Cache hit ────────────────── 12 seconds ─────────────────
L3 Cache hit ────────────────────────── 40 seconds ──────────────────────
DRAM access ─────────────────────────────────────────── 200 seconds ──────────────────────────────────
Memory latency is 200× slower than L1 — this is the "Memory Wall."Why Can't DRAM Be Made Faster?
"Make DRAM as fast as the CPU" — technically very difficult due to physical limitations:
Root causes of DRAM speed limitations:
1. Physical limitations of DRAM cells:
- Capacitor charging/discharging takes time (RC delay)
- Bit Line capacitance is large, charging/discharging is slow
- Refresh operations consume bandwidth (~4.5%)
2. Physical limitations of the memory bus:
- DDR5 memory channel bandwidth can reach 76.8 GB/s
- But single-request latency is determined by physical distance
- From CPU to memory controller to DIMM slot, distance ~10~20cm
- At the speed of light, 20cm round trip ≈ 1.3ns (minimum latency)
- Actual, including transistor delays and protocol overhead: ~60~90ns
3. Cost tradeoff:
- SRAM Cache is on the CPU die, high power, expensive area (6T/Cell)
- DRAM is a separate chip, cheap, high capacity
- Making all memory from SRAM would be prohibitively expensiveCache Fundamentals: Locality
The existence of Cache is based on two important principles of locality:
Temporal Locality:
- If data has just been accessed, it will likely be accessed again soon
- Example: loop variable i is accessed in every iteration
Spatial Locality:
- If data is accessed, nearby data will likely be accessed soon
- Example: accessing array element a[i], then a[i+1], a[i+2] are likely needed next
Cache is designed to exploit both kinds of locality.Cache Line: The Cache is not managed by "byte" but by "Cache Line" (64 bytes). When the CPU reads data at address A, the cache fetches the entire Cache Line (64 bytes) containing address A — because spatial locality means nearby data is also likely to be needed.
How a Cache Line works:
CPU reads address 0x1000 (1 byte)
→ Cache finds this address is not in Cache (Cache Miss)
→ Reads the entire Cache Line (0x1000 ~ 0x103F, 64 bytes) from DRAM
→ Returns 1 byte to the CPU
→ The remaining 63 bytes stay in Cache
Next CPU read 0x1001:
→ Cache hits (Cache Hit)
→ Returns directly from Cache, only ~1ns
Next CPU read 0x1040 (another Cache Line):
→ Cache Miss
→ Must fetch 64 bytes from DRAM againThree-Cache-Level Hierarchy
Modern CPU cache hierarchy (Intel Core i7-13700K):
┌─────────────────────────────────────────────────────┐
│ CPU Core │
│ ┌──────────────┐ │
│ │ L1 Cache │ 32 KB instruction + 32 KB data │
│ │ ~1.2ns │ Per core, private │
│ └──────┬───────┘ │
│ │ │
│ ┌──────┴───────┐ │
│ │ L2 Cache │ 1.25 MB per core, private │
│ │ ~3.5ns │ │
│ └──────┬───────┘ │
│ │ │
│ ┌──────┴───────┐ │
│ │ L3 Cache │ 30 MB, shared by all cores │
│ │ ~11ns │ │
│ └──────┬───────┘ │
│ │ │
└─────────┼───────────────────────────────────────────┘
│
┌───────┴────────┐
│ Memory Bus │
│ ~60~90ns │
└────────────────┘
Key design choices:
- L1: Small, very fast, per core (instruction + data split)
- L2: Medium size, fast, per core (unified I+D)
- L3: Large, slower, shared by all cores (inclusive or exclusive)
- Each level filters traffic to the next: reduces DRAM accessesInclusion vs. Exclusion Policies
Inclusive Cache (Intel):
L3 contains all data from L1+L2
+ Coherence check is simpler (look up L3)
- Wastes space (duplicate data)
Exclusive Cache (AMD):
L3 contains only data NOT in L1+L2
+ Better capacity utilization
- Coherence check needs to visit all levels
NINE (Non-Inclusive Non-Exclusive, ARM):
No strict inclusion relationship
+ Flexible implementation
- Coherence overhead in betweenCache coherency (MESI protocol):
Modified: This cache has the only modified copy (dirty data)
Exclusive: This cache has the only copy (clean data)
Shared: Multiple caches have the same copy
Invalid: This cache's copy is stale
When one core writes a cache line:
1. Core 1 writes to its L1 (Modified)
2. Hardware sends an invalidation message to other cores
3. Other cores mark their copies as Invalid
4. Next access by other cores triggers a cache miss
5. Read the latest data from Core 1's L1 or memory
Comments