Performance gap (typical 2024 desktop CPU):

CPU executes one addition:    ~1 cycle (≈ 0.3ns @ 3.5GHz)
Access L1 Cache:             ~4 cycles (≈ 1.2ns)
Access L2 Cache:             ~12 cycles (≈ 3.5ns)
Access L3 Cache:             ~40 cycles (≈ 11ns)
Access DRAM:                 ~200~300 cycles (≈ 60~90ns)

Memory bandwidth: DDR5-4800 ≈ 76.8 GB/s (dual channel)
CPU compute throughput: 3.5GHz × 8 cores × 4 FLOPs/cycle ≈ 112 GFLOPS

Problem:
  - Memory latency ~60~90ns, if the CPU waited at full speed, it would spend over 200 days per year waiting for memory
  - Although memory bandwidth is high, the latency of individual memory requests improves very slowly

Speed comparison (scaling CPU compute speed to "1 second"):

CPU core compute         ─────────── 1 second ───────────
L1 Cache hit             ───────────── 4 seconds ────────────
L2 Cache hit             ────────────────── 12 seconds ─────────────────
L3 Cache hit             ────────────────────────── 40 seconds ──────────────────────
DRAM access              ─────────────────────────────────────────── 200 seconds ──────────────────────────────────

Memory latency is 200× slower than L1 — this is the "Memory Wall."

Root causes of DRAM speed limitations:

1. Physical limitations of DRAM cells:
   - Capacitor charging/discharging takes time (RC delay)
   - Bit Line capacitance is large, charging/discharging is slow
   - Refresh operations consume bandwidth (~4.5%)

2. Physical limitations of the memory bus:
   - DDR5 memory channel bandwidth can reach 76.8 GB/s
   - But single-request latency is determined by physical distance
   - From CPU to memory controller to DIMM slot, distance ~10~20cm
   - At the speed of light, 20cm round trip ≈ 1.3ns (minimum latency)
   - Actual, including transistor delays and protocol overhead: ~60~90ns

3. Cost tradeoff:
   - SRAM Cache is on the CPU die, high power, expensive area (6T/Cell)
   - DRAM is a separate chip, cheap, high capacity
   - Making all memory from SRAM would be prohibitively expensive

Temporal Locality:
  - If data has just been accessed, it will likely be accessed again soon
  - Example: loop variable i is accessed in every iteration

Spatial Locality:
  - If data is accessed, nearby data will likely be accessed soon
  - Example: accessing array element a[i], then a[i+1], a[i+2] are likely needed next

Cache is designed to exploit both kinds of locality.

How a Cache Line works:

CPU reads address 0x1000 (1 byte)
  → Cache finds this address is not in Cache (Cache Miss)
  → Reads the entire Cache Line (0x1000 ~ 0x103F, 64 bytes) from DRAM
  → Returns 1 byte to the CPU
  → The remaining 63 bytes stay in Cache

Next CPU read 0x1001:
  → Cache hits (Cache Hit)
  → Returns directly from Cache, only ~1ns

Next CPU read 0x1040 (another Cache Line):
  → Cache Miss
  → Must fetch 64 bytes from DRAM again

Modern CPU cache hierarchy (Intel Core i7-13700K):

┌─────────────────────────────────────────────────────┐
│                    CPU Core                          │
│  ┌──────────────┐                                   │
│  │    L1 Cache  │  32 KB instruction + 32 KB data   │
│  │   ~1.2ns     │  Per core, private                │
│  └──────┬───────┘                                   │
│         │                                            │
│  ┌──────┴───────┐                                   │
│  │   L2 Cache   │  1.25 MB per core, private        │
│  │   ~3.5ns     │                                   │
│  └──────┬───────┘                                   │
│         │                                            │
│  ┌──────┴───────┐                                   │
│  │   L3 Cache   │  30 MB, shared by all cores       │
│  │   ~11ns      │                                   │
│  └──────┬───────┘                                   │
│         │                                            │
└─────────┼───────────────────────────────────────────┘
          │
  ┌───────┴────────┐
  │   Memory Bus    │
  │   ~60~90ns     │
  └────────────────┘

Key design choices:
  - L1: Small, very fast, per core (instruction + data split)
  - L2: Medium size, fast, per core (unified I+D)
  - L3: Large, slower, shared by all cores (inclusive or exclusive)
  - Each level filters traffic to the next: reduces DRAM accesses

Inclusive Cache (Intel):
  L3 contains all data from L1+L2
  + Coherence check is simpler (look up L3)
  - Wastes space (duplicate data)

Exclusive Cache (AMD):
  L3 contains only data NOT in L1+L2
  + Better capacity utilization
  - Coherence check needs to visit all levels

NINE (Non-Inclusive Non-Exclusive, ARM):
  No strict inclusion relationship
  + Flexible implementation
  - Coherence overhead in between

Cache coherency (MESI protocol):

  Modified:   This cache has the only modified copy (dirty data)
  Exclusive:  This cache has the only copy (clean data)
  Shared:     Multiple caches have the same copy
  Invalid:    This cache's copy is stale

  When one core writes a cache line:
  1. Core 1 writes to its L1 (Modified)
  2. Hardware sends an invalidation message to other cores
  3. Other cores mark their copies as Invalid
  4. Next access by other cores triggers a cache miss
  5. Read the latest data from Core 1's L1 or memory