80686-Pentium Pro

SMP Symmetric Multiprocessing

The Pentium Pro introduced support for SMP, allowing multiple CPUs to be used in a single machine. There are two primary ways to implement a multiprocessor system. One is called SMP, or symmetric multiprocessing, and the other is NUMA, or Non-Uniform Memory Access. In an SMP system, multiple CPUs have equal access to resources such as the bus, peripherals, and memory. All CPUs share the same physical memory, and each CPU takes the same amount of time to access any address in memory; this is also known as UMA.
SMP is relatively complex to design. Multiple processors sharing resources such as the bus is not a problem when there are only a few CPUs, but as the CPU count increases, bus bandwidth, bus coherence, and other aspects become much more challenging. Consequently, NUMA architecture is typically used for systems with more CPUs. In a NUMA architecture, each CPU has its own memory. A CPU can also access memory attached to another CPU, but at a relatively slower speed. The operating system is aware of this characteristic and, when assigning memory and computational tasks, tries to place them on nearby CPUs and memory.

Symmetric multiprocessor system architecture

NUMA system architecture

Changes to the Cache

In the 80486, Intel moved the L1 Cache into the CPU. With the 80686 (though it was no longer actually called that), namely the Pentium Pro, Intel also moved the L2 Cache into the CPU. In addition, the L1 Cache was split into separate data and instruction caches: an 8 KB data Cache and an 8 KB instruction Cache.

A memory organization that stores data and instructions separately is called the Harvard architecture, in contrast to the von Neumann architecture, which stores them together.

The Harvard architecture is a memory architecture that separates program-instruction storage from data storage. It is a parallel architecture whose principal characteristic is that programs and data are stored in different storage spaces. In other words, program memory and data memory are two independent memories, each with its own addressing and access.
--Baidu Baike

Outside the CPU, of course, DRAM remains a single memory that does not distinguish between instructions and data, just as the hard drive remains a single drive that makes no such distinction. The x86 CPU can therefore be described as using a Harvard architecture internally while retaining a von Neumann architecture externally.
In fact, apart from a small number of devices such as microcontrollers and DSPs, no one separates data from instructions in the outermost storage devices. This combination of an internal Harvard architecture and an external von Neumann architecture therefore appears to have become an industry consensus.

DIB Architecture

This means that the L2 Cache and DRAM (what we commonly call main memory) buses operate in parallel. In earlier systems, including both systems with only an L1 Cache and systems with an external L2 Cache, data retrieval was serial: data was first fetched from the L1 Cache; after a miss, it was fetched from the L2 Cache; and if the L2 Cache also missed, it was fetched from DRAM.

Because the L2 Cache is slower than the L1 Cache, the latency of an L1 Cache miss can still be ignored relative to the latency of retrieving data from DRAM, but the latency of an L2 Cache miss cannot be dismissed so easily.

To reduce the impact of an L2 Cache miss on total memory-access latency, when an L1 Cache miss occurs, the memory address is sent directly to both the L2 Cache controller and the DRAM memory controller. If the L2 Cache hits, its result is used directly; if the L2 Cache misses, the processor waits for the memory-access result from DRAM. This reduces the overall memory-access time when an L2 Cache miss occurs. Of course, there is also a drawback: when the L2 Cache hits, an unnecessary DRAM access has been performed, increasing pressure on the memory bus.

The image below shows that DRAM and the L2 Cache connect to the Bus Interface Unit (BIU) in parallel.

Instruction-Level Parallelism with ILP=3

The previous article, 80586—The Original Pentium’s Superscalar Architecture, explained that the original Pentium could fetch, decode, execute, and write back two instructions simultaneously. In other words, it had two pipelines and two ALUs and could execute at most two instructions with no data dependencies between them at the same time. This approach is called instruction-level parallelism (ILP, Instruction-Level Parallelism). The Pentium Pro has three such pipelines and can execute at most three instructions simultaneously.

12-Stage Pipeline

The red text in the diagram above marks three units in the pipeline. As the diagram shows, there are 12 units in total:

1. Instruction fetch has three units: IFU1, IFU2, and IFU3.
2. Decode has two units: DEC1 and DEC2.
   Two technologies are involved at the decode stage: microcode and branch prediction.
   DEC1 has three decoders. The decoding result is a sequence of micro-ops—that is, CISC instructions are converted into multiple RISC instructions—and is then passed to DEC2.
   After DEC2 decodes the instructions, if it finds a branch instruction, it performs branch prediction and directs the instruction-fetch unit to fetch instructions from the branch considered more likely.
3. Register allocation table
4. Reorder Buffer, the unit that performs out-of-order execution on instructions
5. DIS, presumably the instruction-dispatch unit
6. EX, the execution portion of the pipeline, which includes the usual ALUs, MMX instruction execution units, floating-point units, and so on.
7. RR
8. WB
9. RET

Large Pages

Processor paging typically supports 4 KB pages. Making each page so small causes problems: the number of page-table entries increases, page tables grow larger, and TLB misses also become more frequent. The processor now also supports large 4 MB pages, which can alleviate these problems to some extent.

Correspondingly, an entry was added to the TLB specifically to cache page-table entries for large pages.

In modern processors, as memory has grown ever larger and server memory capacities can reach several TB, large-page support has advanced to include 1 GB pages.