020. The Modern GPU (Part 1)

I had actually finished this article, but a problem with Zhihu’s draft system caused the entire saved draft to be lost. So I have decided that from now on, I will always write articles on GitBook first and then copy them to Zhihu.

I would also like to explain my approach to writing these articles. Some people feel that, in my account of graphics card history, I have covered certain unimportant graphics cards while omitting others that were extremely useful, and they do not quite understand why I write this way. In fact, writing these articles is also a learning process for me and a way to answer some questions that have puzzled me in the past.

I came to understand the x86 architecture because I worked with UEFI for a time. UEFI is the replacement for traditional BIOS; software of this kind is collectively called Firmware. When I was learning UEFI, I had many questions because many of the terms I encountered referred not to features of modern CPUs, but to legacy hardware. Because the x86 architecture maintains such strong backward compatibility, Firmware code must incorporate the x86 evolution process: first treating the processor as an 8086, then as an 80386, and so on, enabling new features one by one until all its capabilities are active and certain old features have been disabled. Only then does it become a modern CPU.

The same is true of the GPU. UEFI code has to treat a graphics card as an ISA graphics card, then as a VESA graphics card, and so on. Graphics card technology also evolved incrementally. Classic features from the products of each generation still leave traces in modern GPUs, whereas graphics card technologies that merely added minor innovations to representative products generally left none. As I write these articles, I am also answering some of the questions that accumulated while I was working with UEFI, so I naturally place greater emphasis on legacy graphics cards whose traces remain in modern GPUs. Knowledge of these graphics cards is more important. Of course, learning about technologies used in other graphics cards is also interesting and may prove rewarding, but whether I can organize that material depends on how much time I have. Moreover, I did not live through that era, so my firsthand understanding is less profound. Some readers in the comments have used these old graphics cards, and submissions are welcome. Please feel free to contact me privately.

From 2D to 3D

A monitor can display only two-dimensional images, while the world is three-dimensional. Beyond text and UI interfaces, what people usually want to display corresponds to the real world. A photograph, for example, captures a plane from the three-dimensional world.

When graphics cards supported only 2D display, people had to represent the three-dimensional world in 2D graphics manually. This was extremely time-consuming and labor-intensive, so character movement could be represented by only a limited number of keyframes. Characters typically moved like shadow puppets, without rotation. Such images did not look very lifelike. People wanted machines to convert 3D objects directly into 2D images.

Converting a 3D object in the real world into a 2D image is very simple: just photograph it with a camera. But this is not actually simple, because every angle through which a person rotates corresponds to a different 2D image. You cannot enumerate all the possible movements a person might make and photograph every one of them. There are two difficulties here: first, the person would get tired from holding all those poses; second, the storage requirements would be enormous.

In a virtual world, people can create a 3D character and easily make it perform all kinds of movements. The character never gets tired, can move however one wishes, and can be controlled by the player. Storage is not a problem either, because once a movement has been performed and rendered as an image, the image only needs to be shown to the player for an instant; it does not need to be saved. Both the movement and the “photography” are performed by the machine. The only problem to solve is making this process real-time: after the player or game logic causes the character to perform a particular movement, how can a 2D image be generated immediately? This is the main function of a modern graphics card, or GPU: it takes a 3D object as input and produces a 2D image as output.

The image above is a classic illustration. How is a 2D image generated from a 3D model composed of polygons? This is fundamentally a mathematical problem: namely, what image of the teapot in the illustration would be seen from the camera’s position?

First, we need to understand what the input is. Put simply, a 3D model supplied to a graphics card consists of two parts (though in reality there are more): a set of points and a set of connections. The connections specify which two points form a line, which three points form a triangle, and so on. For reasons that everyone can probably guess, GPUs today generally use triangles to represent models. Each quadrilateral on the teapot above would be represented by two triangles.

Points and lines alone are not enough to represent a richly detailed object. Sometimes a 3D model must be colored or have an image mapped onto it.

The image above briefly describes the operations a 3D model undergoes after it enters the graphics card.

1. In addition to x, y, and z coordinates, each vertex may also have a color value or an image-map coordinate (called a texture coordinate in technical terminology).
2. These vertices may need to undergo transformations in the GPU, such as scaling up, scaling down, or rotation.
3. Connectivity information between input vertices specifies, for example, that two points form a line or three points form a triangle.
4. Rasterization divides lines or triangles—that is, it discretizes a continuous model. Each resulting unit is called a fragment; a fragment describes information such as a pixel’s z-coordinate and color.
5. Interpolation: colors within a triangle or along the interior of a line are interpolated from the colors at its vertices.
6. Based on occlusion relationships, only the unobscured portions are displayed.
   Because the volume of data to be processed is large and the mathematical operations are highly regular, designers created chips with many processing units. Each processing unit is very simple, but a single chip can contain tens of thousands of them. Every unit processes one pixel (merely as an example; this is not strictly accurate), and all the units work together, resulting in very high processing speed.

From Graphics Cards to AI Accelerators

The process above is a fixed pipeline. Users can determine certain aspects of its operation by configuring the graphics card’s registers—for example, which algorithm to use for interpolation.

People later discovered that producing better-looking images and making graphics cards easier to use required some processing stages to be configured more flexibly. Vertex processing, for example, need not be limited to operations such as scaling, while fragment processing can do more than interpolation or texture mapping: it can also generate the color of each pixel through mathematical calculations. Chip designers therefore made the processing components for these stages programmable. Like simple CPUs, they can execute simple code logic. Because all of them perform the same operation, they execute the same code simultaneously, but their input sources and output destinations differ. Each of these processing components is called a core of the graphics card, and the special programming language that runs on it is called a shader language.

Later, people found that certain forms of large-scale data processing also fit this model: a batch of data needs to pass through the same processing steps, but its inputs and outputs differ. A GPU can process such data extremely quickly. Thus, the graphics card became the GPU.

Since so many people were using GPUs for computation, why not design GPUs to be better suited to this kind of computation? As GPU functionality increasingly moved away from display and toward computation, it acquired a new name: GPGPU.

Within this kind of computation is a small subfield called AI. It became very popular, so some chips were designed specifically for AI and dispensed with display functionality altogether, becoming AI accelerators.

The complete evolution was:
Fixed-pipeline graphics card -> programmable-pipeline GPU -> GPGPU -> AI accelerator.

The next section will explain each stage in detail.