Why Did GPUs Become So Valuable in the Age of AI?

As generative AI has rapidly advanced, the GPU has become far more than a simple graphics device. It is now a critical national-level resource. Training an AI model or responding simultaneously to the questions of countless users requires enormous computation. The device that handles this computation quickly is the GPU.

CPUs can also perform computation, so why has the GPU become especially important? To understand this, we first need to look at what AI actually calculates and how.

What Does AI Actually Calculate?

AI converses, translates, and generates images. But internally, nearly every task boils down to numerical calculation.

Take a text AI as an example. When generating a sentence, the following process repeats internally:

1. The input sentence is converted into a number array.
2. This number array is multiplied by a massive number table stored inside the model.
3. A new number array is produced.
4. This process is repeated across multiple stages.
5. Finally, the score for the next word is calculated.

These "multiple stages" are called layers. Each layer is a computation step that receives an input number array and transforms it into a new number array. Because many such layers are stacked on top of one another, the approach is named deep learning, where "deep" refers to the depth of these stacked layers.

In other words, AI is less a machine that thinks complex thoughts than a system that repeatedly calculates enormous number arrays across many stages.

What Role Does a CPU Play?

The CPU is the central unit of a computer. It executes programs, evaluates conditions, and manages the order of many tasks.

A CPU has the following characteristics:

• Can handle a wide variety of tasks
• Strong at complex conditional logic
• Well suited to processing a single task with precision

The CPU is closer to a "manager." It excels at coordinating the flow of programs and controlling multiple tasks.

But AI computation has a different character. AI repeats the same forms of multiplication and addition millions of times. For this kind of calculation, the ability to process the same operation simultaneously in large quantities matters more than the ability to orchestrate sequential steps.

What Makes GPUs Different?

The GPU was originally built to render graphics. A screen contains millions of pixels, and those pixels must be calculated simultaneously to produce smooth visuals. So GPUs were designed from the start with a "process many calculations at the same time" architecture.

The GPU's characteristics are:

• Can perform the same calculation in massive parallel quantities simultaneously
• Specialized for large-scale numerical computation
• Exceptionally strong at repetitive operations like matrix multiplication

The operation most heavily used in deep learning is matrix multiplication. Matrix multiplication can be broken down into an enormous number of small multiplications and additions that can all be computed simultaneously. This structure is a natural fit for the GPU.

Understanding CPUs and GPUs Through an Analogy

Imagine taking a math exam in a classroom.
There are two possible approaches.

The first approach: one extraordinarily skilled math professor answers every problem personally. Each problem is understood deeply and solved precisely through complex reasoning, but only one problem can be solved at a time.

The second approach: 1,000 students who each know basic arithmetic each take one problem and solve them all simultaneously. Each individual is less skilled than the professor, but the overall speed can be overwhelming.

The CPU resembles the first approach.
The GPU resembles the second.

Most of the computation AI requires is not "complex thinking" so much as repeating the same type of calculation at enormous scale. This is why the GPU architecture is far more advantageous.

Why Does AI Training Require So Many GPUs?

AI has two phases: training and inference.

Inference is the process of a finished model calculating an answer once. Training is the process of the model continuously modifying itself. Because of this difference, training requires far more computation.

As we learned in the previous chapter, training repeats the following process:

1. Receive input and produce a prediction.
2. Compare against the correct answer to calculate how far off it was.
3. Slightly adjust the internal numbers (weights) to reduce that error.

This process is not done just once. It is repeated across enormous amounts of data. For example, if tens of millions of sentences are used, this calculation is performed tens of millions of times or more. And the entire dataset may be processed multiple times.

Furthermore, large models have tens of billions or more of numbers internally. Each time training occurs, this massive numerical structure shifts slightly. In other words, far more computation is required than "calculating an answer once."

Training is therefore generally far heavier than inference. To complete this enormous computation within a realistic timeframe, the ability to process the same calculation simultaneously in large quantities is essential. Because GPUs excel at this repetitive, large-scale numerical computation, AI training requires especially many GPUs.

Why GPUs Are Needed at the Inference Stage Too

In the past, GPUs were primarily needed for training. Training requires continuously updating model weights to reduce error, which demands enormous computation and thus made GPUs essential. But as generative AI has become mainstream, the situation has changed. Now, demand for GPUs has surged not just in training but also in inference, the stage where a model generates a response when a user asks a question.

The biggest reason is the explosive growth in usage itself. In the past, relatively few people used AI models. Now, AI is called upon simultaneously across services like search, writing, translation, coding, and customer support. When user numbers grow, what was "one response at a time" becomes "thousands of responses simultaneously." At this point, each request requires the server to perform enormous matrix operations, making parallel computing capability critical.

Additionally, users now require longer conversations. Beyond answering a short one-line question, there are increasingly many cases where a response must be generated while incorporating a long document or a project context. The longer the context, the more information the model must reference for each response, and the heavier the computation. A single inference is getting progressively more expensive.

On top of this, AI is advancing in the direction of multimodal processing, handling not just text but also images, audio, and video. For example, analyzing an image and describing it, converting voice input to text, and then synthesizing that answer back into speech requires more computation than text alone. When video with a time axis is included, the required operations increase further. The GPU computation required at the inference stage has therefore grown far larger than before.

Finally, most services demand real-time response. Users do not expect AI to answer "a few minutes later." Results must appear within seconds, and response speed must be maintained even when many users arrive simultaneously. Meeting these conditions requires not only performing a single inference quickly, but also processing multiple requests simultaneously. Because GPUs excel at exactly this parallel processing, they have become core infrastructure for the inference stage as well.

In summary, GPUs matter at the inference stage not because "inference is lightweight," but because inference has become more frequent, longer, more complex, and must be faster.

Why GPUs Became So Valuable

The reason GPU prices have skyrocketed is not simply "because they are good components." The explosive growth of generative AI drove a rapid surge in GPU demand, while supply could not keep pace. This imbalance is what made GPUs so valuable.

First, the spread of generative AI itself exploded GPU demand. Many companies wanted to train new large models, and many more wanted to deploy already-trained models as services. Both training and inference require GPUs, so overall market demand grew simultaneously.

Furthermore, as models grow larger, the number of GPUs needed does not simply increase a little; it scales dramatically. As model parameters (weights) increase and the context length that must be processed grows, required computation and memory demands grow together. As a result, running even the same service requires more GPUs. "Building and operating a smarter model" became structurally synonymous with "needing more GPUs."

On the supply side, high-performance GPUs are difficult to produce quickly. High-performance chips have extremely complex manufacturing processes, and expanding production facilities takes a long time. Moreover, demand for AI-suitable high-performance GPUs tends to concentrate on specific generations and product lines, intensifying supply bottlenecks. When demand explodes but supply cannot scale rapidly in the short term, prices respond with greater sensitivity.

And it does not end there. Buying more GPUs does not complete the picture. The data center infrastructure to house them must also be in place. GPUs consume large amounts of power and generate substantial heat. Power supply, cooling systems, rack space, network connectivity: the entire data center must expand in tandem. Including not just the per-unit cost of GPUs but also "operating costs," GPUs become not just a component but a bottleneck for the entire infrastructure.

The reason GPUs became so valuable is clear. The computational demands of the AI industry grew sharply, while production of GPUs (the core resource responsible for that computation) and expansion of infrastructure remained limited. GPUs have therefore become not just hardware, but the critical resource that determines the speed and competitiveness of the AI industry.

Summary

AI is a system that repeatedly calculates enormous number arrays across many layers. This computation involves performing the same operations at massive scale, and GPUs are optimized for exactly this parallel computation.

CPUs are strong at coordinating and managing diverse tasks; GPUs are strong at processing large-scale numerical computation simultaneously. The reason GPUs have become core infrastructure in the era of generative AI is precisely this computational structure.