What a byte is worth

The role of memory in AI systems and its interaction with compute have evolved profoundly over time. I believe we are underestimating the significance of this shift for intelligent systems and the broader economy. Let’s start from the bottom and view it through the lens of arithmetic intensity and machine balance.

In small-batch autoregressive decoding, the workload’s arithmetic intensity is typically below the accelerator’s machine balance, so memory bandwidth limits overall performance before the compute capacity can be fully utilized.

Larger batches add complexity to this scenario, since the same weights can be used for multiple sequences, increasing arithmetic intensity, resulting in certain components of inference being compute-limited. However, put simply:

• While larger batches amortize weight reads across sequences, each sequence still carries its own KV-cache storage and access costs.
• As we have larger batches, we also have more state information to store (memory capacity) and move around (memory bandwidth-related).
• Latency requirements imposed upon the system limit how large those batches can be.

Memory and FLOPs (compute) represent two complementary inputs required to generate economically valuable intelligence. For many personal computing workloads, additional and faster memory provide diminishing returns beyond a certain point. However, in the context of intelligence systems, additional memory enables larger models, longer contexts, and higher levels of concurrency. Greater memory bandwidth can increase token throughput when bandwidth is limiting. Consequently, more fast memory can directly increase revenue (assuming demand exists).

NVIDIA’s published specifications provide illustrations of the increase in memory resources:

I envision five reinforcing forces shaping memory demand and value.

1. Better hardware utilization: Improving the balance between memory and compute unlocks more of the already available compute capacity (the arithmetic intensity and machine balance mismatch).
2. Underlying Industry Growth: Expanding AI adoption increases demand for both memory and compute independent of improvements in the balance between the two.
3. Efficiency can expand usage further: Basically the Jevons Paradox point through three avenues.
   • Cheaper memory lowers serving costs and encourages more AI consumption.
   • Architectural efficiencies allow reduction in KV-cache footprint, allowing longer contexts and more concurrency.
   • Energy efficiency improvements across serving, model, and hardware help support more demand even with limited energy availability.
4. New architectures become viable: Abundant memory changes what is economical to build, potentially encouraging designs that make greater use of memory, thereby changing the arithmetic intensity of future workloads.
5. Specialization: Customization and tighter integration between accelerators and memory can enable workload-specific differentiation even amongst implementations meeting the same interface standard.

Overbuilding and the traditional cyclical aspects of memory demand and supply can remain risks. Rising demand for memory hardware does not guarantee profits to suppliers, but memory does not have to remain the main bottleneck in the AI stack to ensure its economic importance. Memory capacity, memory bandwidth, and compute can all advance together unlocking architectures and applications that are impractical today. Efficient and specialized memory can make intelligence more economical, in turn making more memory worth using.