What a byte is worth
Silicon ·
The most interesting thing about memory today is how AI has changed its relationship with compute. I think the economic importance of that shift is still underappreciated.
Start with arithmetic intensity: FLOPs performed per byte moved. The roofline model relates this to an accelerator’s compute capability and memory bandwidth. Low-batch autoregressive decode often has too little arithmetic intensity to use the available FLOPs. For these workloads, arithmetic is relatively abundant; keeping it fed is the harder problem.
Batching complicates the picture. Reusing weights across sequences raises arithmetic intensity and can make parts of inference compute-bound. Training and large prefills generally have much higher intensity. But batching cannot amortize everything: each sequence carries its own state, and KV-cache capacity and traffic become increasingly important as concurrency and context grow. Latency requirements also limit how aggressively we can batch.
Memory and FLOPs are complementary inputs to economically useful intelligence. For many personal-computing workloads, extra RAM offers diminishing returns once there is enough. In AI serving, additional capacity can support larger models, longer contexts and more concurrent requests; bandwidth can increase token throughput. Relieving a binding memory constraint can therefore translate directly into revenue, provided paying demand exists.
NVIDIA’s published specifications illustrate the growth in memory resources:
| GPU | HBM capacity per GPU | Memory bandwidth |
|---|---|---|
| H100 SXM | 80 GB | 3.35 TB/s |
| B200 | 180 GB | 8 TB/s |
| Rubin, preliminary HGX specification | 288 GB | Up to 22 TB/s |
H100 figures are published directly; B200 figures follow from the eight-GPU DGX specifications. NVIDIA’s Rubin pages currently differ: its NVL72 table lists 19.2 TB/s per GPU, while HGX lists up to 22 TB/s. These are hardware specifications, not sustained application bandwidth.
I see several reinforcing waves. First, improving the balance between compute and memory makes more of the existing arithmetic capability usable. Second, the entire AI pie grows, increasing demand for both resources even as that balance improves. Third, cheaper and more abundant memory changes which architectures become economical.
That third wave is the hardware-lottery effect: the hardware available to us shapes which ideas succeed. Much of today’s optimization, from FlashAttention to quantization and KV compression, economizes on memory capacity or movement. My expectation is that greater abundance would encourage longer contexts, larger expert pools, persistent state and more aggressive caching. Even when compute becomes the constraint, memory can help by retaining reusable results and avoiding recomputation.
Then there is the potential Jevons effect. Cheaper memory can lower AI costs and expand usage, while new architectures can increase memory consumed per workload. Efficiency gains can be spent the same way: a 4× reduction in KV-cache footprint can fund longer contexts or greater concurrency. Aggregate memory demand rises if that expansion outweighs the savings per workload. The rebound is an economic possibility, not an automatic law.
Energy constraints strengthen the case for better memory. When power is scarce, what matters is useful output per joule across the whole system. Custom HBM base dies and closer integration between memory and logic create more room for differentiation in bandwidth, efficiency and workload suitability. Meeting an interface standard does not make every implementation economically equivalent. Not all memory is equal.
This is why I am skeptical of a simple “supply catches up and the opportunity disappears” argument. Demand responds to price, capability and the architectures we build. Overbuilding and price cycles remain possible, and growing bit demand does not guarantee growing supplier profits.
Memory does not need to remain the permanent bottleneck. Capacity, bandwidth and compute can grow together, with each enabling more useful work from the others. If memory becomes cheaper and more abundant, we may discover how much more of it AI can economically use.