How AI Model Size Drives Memory, and Where CXL and Optics Fit
Why model weights and KV cache outgrow the memory on each accelerator, what CXL memory expansion, pooled memory, and optical links each solve, and what that means for the companies that supply them.
For large AI models, memory is now as hard a limit as computing power.
A large model needs more memory than any single GPU holds, and every active user adds more. That spreads models across many GPUs while memory in other servers sits idle. CXL memory and optical links each solve part of the problem, with trade-offs in speed, cost, and power. Below are the questions we work through with AI infrastructure teams.
The questions we hear from AI infrastructure teams
Why won't our model fit on one GPU?
A model needs memory for every parameter. Llama 3.1 405B needs about 810 GB at 16-bit precision, and the largest GPUs today hold 288 GB. So it has to be split across at least three of them before it serves a single user.[1][2][3][4]
Why does memory run out as usage grows?
Each active conversation keeps its own working memory, called the KV cache. For that model, one long 128K-token conversation takes about 68 GB. Thirty-two of them take more than twice the memory of the model itself.[1][2]
Doesn't a mixture-of-experts model need less memory?
It needs less computation per answer, not less memory. Llama 4 Maverick uses 17B of its 400B parameters for each token, but all 400B must stay loaded, because the next token could need any of them.[5]
Can we just buy GPUs with more memory?
Only up to a point. Memory is fixed when the chip is built, and the largest today carry 288 GB. Past that you buy more GPUs, and the connections between them carry more of the work.[3][4]
Are we using the memory we already pay for?
Often not. Microsoft found that up to 25% of server memory sits unused, and about half of cloud VMs use less than half the memory they rent. Before newer techniques such as PagedAttention, serving software left most of its KV cache memory empty.[6][7]
Where does CXL memory fit?
CXL adds memory to a server beyond its slots, and lets several servers share one pool so idle memory gets used. The cost is speed: about 70 to 90 nanoseconds of extra delay for a pool shared by 8 to 16 processors. Some workloads absorb that and some don't.[6][8][9]
When do we need optical links?
When the memory pool spans more than one rack. Copper cables for PCIe and CXL reach about 2 m; past that the links go optical. PCI-SIG now standardizes PCIe over optics, and NVIDIA rates its co-packaged optics at 5 times the power efficiency of plug-in modules.[10][11][12]
A worked example: Llama 3.1 405B
126 layers, 8 KV heads of dimension 128, and a 128K-token context[1]. Weights are parameters times bytes per parameter; the KV cache is a key and a value per layer, KV head, and token at 2 bytes each[2].
| Item | Memory |
|---|---|
| Weights at BF16 | 810 GB |
| Weights at FP8 | 405 GB |
| Weights at FP4 | 203 GB |
| KV cache per token | 516 KB |
| KV cache, one 128K-token request | 67.6 GB |
| KV cache, 32 such requests | 2.16 TB |
FP4 formats also store scale factors, which add a little to the FP4 figure.
| Accelerator | Memory | Needed |
|---|---|---|
| NVIDIA H100 SXM[13] | 80 GB HBM3, 3.35 TB/s | 11 |
| NVIDIA H200[14] | 141 GB HBM3e, 4.8 TB/s | 6 |
| NVIDIA B300 (Blackwell Ultra)[3] | 288 GB HBM3e, 8 TB/s | 3 |
| AMD Instinct MI355X[4] | 288 GB HBM3E, 8 TB/s | 3 |
| Google TPU7x (Ironwood)[15] | 192 GiB HBM, 7.37 TB/s | 4 |
| AWS Trainium3[16] | 144 GB HBM3e, 4.9 TB/s | 6 |
Vendor specifications as of October 22, 2026. The count covers 810 GB of weights only, before KV cache and activations.
Where each memory tier fits
Latency is given relative to local server memory, because absolute figures depend on the processor and the CXL device.
HBM on the accelerator[3][4]
Reach
On the package
Cost
Highest bandwidth, fixed capacity
Fits
Weights and the active KV cache
Host DRAM
Reach
Same server
Cost
Baseline latency
Fits
Conversation memory moved off the GPU, training state
CXL memory expansion[6][17]
Reach
Same server, module or card
Cost
A little slower than local memory
Fits
More capacity per server than DIMM slots allow
Measure before moving data to a slower tier
In Pond's tests of 158 workloads with memory latency increased by 182%, 43% ran within 5% of their local-memory performance, while more than 21% slowed by over 25%. A remote memory tier helps some workloads and hurts others, so it has to be measured per workload.[6]
What a memory assessment covers
We size the memory your models and users really need, then decide which tier each part belongs in and what links connect them.
Plan a memory assessment- How much memory each model needs for its weights, its users, and their conversation lengths
- Which data can move to slower, cheaper CXL memory without hurting response times
- How big a shared memory pool should be, given the memory sitting idle today
- Whether copper is enough or the pool needs optical links
- Which vendors and standards fit, across CXL, PCIe, and optical interconnect
What this means for memory and CXL suppliers
The memory wall is not only a computer architecture problem. It is one of the largest supplier markets in the AI build-out: high-bandwidth memory alone is estimated at $54.6 billion in 2026, up 58%. But pooled or remote memory is not automatically a win. The commercial question is which workloads, architectures, and customers get enough value from it to change the system design.[18]
HBM and DRAM makers
Every model that outgrows one accelerator adds HBM, and every conversation moved off the GPU adds server DRAM. The case to make is capacity and bandwidth per dollar at the next accelerator generation.
CXL controller and switch companies
Pooling turns stranded memory into usable capacity. The case to make is memory saved per server and the latency each workload accepts, with the processors and switches that support it.
Memory module makers
CXL modules add capacity beyond the DIMM slots. The case to make is cost per gigabyte against adding servers, and reliability in a production fleet.
Optical interconnect companies
Pools that span racks need optical links. The case to make is reach, power per bit, and latency at the pool size the buyer actually needs.
Packaging and test companies
More memory per accelerator means harder packaging and more test. The case to make is yield, capacity, and qualification time.
Serving software companies
Software decides what lives in HBM and what moves to slower tiers. The case to make is cost per answer at the same response time.
The numbers your business case has to move
We turn these engineering numbers into an account-specific case: which hyperscaler teams own the problem, which platform partners you need, and what benchmarks will prove it.
How suppliers win hyperscaler design-ins- Higher accelerator use
- Less memory stranded in individual servers
- Lower cost per answer served
- Latency the workload accepts
- Support from the servers, processors, and switches the buyer already runs
More from this practice
Executives sizing a market
AI Infrastructure Market Map
Where the trillion dollars goes, layer by layer, and which part of it your product can compete for.
Business development and sales leaders
Winning Hyperscaler Design-Ins
The five gates between a first meeting and a design win, and who inside a hyperscaler decides at each one.
Optics and interconnect companies
Photonics Go-to-Market
Where optics create enough system value to replace copper or pluggables, and who has to approve the change.
Power, cooling, and facility suppliers
Power and Cooling Go-to-Market
How time to power and rack density became buying criteria, and how to sell into facility design.
Frequently Asked Questions
Every parameter has to stay in memory while the model answers. At 16-bit precision that is 2 bytes each, so a 405 billion parameter model needs about 810 GB before it serves anyone. That is more than any single GPU holds.
Each conversation keeps its own working memory, called the KV cache. For Llama 3.1 405B that is about 516 KB per token, so one long 128K-token conversation needs about 68 GB on top of the model.
When you run out of memory before you run out of speed: conversation memory moved off the GPU, large lookup tables, or memory sitting idle in individual servers. CXL memory is a little slower than memory in a server's own slots, so test each workload before moving it.
Copper cables for PCIe and CXL reach about 2 m, enough inside a rack. Pools that span racks need optical links, which PCI-SIG now standardizes and which use less power than plug-in optical modules when built into the switch package.
References
Figures checked against these sources on October 22, 2026.
- [1] The Llama 3 Herd of Models (Meta, 2024), Table 3. https://arxiv.org/abs/2407.21783
- [2] Mastering LLM Techniques: Inference Optimization (NVIDIA, 2023). https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
- [3] Inside NVIDIA Blackwell Ultra (NVIDIA, 2025). https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/
- [4] AMD Instinct MI355X specifications. https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html
- [5] The Llama 4 herd (Meta, 2025). https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- [6] Pond: CXL-Based Memory Pooling Systems for Cloud Platforms (ASPLOS 2023). https://arxiv.org/abs/2203.00241
- [7] Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://arxiv.org/abs/2309.06180
- [8] CXL 3.0 specification release (CXL Consortium, 2022). https://computeexpresslink.org/wp-content/uploads/2023/12/CXL_3.0-Specification-Release_FINAL.pdf
- [9] CXL 4.0 specification release (CXL Consortium, 2025). https://computeexpresslink.org/wp-content/uploads/2025/11/CXL_4.0-Specification-Release_FINAL_Website-Copy.pdf
- [10] PCI-SIG CopprLink cable specifications (2024). https://www.businesswire.com/news/home/20240501529875/en/
- [11] PCI-SIG optical interconnect solution for PCIe (2025). https://www.design-reuse.com/news/202528858-pci-sig-reg-announces-pcie-reg-optical-interconnect-solution/
- [12] NVIDIA silicon photonics networking. https://www.nvidia.com/en-us/networking/products/silicon-photonics/
- [13] NVIDIA H100 Tensor Core GPU specifications. https://www.nvidia.com/en-us/data-center/h100/
- [14] NVIDIA H200 Tensor Core GPU specifications. https://www.nvidia.com/en-us/data-center/h200/
- [15] Google Cloud TPU7x (Ironwood) documentation. https://docs.cloud.google.com/tpu/docs/tpu7x
- [16] AWS Trainium. https://aws.amazon.com/ai/machine-learning/trainium/
- [17] Samsung CMM-D CXL memory module. https://semiconductor.samsung.com/cxl-memory/cmm-d/
- [18] 2026 market outlook: focus on the HBM-led memory supercycle, citing a BofA estimate (SK hynix Newsroom, January 2026). https://news.skhynix.com/en/2026-market-outlook-focus-on-the-hbm-led-memory-supercycle/
Planning memory for a large model?
Bring one model, its context lengths, and your expected users. We work out the memory it needs and where each part should live.