Agentic AI represents a fundamental departure from stateless chatbots, moving in the direction of tools capable of managing sophisticated, multi-step workflows. Scaling these requires substantial rethinking of how one designs memory.
As foundation models scale to trillions of parameters and context windows reach millions of tokens, the cost of storing and recalling past information is growing much faster than raw processing power. This imbalance is becoming a serious challenge for organisations that deploy advanced agentic systems.
Many teams are now running into a fundamental infrastructure limitation: the vast size of long-term inference memory (so-called Key-Value (KV) cache), which is starting to outgrow the capabilities of existing hardware architectures.
Today’s systems come with an uncomfortable trade-off. Inference context can be maintained in limited high-performance GPU high-bandwidth memory (HBM), but this is prohibitively expensive at scale, or pushed into conventional storage layers, introducing delays too large for real-time agentic behaviour. Neither option is very appropriate for large, persistent contexts.
To close the growing gap that is blocking productive agentic AI, NVIDIA has introduced Inference Context Memory Storage (ICMS) as part of its Rubin architecture. This new platform presents a dedicated memory tier explicitly engineered for AI workloads that create rapidly changing, short-lived, and high-volume memory data.
“AI is not only the greatest software upgrade in computer history, the computing model isn’t simply your devices or cellphones sustaining you with computing, these things are being absorbed into you, physically and organizationally,” said NVIDIA CEO Jensen Huang. “AI systems are evolving beyond simple chatbots into intelligent partners that can understand the real world, reason over extended timeframes, remain fact-aware, utilizing software tools to enrich work experience in productive ways and maintaining content in both short and long term memory.”
Read Also: What Are Bayesian Networks?
At the core of this challenge is the way transformer models operate. To avoid re-computing the whole conversation or task history every time a new token is produced, such models now also store intermediate states in memory (KV cache). In agentic workflows, this cache effectively becomes a form of persistent memory shared across tools and sessions, expanding continuously as interactions grow longer.
Such behaviour establishes a new class of data. Unlike regulated records or archival datasets, KV cache is generated on the fly and exists primarily to maintain performance. It doesn’t require the extreme durability, redundancy, or metadata overhead of enterprise storage systems. Yet, traditional storage stacks (typically controlled by general-purpose CPUs) continue to be burdened with these costs, inefficiently wasting resources towards replication and bookkeeping that agentic AI workloads do not need.
Overall performance degrades dramatically when operating context data is transferred from the GPU memory space (G1) to system memory space (G2) and into shared storage layers (G4). After the working context has passed through the G4 layer, access delays span the range of milliseconds, and the energy consumed per generated token rises significantly. During these delays, high-cost GPUs remain underutilised, waiting for data instead of performing inference.
For businesses, this inefficiency translates into an inflated Total Cost of Ownership (TCO). A growing share of power and resources is consumed by the memory movement and storage overhead, rather than actual AI reasoning, driving up operational costs without delivering proportional performance gains.
A new memory tier for the AI factory
To overcome this limitation, the industry is introducing a specialised layer within the memory hierarchy. NVIDIA’s ICMS platform adds what can be described as a “G3.5” tier—an Ethernet-attached flash memory tier designed specifically for supporting inference at a massive scale.
In this design, storage is directly bound with the compute pod. With NVIDIA BlueField-4 data processing unit, burden of managing the inference context is offloaded from the host CPU. This architecture ensures that each pod can access petabytes of shared memory allowing agentic AI systems to maintain extensive historical context without consuming costly GPU high-bandwidth memory, thereby improving scalability and efficiency.
The benefits become clear when we look at energy consumption. This architecture eliminates the inefficiencies linked to general-purpose storage protocols. As a result, it achieves power efficiency that is up to five times better than traditional methods.
Integrating the data plane
However, adopting this new design means that IT teams rethink traditional approaches to storage networking. NVIDIA Spectrum- X Ethernet is enabling the ICMS platform to provide high-bandwidth, low-latency performance that can efficiently access flash storage in a manner that closely resembles local memory.
The orchestration layer is the central point of collaboration for enterprise infrastructure teams. Workflow of KV data blocks across different memory tiers is under the managing of software frameworks such as NVIDIA Dynamo and its Inference Transfer Library (NIXL).
These orchestration tools cooperate with the storage system to ensure that necessary context data is placed into GPU memory (G1) or system memory (G2) on time when the AI model requires it. To facilitate this, NVIDIA’s DOCA, as a provisioning framework, has introduced its own KV communication layer, which focuses primarily on the inference context cache as a core system resource rather than an afterthought.
Storage vendors are already adapting their platforms to support this emerging architecture. The BlueField-4 will be supported by industry players, including AIC, Cloudian, DDN, Dell Technologies, HPE, Hitachi, Vantara, IBM, and Nutanix, who are already developing offerings based on the adapter. These offerings are expected to reach the market in the latter half of the year.
Redefining infrastructure for scaling agentic AI
Introducing a dedicated tier for context memory has immediate impacts on both capacity planning and overall datacentre architecture.
Redefining data categories:
Technology leaders must begin treating KV cache as its own class of data. It exists in a space that is temporary yet extremely sensitive to latency, making it fundamentally different from long-lived compliance records or archival datasets. The G3. 5 layer manages this quickly changing context data, whereas the traditional G4 storage could concentrate on keeping durable logs, checkpoints, and historical artefacts.
Advanced orchestration requirements:
Efficient deployment relies on smart software allocation. Topology-aware orchestration, enabled through platforms like NVIDIA Grove, ensures that workloads are scheduled near where their cached context resides. This alleviates unnecessary data transmission among the network fabric and improves overall system responsiveness.
Power and density considerations:
By packing more usable memory capacity into the same physical rack space, organisations can maximise existing data center investments and delay costly expansions. At the same time, with increasing compute density per square metre, it places greater demands on cooling systems and power delivery, making careful infrastructure planning essential.
Read Also: What Is Decision Intelligence?
Final Thought
The shift toward agentic AI is directly impacting the layout of data centers. Traditional designs that entirely separate compute from slow, persistent storage are no longer adapted for agents that rely on rapid access to extensive contextual memory.
By adding a specialised context memory layer, enterprises can separate the expansion of model memory demands from the rising cost of GPU high-bandwidth memory. This agentic AI architecture enables multiple agents to draw from a shared, energy-efficient memory pool, reducing the cost of handling complex workloads while also increasing reasoning throughput.
As organizations plan their next wave of infrastructure upgrades, evaluating the performance and efficiency of the memory hierarchy will be just as important as the GPU selection itself.
