If you run an enterprise AI application today, chances are overwhelming that you do not own the physical silicon processing your prompts. To keep cloud AI bills manageable, enterprises deploy Large Language Models inside Docker or Kubernetes containers hosted on shared public cloud infrastructure. Cloud orchestration systems like Kubernetes, Slurm, and Run:ai pack multiple client workloads onto single physical GPUs using time-slicing or multi-process services, assuring clients that modern Linux container namespaces and hypervisors provide an unbreachable wall of cryptographic isolation.
That assurance is a dangerous optical illusion. Cybersecurity researchers uncovered LeftoverLocals (CVE-2023-4969): a profound hardware vulnerability in modern Graphics Processing Units that allows an unprivileged, low-level process running in one container to directly read uninitialized memory arrays left behind by a completely different tenant’s preceding AI model. Without breaking a single operating system password or exploiting an application bug, an attacker can siphon proprietary model weights, Key-Value attention vectors, and confidential user chat sessions in plain floating-point numbers.
The Architectural Blindspot: Global VRAM vs. On-Chip Local SRAM
To grasp why LeftoverLocals slipped past hardware security auditors for years, you have to understand the microscopic memory hierarchy of a modern graphics card.
When an operating system kernel protects memory, it manages Global Device Memory (the 24GB to 144GB of GDDR6 or HBM3e VRAM soldered onto the graphics card). Just like host system RAM, global VRAM is mapped through Page Table Entries (PTEs) in the GPU’s Memory Management Unit (MMU). When Tenant A finishes running, the GPU driver marks those global memory pages as inaccessible to Tenant B, preventing direct memory snooping.
However, modern GPUs also contain hundreds of high-speed, on-chip scratchpad memories located directly inside each Streaming Multiprocessor (SM) or Compute Unit (CU). In GPU programming languages like OpenCL, Apple Metal, and Vulkan, these memories are known as Local Memory or Threadgroup Shared Memory:
- Ultra-Low Latency: Local memory sits physically adjacent to the arithmetic logic units (ALUs), operating at single-clock latencies (<2 nanoseconds).
- Massive Bandwidth: Tens of terabytes per second of aggregate bandwidth feeding matrix tensor cores.
- Hardware Reality: It is physical SRAM, typically ranging from 64KB to 160KB per compute unit across dozens of cores.
And here is the fatal hardware flaw: GPU microarchitecture does not zeroize or clear local SRAM registers between kernel context switches!
The Attack Mechanics: Siphoning Floating-Point Tokens
When Tenant A executes an LLM inference pass (e.g., Llama-3 or Mistral), the model loads neural network weights and token embeddings into fast local SRAM to compute matrix multiplications and attention mechanisms. Once Tenant A’s GPU time slice expires, the scheduler suspends Tenant A and launches Tenant B’s compute kernel.
In standard CPU computing, the operating system kernel zeroes out memory registers and cache lines to maintain process isolation. But in graphics processors from vendors including AMD, Apple, and Qualcomm, the hardware simply hands control of the Streaming Multiprocessor to Tenant B with Tenant A’s electrical charges still trapped inside the local SRAM capacitors!
All Tenant B needs to do is run a simple, unprivileged OpenCL or Metal compute shader containing a single malicious maneuver:
- Allocate a large local array in threadgroup shared memory (e.g., `local float buffer[4096];`).
- Crucially, do not initialize or write anything to the array.
- Immediately read the array values back and stream them to host disk or network storage!
Instead of receiving clean zeroes, Tenant B receives a high-fidelity dump of Tenant A’s raw floating-point calculations! In laboratory demonstrations, researchers recovered between 5MB and 181MB of raw residual data per second across a single GPU. By feeding this leaked telemetry into simple reconstructor models, researchers reconstructed complete output sentences from confidential corporate AI chats with over 90% word accuracy!
Vulnerability Telemetry: LeftoverLocals Leak Rates (Trail of Bits Audit)
Empirical laboratory metrics documented across impacted GPU architectures under multi-tenant co-location:
- AMD Radeon RX 7900 XT: Leaked ~5.5 MB of residual local memory per GPU kernel invocation.
- Apple Silicon (M2 / M3 Metal): Recovered intermediate LLM matrix multiplication states directly from threadgroup memory.
- Reconstruction Accuracy: Extracted 7B-parameter LLM weights and attention vectors with >90% cosine similarity.
- Mitigation Performance Cost: Software-level zero-fill scrubbing imposes a 12% to 22% compute throughput penalty.
Why Hardware Vendors Skipped Zeroization: The Framerate Trap
Why did world-class semiconductor architects leave such a glaring vulnerability in their silicon? The answer is rooted in the historical DNA of graphics cards: GPUs were engineered for 3D gaming, not zero-trust multi-tenant cloud security.
In a gaming PC, a single user owns the entire graphics pipeline. Zeroing out 160KB of SRAM across 80 compute units thousands of times every second wastes precious clock cycles and degrades gaming framerates. Hardware engineers deliberately skipped automatic memory zeroization to squeeze out every drop of graphics performance.
When the generative AI boom erupted, enterprise cloud providers retrofitted consumer and workstation GPUs into multi-tenant cloud pods without realizing that the underlying silicon lacked the fundamental hardware isolation primitives mandated by enterprise security.
| Security Domain | Traditional CPU Memory Sandboxing | Vulnerable Multi-Tenant GPU (LeftoverLocals) |
|---|---|---|
| Memory Zeroization | Hardware/OS zeroes pages before reassignment | Local SRAM retains data across context switches |
| Address Isolation | Enforced by hardware MMU & Ring 0 CPU rings | Local memory has NO MMU or page table isolation |
| Cross-Container Read | Impossible without kernel exploit (Ring 0 breach) | Trivially achieved via uninitialized local array read |
| Impacted Assets | Operating system memory, passwords, files | LLM model weights, KV-cache vectors, user prompts |
| Remediation Method | Firmware & OS microcode isolation | Driver-level manual zero-scrubbing or dedicated GPU allocation |
Download Technical Whitepaper (.PDF)
Access our complete engineering dossier: LeftoverLocals (CVE-2023-4969) GPU Memory Isolation Audit with verified laboratory telemetry, mathematical derivations, and architecture diagrams.
The Cloud Defense Playbook: How to Protect Your AI Workloads
Much like the physical memory attacks we explored in our investigation of ZenHammer DDR5 bit-flipping on AMD chips, LeftoverLocals proves that software containers cannot save you if the underlying hardware leaks.
If your organization runs sensitive, proprietary, or regulated AI models in public cloud environments, adopt these immediate hardening rules:
- Enforce Single-Tenant GPU Allocation: Disable GPU time-slicing and multi-instance sharing for confidential workloads. Ensure each physical PCIe card is assigned exclusively to one client pod.
- Verify Driver-Level Zero-Fill Patches: Ensure cloud providers have applied driver updates that inject automated memset zero-scrubbing routines before and after kernel launches (even at the cost of a 10% compute slowdown).
- Audit Open-Source Inference Engines: Ensure inference frameworks (vLLM, Ollama, TensorRT-LLM) explicitly manage threadgroup memory allocation with defensive padding.
LeftoverLocals is a sobering reminder that as artificial intelligence migrates into mission-critical infrastructure, the silicon beneath our code must be scrutinized with zero-trust rigor down to the individual memory transistor.
Frequently Asked Questions
Does LeftoverLocals affect private, dedicated local GPUs?
No. If you run AI models on a local desktop or workstation where only trusted local applications execute, LeftoverLocals cannot be exploited remotely without an attacker first achieving native local execution on your machine. The vulnerability primarily threatens multi-tenant cloud environments where untrusted user pods share a single physical GPU.
Why can’t GPUs simply clear memory with hardware circuitry?
Hardware-level automatic zeroization requires adding dedicated discharge logic to SRAM blocks and stalling the instruction pipeline while registers clear. While newer enterprise server chips are implementing hardware zero-clear features, hundreds of millions of existing GPUs lack this hardware logic and must rely on slower software driver workarounds.
Did Nvidia GPUs suffer from LeftoverLocals?
Nvidia enterprise GPUs (A100, H100) utilize specialized hardware partitioning called Multi-Instance GPU (MIG) that physically isolates memory and compute slices at the silicon level, preventing LeftoverLocals cross-talk. However, unpartitioned consumer GPUs running third-party shared compute services must ensure active driver-level memory scrubbing is enabled.
