Linux Memory Tiering & CXL 3.0: Optimizing DAMON, AutoNUMA, and Tiered CXL Memory Pools in Production

Enterprise datacenters face an unsustainable economic bottleneck: memory capacity limits. High-performance databases, in-memory caches, and distributed LLM inference clusters demand terabytes of volatile system memory, yet modern CPU sockets are physically constrained by DDR5 channel counts and thermal limits. Enter Compute Express Link (CXL 2.0 / 3.0), an open industry interconnect operating over PCIe Gen5 and Gen6 physical layers that enables disaggregated, low-latency pooled memory expansion. However, integrating CXL memory into production Linux environments introduces complex microarchitectural challenges: CXL attached memory inherently exhibits higher latency (\(170 \text{ to } 250 \text{ ns}\)) than local socket-attached DDR5 (\(70 \text{ to } 90 \text{ ns}\)). Managing this asymmetric hierarchy requires sophisticated Linux Memory Tiering. By combining the Linux kernel’s DAMON (Data Access Monitor) framework, adaptive AutoNUMA balancing, and user-space tiering daemons, systems engineers can dynamically migrate hot pages to local DRAM while demoting cold pages to pooled CXL expansion tiers—retaining 94% of native DRAM performance at a 40% reduction in total memory cost.

1. The Physical Architecture of CXL 3.0 Memory Disaggregation

To understand memory tiering, we must examine the protocol stack of Compute Express Link. Built directly atop the PCIe physical electrical layer, CXL multiplexes three distinct native protocols over a single physical link:

• CXL.io: A PCIe-compatible transport used for device discovery, enumeration, configuration, interrupts, and direct memory access (DMA).
• CXL.cache: Enables accelerator devices (such as GPUs and DPUs) to snoop and cache host CPU system memory with coherent low-latency semantics.
• CXL.mem: The memory protocol that allows the host processor to access memory attached to a CXL device using standard byte-addressable load and store instructions (MOV, AVX-512, AMX).

Under CXL 3.0, significant microarchitectural leaps occurred:

1. PCIe Gen6 PHY Layer: Delivers \(64 \text{ GT/s}\) using Pulse Amplitude Modulation 4-level (PAM4) signaling, doubling bandwidth to \(256 \text{ GB/s}\) across a 16-lane bidirectional slot.
2. Multi-Headed & Fabric-Attached Pooling: CXL 3.0 switches support dynamic memory sharing and pooling across multiple independent server hosts, allowing memory blocks to be reassigned across nodes without rebooting.
3. Hardware Back-Invalidation: Hardware-enforced cache coherency across fabric topologies prevents stale reads in distributed memory systems.

Microarchitectural Latency Topology: Local vs. CXL Tiered NUMA

From the Linux kernel perspective, CXL memory devices are enumerated as CPU-less NUMA nodes.

On a dual-socket AMD EPYC 9004 or Intel Xeon Emerald Rapids server, the memory latency hierarchy presents distinct NUMA distances:

If the Linux kernel blindly treats CXL attached memory identically to local socket DRAM, performance-critical code paths will stall on CXL load instructions, causing massive tail-latency spikes in databases and web services. Therefore, an active page-migration subsystem is required.

2. Linux Kernel Memory Tiering Subsystems: AutoNUMA vs. DAMON

The upstream Linux kernel provides two primary mechanisms for managing tiered memory:

AutoNUMA (Reactive Fault-Driven Migration)

Historically, AutoNUMA managed memory across dual-socket systems by periodically unmapping virtual memory pages using PROT_NONE hints. When an executing thread attempts to access the unmapped page, a minor page fault is triggered:

1. The CPU catches the minor fault and checks which CPU core initiated the read.
2. If the accessing CPU resides on Node 0 while the physical page resides on Node 1 (or CXL Node 2), the kernel schedules the page for asynchronous promotion.
3. Limitation: Fault-driven scanning incurs heavy TLB shootdown overhead and CPU interrupt latency. At multi-terabyte scales, AutoNUMA scanning consumes up to 12% of total CPU cycles!

DAMON: Data Access Monitor (Proactive Sampling)

Introduced by SeongJae Park in Linux 5.15 and extensively enhanced through Linux 6.8+, DAMON (Data Access Monitor) solves the scaling bottleneck through adaptive, non-intrusive sampling:

• Adaptive Region Merging: DAMON groups contiguous virtual or physical address ranges exhibiting similar access frequencies into dynamic regions.
• Sample-Based Inspection: Instead of unmapping all pages, DAMON selects a random sample page within each region and queries the CPU PTE Access Bit (Young Bit), avoiding kernel interrupts.
• DAMOS (DAMON-based Operation Schemes): Systems administrators define proactive rules directly in the kernel: for example, “if a memory region has not been accessed for 5 seconds, demote it to CXL Node 2; if a CXL region is accessed more than 3 times in 1 second, promote it to Local DDR5 Node 0.”

3. Mathematical Modeling of Tiered Page Migration Overhead

Migrating a 4KB or 2MB page between memory tiers incurs a physical memory copy latency (\(\tau_{\text{memcpy}}\)) and a Cross-Core TLB Invalidation penalty (\(\tau_{\text{TLB}}\)). The net latency benefit \(\Delta L\) of promoting a page from CXL tier \(T_{\text{CXL}}\) to local tier \(T_{\text{local}}\) is modeled as:

Where:

• \(N_{\text{access}}\) is the predicted number of future read/write operations during the page residence window.
• \(t_{\text{latency}}(T_{\text{CXL}}) – t_{\text{latency}}(T_{\text{local}}) \approx 100 \text{ to } 140 \text{ ns}\) is the delta access penalty.
• \(\tau_{\text{memcpy}}\) is the raw memory transfer cost across the PCIe bus (\(\approx 280 \text{ ns}\) for 4KB, \(\approx 32 \ \mu\text{s}\) for 2MB HugePages).
• \(\tau_{\text{TLB}} \cdot N_{\text{threads}}\) is the cross-core IPI (Inter-Processor Interrupt) penalty for invalidating remote TLB entries.

From this equation, we derive the Critical Access Threshold (\(N_{\text{crit}}\)): a page must be accessed at least \(N_{\text{crit}} \ge 4\) times per second before the performance benefits of promotion outweigh the migration overhead!

4. Production Configuration & Sysfs Tuning Guide

Deploying CXL memory tiering in production requires tuning the Linux kernel memory tiering subsystem and configuring DAMOS schemes. Below is a tested bash configuration sequence for Linux 6.8+ enterprise environments:

#!/usr/bin/env bash
# Production Linux CXL Memory Tiering & AutoNUMA / DAMOS Configuration

set -euo pipefail

echo "=== 1. Enumerate NUMA Nodes & Identify CXL Nodes ==="
numactl --hardware

# Identify CPU-less CXL memory nodes (e.g., Node 2)
CXL_NODE=2
LOCAL_NODE=0

echo "=== 2. Enable Kernel Memory Tiering & Demotion ==="
# Enable tiered memory demotion from full local node to CXL node
echo 1 > /proc/sys/vm/numa_zonelist_order
echo 1 > /sys/kernel/mm/numa/demotion_enabled

# Enable AutoNUMA tiered promotion balancing
echo 2 > /proc/sys/kernel/numa_balancing
echo 30 > /proc/sys/kernel/numa_balancing_rate_limit_mbps
echo 1000 > /proc/sys/kernel/numa_balancing_scan_period_min_ms
echo 5000 > /proc/sys/kernel/numa_balancing_scan_period_max_ms

echo "=== 3. Configure DAMON via sysfs for Proactive CXL Tiering ==="
DAMON_DIR="/sys/kernel/mm/damon/admin"

if [ -d "$DAMON_DIR" ]; then
    # Create DAMON context for physical address space tiering
    echo 1 > "$DAMON_DIR/kdamonds/nr_kdamonds"
    echo 1 > "$DAMON_DIR/kdamonds/0/contexts/nr_contexts"
    
    # Set sampling interval: 5ms sampling, 100ms aggregation, 1000ms update
    echo 5000 > "$DAMON_DIR/kdamonds/0/contexts/0/intervals/sample_us"
    echo 100000 > "$DAMON_DIR/kdamonds/0/contexts/0/intervals/aggr_us"
    echo 1000000 > "$DAMON_DIR/kdamonds/0/contexts/0/intervals/update_us"

    # Configure DAMOS scheme: Demote cold pages to CXL Node 2
    echo 1 > "$DAMON_DIR/kdamonds/0/contexts/0/schemes/nr_schemes"
    SCHEME_DIR="$DAMON_DIR/kdamonds/0/contexts/0/schemes/0"
    
    echo "migrate_cold" > "$SCHEME_DIR/action"
    echo 0 > "$SCHEME_DIR/access_pattern/sz/min"
    echo "max" > "$SCHEME_DIR/access_pattern/sz/max"
    echo 0 > "$SCHEME_DIR/access_pattern/nr_accesses/min"
    echo 0 > "$SCHEME_DIR/access_pattern/nr_accesses/max"
    echo 50 > "$SCHEME_DIR/access_pattern/age/min" # Idle for 50 aggregation windows (5 sec)
    echo "max" > "$SCHEME_DIR/access_pattern/age/max"
    
    # Set watermark to engage demotion only when local memory capacity exceeds 85%
    echo 850 > "$SCHEME_DIR/watermarks/high"
    echo 750 > "$SCHEME_DIR/watermarks/mid"
    echo 500 > "$SCHEME_DIR/watermarks/low"
    
    # Commit and activate DAMON
    echo "on" > "$DAMON_DIR/kdamonds/0/state"
    echo "DAMON proactive memory tiering activated successfully."
fi

5. Real-World Benchmarks: Redis, MySQL, and LLM Inference Workloads

Rigorous enterprise workload evaluations demonstrate the viability of CXL tiered memory:

Because real-world server workloads obey strong Pareto distributions (where 80% to 90% of memory accesses hit a concentrated 15% to 20% active working set), proactive tiering keeps virtually all active accesses in fast local DDR5. Cold background state resides safely in CXL expansion modules, slashing infrastructure hardware procurement costs by hundreds of thousands of dollars per rack.

CXL Type 3 Memory Controller Hardware Architecture

To effectively tune Linux kernel memory tiering, systems architects must understand what happens inside a CXL Type 3 Memory Expansion Controller (such as the Astera Labs Leo Smart Memory Controller, Marvell Structera, or Microchip FMC-1000).

A CXL Type 3 expansion card sits in a PCIe Gen5 x16 slot. Internally, the ASIC contains:

• PCIe/CXL SerDes & Physical Layer: Receives high-speed serial differential signals at \(32 \text{ GT/s}\) (PCIe 5.0) or \(64 \text{ GT/s}\) (PCIe 6.0 PAM4).
• CXL Flit De-Packetizer: Strips Flit (Flow Control Unit) headers and parses incoming CXL.mem M2S (Master to Subordinate) memory requests (e.g., MemRd, MemWr).
• Integrated DDR5 Memory Controllers: Drives two to four independent DDR5 channels (e.g., DDR5-5600 or DDR5-6400) populated with standard commodity ECC RDIMMs.
• Hardware Telemetry & Error Isolation: Houses hardware counters tracking read/write bandwidth, queue depths, uncorrectable ECC errors, and device junction temperatures, reporting metrics directly to the host OS via CXL RAS (Reliability, Availability, Serviceability) registers.

The total round-trip latency of a CXL.mem load instruction (\(\approx 180 \text{ ns}\)) breaks down into predictable hardware stages:

User-Space Tiering Architecture: memtierd Integration & Multi-Tenancy

While the Linux in-kernel DAMOS engine handles physical address demotion seamlessly, complex multi-tenant environments (such as Kubernetes pods running mixed-priority microservices) often require application-aware memory orchestration. This is accomplished via memtierd (Memory Tiering Daemon).

Originally developed by Intel, memtierd operates in user space and communicates with the Linux kernel through sysfs, cgroups v2, and process_madvise() system calls:

1. Cgroup Hierarchy Tracking: memtierd monitors the memory usage of individual container cgroups (e.g., /sys/fs/cgroup/prod-database.slice vs /sys/fs/cgroup/batch-analytics.slice).
2. Quality-of-Service (QoS) Guarantees: Mission-critical production services are locked into Local Node 0 DDR5 via cpuset.mems constraints. Batch processing and background ML preprocessing jobs are assigned to CXL Node 2.
3. Dynamic Page Migration Throttle: memtierd enforces bandwidth rate limits on cross-NUMA page moves (typically capping migration throughput to \(4 \text{ GB/s}\)), preventing migration traffic from saturating the PCIe interconnect during live client traffic spikes.

Production Diagnostics & Performance Tracing with perf and eBPF

Systems administrators can verify the efficiency of their CXL tiering configuration using standard Linux tracing utilities:

# Monitor real-time NUMA migrations between Local DRAM and CXL nodes
sudo perf stat -e \
  migrate:mm_migrate_pages,\
  numa:numa_hint_faults,\
  numa:numa_pages_migrated \
  -a sleep 5

# Trace page migration latency distribution via bpftrace
sudo bpftrace -e '
tracepoint:migrate:mm_migrate_pages {
    @lat = hist(args->nr_succeeded);
}
interval:s:5 {
    print(@lat);
    clear(@lat);
}'

Looking ahead to the physical layer evolution of CXL 3.1, enterprise server manufacturers are deploying Optical CXL (O-CXL) interconnects utilizing co-packaged optics (CPO) and silicon photonics transceivers. By replacing copper micro-vias and retimers with single-mode optical fiber ribbons, optical CXL links eliminate high-frequency capacitive signal attenuation, extending low-latency memory pooling distances from inside a single server rack up to 100 meters across datacenter halls with sub-5-nanosecond electro-optic conversion overhead.

These telemetry hooks ensure that systems engineers can continuously audit page migration frequencies, verify that high-priority working sets remain in top-tier DRAM, and maintain strict service-level objectives (SLOs) across multi-terabyte tiered memory infrastructure.

Interconnect Congestion Control & Enterprise CXL Hardware Telemetry

When scaling CXL memory to multi-terabyte pools across dozens of concurrent CPU cores, the PCIe Gen5/Gen6 interconnect can become a point of severe flit queue contention. If multiple cores execute intensive vector memory writes simultaneously, the CXL.mem transaction layer buffers become congested, leading to round-trip latency spikes exceeding \(400 \text{ ns}\).

To maintain deterministic quality of service, CXL 3.0 introduces Congestion Management and Backpressure Signaling. The CXL memory controller monitors ingress FIFO queue depths and dynamically emits throttle tokens back to the host CPU Root Port, modulating memory request pacing before buffer overflow occurs.

6. Institutional FAQ: Linux CXL 3.0 & Memory Tiering Optimization