7 min read #linux-kernel #memory-management
On this page

Physical Memory Management

Coverage: NUMA → Zone (DMA/DMA32/Normal/Movable) → Buddy Allocator → page allocator API → page struct → memory hotplug Kernel version: 2.6 ~ 6.x

Overview

Virtual memory management answers "which virtual address maps to which physical page," while physical memory management answers a more fundamental question: "Who does this 4KB physical page belong to, is it free or allocated, and from which NUMA node should it be taken?"

The core of physical memory management is the buddy allocator, which organizes free pages into contiguous blocks of different orders, efficiently allocating and reclaiming them through splitting and merging. Above the buddy allocator lies slab/slub (small object allocators), and below it lies the NUMA-aware zone and node topology.

NUMA Architecture

UMA (Uniform Memory Access):
  Consistent latency for all CPUs accessing all memory
  Typical: Single socket, desktop

NUMA (Non-Uniform Memory Access):
  Memory is divided into multiple nodes
  Each CPU has its own local memory (fast access)
  Accessing remote memory is slower (30%~40% slower)
  Typical: Multi-socket servers

View: numactl --hardware
      ls /sys/devices/system/node/

pg_data_t: NUMA Node Descriptor

// include/linux/mmzone.h
typedef struct pglist_data {
    struct zone     node_zones[MAX_NR_ZONES];
    struct zonelist node_zonelists[MAX_ZONELISTS];  // Allocation fallback order
    int             nr_zones;
    unsigned long   node_start_pfn;   // Starting physical page number
    unsigned long   node_present_pages;
    unsigned long   node_spanned_pages;
    int             node_id;
    struct page     *node_mem_map;    // Array of all page structs in this node

    // Reclaim
    struct lruvec   lruvec;          // LRU lists (anonymous/file)
    wait_queue_head_t kcompactd_wait;

    // Statistics
    struct per_cpu_nodestat *per_cpu_nodestats;
} pg_data_t;

Zone: Physical Memory Partitioning

Why Zones Are Needed

The fundamental reason for zones: Hardware limitations prevent certain physical memory from being used for specific purposes

16-bit ISA DMA:  Can only access physical addresses 0~16MB
32-bit PCI DMA:  Can only access 0~4GB (unless IOMMU is used)
32-bit kernel:   Can only directly map 896MB (highmem)
64-bit:          No highmem, but zone structure retains DMA/DMA32 limitations

Zone Types

// include/linux/mmzone.h
enum zone_type {
    ZONE_DMA,       // 0-16MB:    ISA DMA
    ZONE_DMA32,     // 0-4GB:     32-bit PCI DMA
    ZONE_NORMAL,    // Direct mapping (kernel can access directly, linear mapping)
    ZONE_HIGHMEM,   // 32-bit only: Dynamic mapping (no such zone in 64-bit)
    ZONE_MOVABLE,   // Movable pages (used for memory defragmentation/hotplug)
    ZONE_DEVICE,    // PMEM / GPU VRAM / CXL
    __MAX_NR_ZONES
};

struct zone

// include/linux/mmzone.h
struct zone {
    unsigned long   _watermark[NR_WMARK];  // low, min, high watermarks
    unsigned long   managed_pages;         // Number of pages managed by buddy
    unsigned long   spanned_pages;
    unsigned long   present_pages;

    // Per-CPU pagesets (cache hot single pages to avoid taking zone lock)
    struct per_cpu_pages  pcp;

    // Buddy free lists
    struct free_area    free_area[MAX_ORDER];  // order 0~10

    // LRU (anonymous/file)
    struct lruvec       lruvec;

    // Lock: protects free_area
    spinlock_t          lock;
};

Watermarks and Memory Pressure

Watermarks: Three watermarks trigger different intensities of reclaim pages 0 high high watermark kswapd wakes up, starts background reclaim low low watermark Kernel direct reclaim (synchronous, blocking allocation) min min watermark Only emergency allocations are allowed to breach this line Watermarks warn progressively from high to low: falling below high → kswapd background async reclaim; falling below low → kernel synchronous direct reclaim (blocking allocation); Falling below min → only emergency allocations can breach, approaching the memory exhaustion boundary. 
// Watermark check during allocation:
// zone_watermark_ok(zone, order, mark, classzone_idx, alloc_flags)
//   Calculates whether the zone's free pages satisfy the order allocation and do not fall below mark
//   If insufficient → triggers reclaim (kswapd or direct reclaim)

Buddy Allocator

Data Structures

// mm/page_alloc.c
// Each zone has a free_area for each order
// free_area[order]: Linked list of free blocks of size 2^order pages

#define MAX_ORDER 11  // Typically: max 2^10 = 4MB contiguous (1024 × 4KB)

struct free_area {
    struct list_head    free_list[MIGRATE_TYPES];  // Categorized by migration type
    unsigned long       nr_free;                   // Number of free blocks for this order
};

// Migration types (anti-fragmentation):
enum migratetype {
    MIGRATE_UNMOVABLE,  // Kernel data structures (cannot be moved)
    MIGRATE_MOVABLE,    // User pages (migratable → anti-fragmentation)
    MIGRATE_RECLAIMABLE,// Reclaimable (e.g., inode cache)
    MIGRATE_PCPTYPES,   // Per-CPU reserved
    MIGRATE_HIGHATOMIC, // High atomic allocation (emergency)
    MIGRATE_CMA,        // Contiguous Memory Allocator
    MIGRATE_ISOLATE,    // Isolated (memory hotplug / compaction)
};

Core Algorithm

// Allocation:
// alloc_pages(gfp_mask, order) → __alloc_pages()
//   → get_page_from_freelist()  // Try to allocate from a suitable zone
//     → rmqueue()                // Take from buddy free_area
//       → __rmqueue_smallest()   // Find the smallest free block >= order
//         → expand()             // If the found block is larger than needed → split
//              The lower half is put back into buddy free_area

// Example: Request order=1 (2 pages = 8KB)
//   free_area[2] has 16KB free → expand():
//     → Allocate the first 8KB (order 0~1) to the caller
//     → Put the remaining 8KB into free_area[1]

// Freeing:
// free_pages(addr, order) → __free_pages()
//   → __free_one_page()
//     → Check if the buddy (partner) is also free
//     → If buddy is free → merge into an order+1 block → recursive merge
//     → If buddy is not free → put into the corresponding order's free_area

// Buddy check:
//   buddy_pfn = page_pfn ^ (1 << order)  // XOR: PFN of the buddy
//   If buddy is also free and of the same order → merge

GFP Flags

// include/linux/gfp_types.h
// Control of allocation behavior:

// Zone modifiers:
#define __GFP_DMA       // Allocate from ZONE_DMA
#define __GFP_DMA32     // Allocate from ZONE_DMA32
#define __GFP_HIGHMEM   // Allow allocation from HIGHMEM (32-bit only)
#define __GFP_MOVABLE   // Allocate movable type

// Watermark modifiers:
#define __GFP_ATOMIC    // High priority, can use emergency reserves
#define __GFP_HIGH      // Allow use of emergency reserved memory
#define __GFP_MEMALLOC  // Allow ALLOC_NO_WATERMARKS (extremely urgent)

// Reclaim modifiers:
#define __GFP_IO        // Allow IO (write back swap)
#define __GFP_FS        // Allow filesystem operations (clear inode cache)
#define __GFP_DIRECT_RECLAIM  // Allow synchronous reclaim
#define __GFP_RECLAIM   // New name for __GFP_DIRECT_RECLAIM
#define __GFP_NORETRY   // Do not retry

// Common combinations:
#define GFP_KERNEL      (__GFP_RECLAIM | __GFP_IO | __GFP_FS)
                        // Most common: allows sleeping/reclaim/IO
#define GFP_ATOMIC      (__GFP_HIGH)        
                        // Cannot sleep: used in interrupt/atomic context
#define GFP_USER        (__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL)
                        // User space allocation
#define GFP_NOWAIT      (0)                 
                        // Do not wait: return NULL immediately if allocation fails

Per-CPU Pages (PCP)

// mm/page_alloc.c
// Each CPU has a per-CPU 1-page cache (struct per_cpu_pages)
// 
// Why? 
//   Allocating a single page (order 0) is the most common operation
//   If every time you have to take zone->lock (global lock) → severe lock contention
//   PCP acts as L1 cache: locally prefetch a batch of pages
//     → alloc: PCP has it → take directly (lock-free)
//              PCP empty → take a batch from buddy (take zone lock once)
//     → free:  PCP not full → put back into PCP (lock-free)
//              PCP full → put back into buddy (take zone lock once)
//
// Design principle: Batch operations amortize lock overhead

struct page

// include/linux/mm_types.h
// Each physical page has a struct page (4KB page manages 64 bytes → ~1.5% overhead)

struct page {
    unsigned long flags;       // Page flags (PG_locked, PG_dirty, PG_uptodate, ...)

    union {
        // In buddy allocator: linked list node
        struct list_head lru;

        // In slab: slab information
        struct slab *slab_cache;

        // As compound page head: number of tail pages
        unsigned long compound_head;
    };

    union {
        atomic_t _mapcount;    // Number of PTEs mapping this page
        atomic_t _refcount;    // Reference count (get_page/put_page)
    };

    // Pointer to address_space (if this page is part of page cache)
    struct address_space *mapping;
    pgoff_t index;             // Offset in the mapping

    // Private data (interpreted based on usage):
    //   Anonymous page: anon_vma
    //   Slab:   slab information
    //   Buffer: buffer_head
    void *private;
};

struct page is an overloaded structure—the same field has different interpretations depending on the page's usage (buddy free, slab, anonymous memory, page cache, THP compound). The kernel uses bits in flags to distinguish them.

Memory Hotplug

// mm/memory_hotplug.c
// Physically adding/removing memory sticks:
//   online: add_memory() → __add_memory() → initialize page struct → add to buddy
//   offline: remove_memory() → __remove_memory() → migrate used pages → remove from buddy

// DIMM (NVDIMM/CXL):
//   Can be made into ZONE_DEVICE (not managed by buddy)
//   Or made into normal memory (ZONE_NORMAL), allocated by buddy

Debugging

# Buddy allocator status
cat /proc/buddyinfo
# Node 0, zone Normal: 11 4 2 1 0 0 ...
#   order 0: 11 blocks (44KB free as 4KB pages)
#   order 1: 4 blocks  (32KB free as 8KB chunks)

# Zone information
cat /proc/zoneinfo
# Contains: per-zone watermarks, free pages, protection

# NUMA statistics
numastat  # per-node allocation/miss statistics

# Memory fragmentation index
cat /sys/kernel/debug/extfrag/unusable_index
# 1.000 = Fully fragmented, unable to allocate contiguous pages
# 0.000 = No fragmentation

# Triggered reclaim statistics
cat /proc/vmstat | grep -E 'pgsteal|pgscan|compact'

References and Further Reading

  • Kernel Documentation: Documentation/mm/page_alloc.rst, Documentation/admin-guide/sysctl/vm.rst
  • LWN:
    • "The design of the buddy allocator" (lwn.net/Articles/259832/)
    • "Memory management APIs" series
  • Source Files:
    • mm/page_alloc.c — buddy allocator (~7000 lines)
    • include/linux/mmzone.h — zone, pg_data_t, free_area
    • include/linux/mm_types.h — struct page
    • include/linux/gfp_types.h — GFP flags

Keywords: NUMA, Zone, Buddy Allocator, GFP_KERNEL, GFP_ATOMIC, struct page, watermark, kswapd, memory hotplug, compaction

Related pages

© 2026 liz's doc