K08 — Inter-Process Shared Memory: shm and mmap(MAP_SHARED)

Key Insight: The fastest inter-process communication is no communication at all — directly share the same block of memory.


Three Ways to Share Memory Between Processes

Linux supports three mechanisms for inter-process shared memory:

Bash
1. System V Shared Memory (shmget/shmat)
   - Long history, pre-POSIX API
   - Managed through kernel objects (shmid_kernel)

2. POSIX Shared Memory (shm_open/shm_unlink)
   - File mapping based on tmpfs
   - Operated through file descriptors

3. mmap(MAP_SHARED)
   - Anonymous sharing (parent-child processes)
   - File-backed sharing (file as backend)

Commonality of all three:
  - They all map physical pages into the virtual address space of multiple processes
  - Write once, all processes see it immediately (no copying)

Bash
Performance Comparison:

                    System V shm    POSIX shm(/dev/shm)    mmap(MAP_SHARED)
Creation overhead   High (kernel obj) Medium (tmpfs file)      Low (anon) or Medium (file)
Access latency      Same (page table) Same                    Same
Lifetime            Explicit delete   Last ref close          Last mapping close
Debug convenience   ipcs -m          ls /dev/shm             /proc/PID/maps
POSIX standard      No               Yes                      Yes
Cross-node (NUMA)   Supported        Supported               Supported

System V Shared Memory Kernel Path

Bash
shm kernel data structure path:

  Userspace: shmget(key, size, IPC_CREAT|0666)
              ↓
  Kernel: shmid_kernel structure (struct shmid_kernel)
              ↓
          Associated with an anonymous tmpfs file (no real filename)
              ↓
          struct file → struct address_space
              ↓
          Physical pages (from buddy system, added to page cache)

  Userspace: shmat(shmid, NULL, 0)
              ↓
  Kernel: Creates a VMA (Virtual Memory Area) for the current process
              ↓
          VMA.vm_ops → shm_vm_ops (file operations)
              ↓
          Page table mapping: virtual address → struct page → physical page

Key Points:
  - shm physical pages are part of the page cache
  - Can be reclaimed by tmpfs reclaim mechanism (under memory pressure)
  - shmctl(IPC_RMID) only marks for deletion; actual page release happens after the last process detaches

Viewing shared memory (ipcs):
  $ ipcs -m
  ------ Shared Memory Segments --------
  key     shmid   owner   perms   bytes   nattch   status
  0x1234  65536   user    666     1048576   2        dest

  nattch = number of processes attached to this segment
  status = dest means "marked for deletion, waiting for last process to detach"

System V Shared Memory

Bash
Using shmget / shmat:

key_t key = ftok("/some/path", 1);
int shmid = shmget(key, 1024*1024, IPC_CREAT | 0666);
if (shmid == -1) { perror("shmget"); exit(1); }

void *ptr = shmat(shmid, NULL, 0);
if (ptr == (void*)-1) { perror("shmat"); exit(1); }

// Now multiple processes can access this memory simultaneously
strcpy(ptr, "Hello from process A");

// ... after use
shmdt(ptr);

// Only the last process to detach needs to delete it
shmctl(shmid, IPC_RMID, NULL);

POSIX Shared Memory

Bash
POSIX shared memory is file-based:

int fd = shm_open("/myshm", O_CREAT | O_RDWR, 0666);
if (fd == -1) { perror("shm_open"); exit(1); }

ftruncate(fd, 1024*1024);

void *ptr = mmap(NULL, 1024*1024, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (ptr == MAP_FAILED) { perror("mmap"); exit(1); }

close(fd);  // The mapping stays, fd can be closed

// Use ptr for shared memory...

// Cleanup
shm_unlink("/myshm");

Bash
Comparison with System V:

System V shm:
  $ ipcs -m  # List all shared memory segments
  $ ipcrm -m <shmid>  # Remove

POSIX shm:
  $ ls -la /dev/shm/  # View all POSIX shared memory files
  $ rm /dev/shm/myshm  # Remove

mmap(MAP_SHARED) — Anonymous Shared Memory

Bash
Anonymous mmap (MAP_ANONYMOUS | MAP_SHARED), typically used between parent and child:

// Parent process
void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
                  MAP_SHARED | MAP_ANONYMOUS, -1, 0);

pid_t pid = fork();
if (pid == 0) {
    // Child process: can read/write the same memory as parent
    printf("Child reads: %sn", (char*)ptr);
    strcpy(ptr, "Written by child");
    exit(0);
}
// Parent
wait(NULL);
printf("Parent reads: %sn", (char*)ptr);

Bash
File-backed shared mmap:

// An existing file as the backend for shared memory
int fd = open("/tmp/shared.dat", O_RDWR | O_CREAT, 0666);
ftruncate(fd, 4096);

void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
                  MAP_SHARED, fd, 0);
// Multiple processes can mmap the same file → shared memory

Data Structures

Bash
shmid_kernel (kernel internal):

struct shmid_kernel {
    struct kern_ipc_perm    shm_perm;    // Permission information
    struct file            *shm_file;    // Corresponding anonymous tmpfs file
    unsigned long           shm_nattch;  // Number of attached processes
    unsigned long           shm_segsz;   // Segment size
    time64_t                shm_atim;    // Last attach time
    time64_t                shm_dtim;    // Last detach time
    time64_t                shm_ctim;    // Last change time
    pid_t                   shm_cprid;   // Creator PID
    pid_t                   shm_lprid;   // Last operator PID
    struct user_struct     *mlock_user;  // mlock accounting
};

View in /proc:
  $ cat /proc/sysvipc/shm

Practical Considerations

Concern Solution
Synchronization Use mutex/semaphore/futex with shared memory
Cross-platform Prefer POSIX shm or mmap
Security Set permissions correctly, use unique key names
Debugging ipcs, /proc/PID/maps, strace, gdb
Performance Minimize lock contention, align data structures to cache lines
Persistence shm has no persistence (tmpfs is ephemeral)

Bash
Synchronization example (using POSIX semaphore with shared memory):

sem_t *sem = sem_open("/mysem", O_CREAT, 0666, 1);
void *shm = shm_open(...);
// ...
sem_wait(sem);
// Critical section operating on shared memory
sem_post(sem);
Last modified: 2024年3月13日

Author

Comments

Write a Reply or Comment

Your email address will not be published.