Linux System Programming: Complete Deep-Dive Tutorial

📘 The Complete Tutorial

Introduction: The Boundary Between User and Kernel

Linux system programming is the art of writing software that lives at the boundary between user space and kernel space. Every file you open, every process you spawn, every byte you send over a socket — all of it crosses this boundary via system calls.

Understanding this layer transforms you from a programmer who uses the OS into an engineer who understands it.

Module 01: Linux Architecture

Linux follows a monolithic kernel architecture with a clear separation:

┌─────────────────────────────────────┐
│           User Space                 │
│  ┌─────────┐ ┌─────────┐ ┌──────┐ │
│  │  Shell  │ │  Apps   │ │ libc │ │
│  └────┬────┘ └────┬────┘ └──┬───┘ │
│       └─────────────┴─────────┘     │
│              System Calls            │
├─────────────────────────────────────┤
│           Kernel Space               │
│  ┌─────────┐ ┌─────────┐ ┌──────┐ │
│  │  VFS    │ │ Scheduler│ │ MMU │ │
│  │  Network│ │  Memory  │ │ IPC │ │
│  └─────────┘ └─────────┘ └──────┘ │
├─────────────────────────────────────┤
│            Hardware                  │
│         CPU · RAM · Devices          │
└─────────────────────────────────────┘

Key principle: User space cannot access hardware directly. It must ask the kernel via system calls.

Module 02: User Space

User space is where your application runs. It has:

• Restricted memory access — can only touch its own virtual address space
• No direct hardware access — no reading physical memory or I/O ports
• Preemptive multitasking — the kernel can pause your process at any time
• Standard libraries — libc, libpthread, etc.

Every user-space program starts with main() and ends with exit(). Everything in between is either pure computation or a system call.

Module 03: Kernel Space

The kernel is the privileged core of the OS. It has:

• Full hardware access — direct memory, I/O ports, CPU registers
• Process scheduling — decides which process runs and for how long
• Memory management — virtual memory, page tables, swapping
• Device drivers — translates generic system calls to hardware-specific operations
• System call handler — the gateway from user to kernel space

Context switch cost: Entering and leaving kernel space takes hundreds of CPU cycles. Batch your system calls when possible.

Module 04: Processes

A process is an instance of a running program. It consists of:

View processes: ps aux, top, htop, /proc/[pid]/

Module 05: Threads

A thread is a lightweight execution unit within a process. All threads in a process share:

• Text, data, and heap segments
• Open file descriptors
• Signal handlers

Each thread has its own:

• Stack
• Register set (including PC/RIP)
• Thread-local storage (__thread variables)

Linux implementation: Threads are implemented as processes that share memory (clone() with CLONE_VM).

Module 06: fork()

fork() creates a copy of the current process. The child gets a new PID but inherits everything else.

cCopy
1pid_t pid = fork();
2if (pid == 0) {
3    // Child process
4    printf("I am child, PID=%d\n", getpid());
5} else if (pid > 0) {
6    // Parent process
7    printf("I am parent, child PID=%d\n", pid);
8} else {
9    perror("fork failed");
10}

Copy-on-Write (COW): Pages are shared until either process writes to them, then a physical copy is made. This makes fork() fast.

Module 07: exec()

exec() replaces the current process image with a new program. It does not create a new process — it transforms the existing one.

cCopy
1execl("/bin/ls", "ls", "-la", NULL);
2// If execl returns, it failed
3perror("execl failed");

Common variants:

• execl() — list of arguments
• execv() — array of arguments
• execvp() — searches PATH
• execve() — specifies environment variables

The fork() + exec() pattern: This is how shells spawn commands — fork() to create a child, then exec() to run the program.

Module 08: wait()

wait() and waitpid() allow a parent to synchronize with a child's termination.

cCopy
1pid_t pid = fork();
2if (pid == 0) {
3    sleep(2);
4    exit(42);  // Child exits with code 42
5} else {
6    int status;
7    waitpid(pid, &status, 0);
8    if (WIFEXITED(status)) {
9        printf("Child exited with code %d\n", WEXITSTATUS(status));
10    }
11}

Zombie processes: A child that exits before the parent calls wait() becomes a zombie (defunct) until reaped.

Module 09: Pipes

A pipe is a unidirectional byte stream between two processes.

cCopy
1int pipefd[2];
2pipe(pipefd);  // pipefd[0]=read end, pipefd[1]=write end
3
4pid_t pid = fork();
5if (pid == 0) {
6    close(pipefd[0]);          // Child doesn't read
7    dup2(pipefd[1], STDOUT_FILENO);  // Redirect stdout to pipe
8    execlp("ls", "ls", NULL);
9} else {
10    close(pipefd[1]);          // Parent doesn't write
11    char buf[1024];
12    read(pipefd[0], buf, sizeof(buf));
13    printf("Output: %s\n", buf);
14    wait(NULL);
15}

Named pipes (FIFOs): mkfifo("/tmp/myfifo", 0666) — persist on filesystem.

Module 10: Signals

Signals are asynchronous notifications sent to processes.

cCopy
1void handler(int sig) {
2    printf("Caught signal %d\n", sig);
3}
4
5signal(SIGINT, handler);       // Simple, not recommended
6sigaction(SIGINT, &sa, NULL);  // Robust, preferred

Signal safety: Only async-signal-safe functions can be called in a signal handler (write(), _exit(), not printf() or malloc()).

Module 11: Shared Memory

Shared memory allows multiple processes to access the same physical memory region.

cCopy
1int shm_fd = shm_open("/myshm", O_CREAT | O_RDWR, 0666);
2ftruncate(shm_fd, 4096);
3
4void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, 0);
5strcpy((char*)ptr, "Hello from shared memory!");

Fastest IPC: No kernel copy — processes read/write directly to the same page.

Module 12: mmap()

mmap() maps files or devices into memory.

cCopy
1// Memory-map a file
2int fd = open("data.bin", O_RDONLY);
3void *addr = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
4// Access via pointers: int *data = (int*)addr;
5munmap(addr, file_size);

Use cases:

• File I/O without read()/write() system calls
• Shared memory between processes (MAP_SHARED)
• Allocating anonymous memory (MAP_ANONYMOUS)
• Loading shared libraries

Module 13: Files

In Linux, everything is a file. This includes:

• Regular files (-)
• Directories (d)
• Devices (b block, c character)
• Sockets (s)
• Pipes (p)
• Symbolic links (l)

Module 14: File Descriptors

A file descriptor (fd) is a small non-negative integer representing an open file.

cCopy
1int fd = open("file.txt", O_RDONLY);
2read(fd, buf, sizeof(buf));
3close(fd);

File descriptor table: Each process has its own table. fork() duplicates it. exec() preserves it (unless O_CLOEXEC is set).

Module 15: Sockets

Sockets are endpoints for network communication.

cCopy
1// TCP server
2int sock = socket(AF_INET, SOCK_STREAM, 0);
3bind(sock, (struct sockaddr*)&addr, sizeof(addr));
4listen(sock, 5);
5int client = accept(sock, NULL, NULL);

Socket types:

• SOCK_STREAM — TCP (reliable, ordered, connection-oriented)
• SOCK_DGRAM — UDP (unreliable, unordered, connectionless)
• SOCK_RAW — Direct IP access (requires root)

Module 16: epoll

epoll is Linux's scalable I/O event notification mechanism. It replaces select() and poll() for handling thousands of connections.

cCopy
1int epoll_fd = epoll_create1(0);
2
3struct epoll_event ev;
4ev.events = EPOLLIN;
5ev.data.fd = listen_sock;
6epoll_ctl(epoll_fd, EPOLL_CTL_ADD, listen_sock, &ev);
7
8struct epoll_event events[100];
9int n = epoll_wait(epoll_fd, events, 100, -1);
10for (int i = 0; i < n; i++) {
11    if (events[i].data.fd == listen_sock) {
12        // New connection
13        int client = accept(listen_sock, NULL, NULL);
14    } else {
15        // Data available on existing connection
16        read(events[i].data.fd, buf, sizeof(buf));
17    }
18}

Why epoll scales: O(1) per operation vs. O(n) for select()/poll().

Module 17: pthreads

POSIX threads (pthreads) are the standard threading API on Linux.

cCopy
1void* thread_func(void* arg) {
2    printf("Thread %ld running\n", (long)arg);
3    return NULL;
4}
5
6pthread_t tid;
7pthread_create(&tid, NULL, thread_func, (void*)1);
8pthread_join(tid, NULL);  // Wait for thread to finish

Thread attributes: Stack size, detach state, scheduling policy, affinity.

Module 18: Synchronization

When multiple threads/processes share data, synchronization is required to prevent race conditions.

The problem:

cCopy
1// Both threads execute:
2counter++;  // Not atomic! Read → Increment → Write

The solution: Mutexes, semaphores, atomic operations.

Module 19: Mutex

A mutex (mutual exclusion lock) allows only one thread to access a critical section at a time.

cCopy
1pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
2
3pthread_mutex_lock(&lock);
4// Critical section — only one thread here at a time
5counter++;
6pthread_mutex_unlock(&lock);

Rules:

• Always lock before accessing shared data
• Always unlock after (use pthread_cleanup_push for safety)
• Never lock a mutex you already hold (deadlock!)

Module 20: Semaphore

A semaphore is a counter that controls access to a resource pool.

cCopy
1sem_t sem;
2sem_init(&sem, 0, 5);  // Allow 5 concurrent accesses
3
4sem_wait(&sem);     // Decrement — block if 0
5// Use resource
6sem_post(&sem);     // Increment — wake a waiter

Mutex vs. Semaphore:

• Mutex = binary (0 or 1), ownership-based
• Semaphore = counting (0 to N), no ownership

Named semaphores: sem_open("/mysem", O_CREAT, 0666, 1) — shared across processes.

Module 21: Atomics

Atomic operations are indivisible — no thread can observe a half-finished operation.

cCopy
1#include <stdatomic.h>
2
3_Atomic int counter = 0;
4atomic_fetch_add(&counter, 1);  // Thread-safe increment
5
6// Lock-free check
7if (atomic_is_lock_free(&counter)) {
8    printf("Counter is lock-free!\n");
9}

Memory ordering:

• memory_order_relaxed — No ordering guarantees (fastest)
• memory_order_acquire/release — Synchronize data visibility
• memory_order_seq_cst — Sequential consistency (default, safest)

The System Call Path

Application
     ↓
   libc  (wrapper: open() → syscall instruction)
     ↓
System Call  (int 0x80 / syscall / sysenter)
     ↓
Linux Kernel  (system_call() entry point)
     ↓
   VFS / Scheduler / Memory / Network / Drivers
     ↓
   Hardware

Key insight: libc functions like fopen(), malloc(), pthread_create() are wrappers around raw system calls. strace ./program shows every system call your program makes.

🛠 Capstone Project: Multi-Process IPC Server

This complete C program demonstrates fork, pipes, shared memory, signals, pthreads, mutex, and semaphores in a single cohesive example: a multi-process computation server.

Download: linux_sysprog_demo.c

cCopy
1/*
2 * linux_sysprog_demo.c
3 * Linux System Programming Capstone
4 * Build: gcc -pthread -o linux_sysprog_demo linux_sysprog_demo.c -lrt
5 * Run:   ./linux_sysprog_demo
6 *
7 * Demonstrates: fork, exec, pipe, signal, shared memory, mmap,
8 *               pthread, mutex, semaphore, atomic operations
9 */
10
11#include <stdio.h>
12#include <stdlib.h>
13#include <string.h>
14#include <unistd.h>
15#include <sys/types.h>
16#include <sys/wait.h>
17#include <sys/mman.h>
18#include <sys/stat.h>
19#include <fcntl.h>
20#include <signal.h>
21#include <pthread.h>
22#include <semaphore.h>
23#include <stdatomic.h>
24#include <errno.h>
25
26#define NUM_WORKERS 4
27#define TASK_COUNT 16
28#define SHM_NAME "/linux_demo_shm"
29
30/* Shared data structure in shared memory */
31typedef struct {
32    _Atomic int tasks_completed;
33    _Atomic int tasks_failed;
34    pthread_mutex_t mutex;
35    sem_t semaphore;
36    int pipe_fd[2];
37    int results[TASK_COUNT];
38} shared_data_t;
39
40static shared_data_t *g_shared = NULL;
41static volatile sig_atomic_t g_shutdown = 0;
42
43/* ============================================
44 * Signal Handler
45 * ============================================ */
46void signal_handler(int sig) {
47    /* Write is async-signal-safe; printf is NOT */
48    const char msg[] = "[Signal] Caught SIGINT, shutting down...\n";
49    write(STDERR_FILENO, msg, sizeof(msg) - 1);
50    g_shutdown = 1;
51}
52
53/* ============================================
54 * Worker Thread (pthread)
55 * ============================================ */
56void* worker_thread(void* arg) {
57    long id = (long)arg;
58    
59    while (!g_shutdown) {
60        /* Wait on semaphore for a task slot */
61        if (sem_wait(&g_shared->semaphore) != 0) {
62            if (errno == EINTR) continue;
63            break;
64        }
65        
66        pthread_mutex_lock(&g_shared->mutex);
67        int task_id = g_shared->tasks_completed;
68        g_shared->tasks_completed++;
69        pthread_mutex_unlock(&g_shared->mutex);
70        
71        if (task_id >= TASK_COUNT) {
72            sem_post(&g_shared->semaphore);
73            break;
74        }
75        
76        /* Simulate work */
77        usleep(50000);  /* 50ms */
78        g_shared->results[task_id] = task_id * task_id;
79        
80        /* Write result to pipe for parent notification */
81        int msg = task_id;
82        write(g_shared->pipe_fd[1], &msg, sizeof(msg));
83        
84        printf("[Thread %ld] Completed task %d -> result %d\n",
85               id, task_id, g_shared->results[task_id]);
86    }
87    
88    return NULL;
89}
90
91/* ============================================
92 * Child Process (fork + exec demo)
93 * ============================================ */
94void run_child_process(void) {
95    int pipefd[2];
96    pipe(pipefd);
97    
98    pid_t pid = fork();
99    if (pid == 0) {
100        /* Child: redirect stdout to pipe, run 'date' */
101        close(pipefd[0]);
102        dup2(pipefd[1], STDOUT_FILENO);
103        close(pipefd[1]);
104        
105        execlp("date", "date", "+%Y-%m-%d %H:%M:%S", NULL);
106        _exit(1);  /* exec failed */
107    } else if (pid > 0) {
108        /* Parent: read date from child via pipe */
109        close(pipefd[1]);
110        char buf[128];
111        ssize_t n = read(pipefd[0], buf, sizeof(buf) - 1);
112        if (n > 0) {
113            buf[n - 1] = '\0';  /* Remove newline */
114            printf("[Parent] Child reported date: %s\n", buf);
115        }
116        close(pipefd[0]);
117        waitpid(pid, NULL, 0);
118    }
119}
120
121/* ============================================
122 * Main
123 * ============================================ */
124int main(void) {
125    printf("\n");
126    printf("╔══════════════════════════════════════════════════════════════╗\n");
127    printf("║     LINUX SYSTEM PROGRAMMING: CAPSTONE DEMO                  ║\n");
128    printf("╚══════════════════════════════════════════════════════════════╝\n\n");
129
130    /* Setup signal handler */
131    struct sigaction sa;
132    memset(&sa, 0, sizeof(sa));
133    sa.sa_handler = signal_handler;
134    sigaction(SIGINT, &sa, NULL);
135    sigaction(SIGTERM, &sa, NULL);
136
137    /* Create shared memory */
138    int shm_fd = shm_open(SHM_NAME, O_CREAT | O_RDWR, 0666);
139    if (shm_fd < 0) {
140        perror("shm_open");
141        return 1;
142    }
143    ftruncate(shm_fd, sizeof(shared_data_t));
144    
145    g_shared = mmap(NULL, sizeof(shared_data_t),
146                    PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, 0);
147    if (g_shared == MAP_FAILED) {
148        perror("mmap");
149        return 1;
150    }
151    
152    memset(g_shared, 0, sizeof(shared_data_t));
153    
154    /* Initialize synchronization primitives */
155    pthread_mutexattr_t mattr;
156    pthread_mutexattr_init(&mattr);
157    pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
158    pthread_mutex_init(&g_shared->mutex, &mattr);
159    pthread_mutexattr_destroy(&mattr);
160    
161    sem_init(&g_shared->semaphore, 1, NUM_WORKERS);  /* pshared=1 */
162    pipe(g_shared->pipe_fd);
163    
164    /* Fork + Exec demo */
165    printf("[1] Fork + Exec demonstration:\n");
166    run_child_process();
167    printf("\n");
168
169    /* Pthread + Mutex + Semaphore demo */
170    printf("[2] Thread pool with mutex, semaphore, and atomics:\n");
171    pthread_t workers[NUM_WORKERS];
172    for (long i = 0; i < NUM_WORKERS; i++) {
173        pthread_create(&workers[i], NULL, worker_thread, (void*)i);
174    }
175    
176    /* Parent reads notifications from pipe */
177    printf("[3] Parent reading results from pipe (non-blocking):\n");
178    int completed = 0;
179    while (completed < TASK_COUNT && !g_shutdown) {
180        int msg;
181        fd_set fds;
182        FD_ZERO(&fds);
183        FD_SET(g_shared->pipe_fd[0], &fds);
184        struct timeval tv = { .tv_sec = 0, .tv_usec = 100000 };
185        
186        if (select(g_shared->pipe_fd[0] + 1, &fds, NULL, NULL, &tv) > 0) {
187            if (read(g_shared->pipe_fd[0], &msg, sizeof(msg)) == sizeof(msg)) {
188                printf("[Parent] Received completion for task %d\n", msg);
189                completed++;
190            }
191        }
192    }
193    
194    /* Signal shutdown and join threads */
195    g_shutdown = 1;
196    for (int i = 0; i < NUM_WORKERS; i++) {
197        sem_post(&g_shared->semaphore);  /* Wake any waiting threads */
198    }
199    for (int i = 0; i < NUM_WORKERS; i++) {
200        pthread_join(workers[i], NULL);
201    }
202    
203    /* Print final results */
204    printf("\n[4] Final results (shared memory):\n");
205    printf("    Tasks completed: %d\n", g_shared->tasks_completed);
206    printf("    Tasks failed:    %d\n", g_shared->tasks_failed);
207    printf("    Results: ");
208    for (int i = 0; i < TASK_COUNT; i++) {
209        printf("%d ", g_shared->results[i]);
210    }
211    printf("\n");
212
213    /* Cleanup */
214    close(g_shared->pipe_fd[0]);
215    close(g_shared->pipe_fd[1]);
216    pthread_mutex_destroy(&g_shared->mutex);
217    sem_destroy(&g_shared->semaphore);
218    munmap(g_shared, sizeof(shared_data_t));
219    shm_unlink(SHM_NAME);
220    
221    printf("\n✅ All demonstrations completed successfully.\n");
222    return 0;
223}

🎓 Summary: The Linux Systems Mindset

"In Linux, everything is a file descriptor, every resource is a process, and every boundary is a system call. Master these three truths, and you master the operating system."

Master Linux system programming, and you can build anything from web servers to containers to real-time systems.