---
title: Debugging and Testing
url: https://doc.liz6.com/en/linux-kernel/11-kernel-development-and-build/02-debugging-and-testing
locale: en
area: linux-kernel
tags:
- linux-kernel
- kernel-development-and-build
date: 2026-06-30
modified: 2026-07-16
description: 'Coverage: printk/dynamic debug → KGDB → KUnit → kselftest → LTP → syzkaller → fault injection → lockdep integration Kernel versions: 2.6 ~ 6.x'
---

# Debugging and Testing

> Coverage: printk/dynamic debug → KGDB → KUnit → kselftest → LTP → syzkaller → fault injection → lockdep integration
> Kernel versions: 2.6 ~ 6.x

## printk: Kernel Logging

```c
// kernel/printk/printk.c
printk(KERN_ERR "error: %d\n", err);

// Log level filtering:
//   KERN_EMERG(0) KERN_ALERT(1) KERN_CRIT(2) KERN_ERR(3)
//   KERN_WARNING(4) KERN_NOTICE(5) KERN_INFO(6) KERN_DEBUG(7)

// Console output filtering:
echo 4 > /proc/sys/kernel/printk  // Only output ERROR and above to console

// 5.x+ Structured logging: dev_printk, pr_fmt
```

## Dynamic Debug

```bash
# Enable debug logging by module/function/file/line (runtime, no recompilation needed)
echo "module nfs +p" > /sys/kernel/debug/dynamic_debug/control
echo "func tcp_rcv_established +p" > dynamic_debug/control
echo "file fs/nfs/* +p" > dynamic_debug/control
# p=printk, f=include func name, l=include line, t=include thread id

# View enabled entries
cat /sys/kernel/debug/dynamic_debug/control | grep "=p"
```

## KGDB: Kernel-level GDB

```bash
# Remote kernel debugging via serial port (similar to gdbserver)
# boot: kgdboc=ttyS0,115200 kgdbwait
# gdb vmlinux → target remote /dev/ttyS0 → set breakpoints, step through
# SysRq-g to enter KGDB

echo g > /proc/sysrq-trigger  # Trigger entry into KGDB
```

## KUnit: Kernel Unit Testing

```c
// lib/kunit/
// Similar to user-space unit testing frameworks, executes entirely within the kernel
#include <kunit/test.h>

static void my_test(struct kunit *test) {
    KUNIT_EXPECT_EQ(test, my_function(42), 84);
}

static struct kunit_case my_test_cases[] = {
    KUNIT_CASE(my_test),
    {},
};

// Run: Built-in at compile time → Automatically executed at boot → Results in dmesg
```

## kselftest: Kernel Self-Test Suite

```bash
# Kernel self-contained tests (tools/testing/selftests/)
make -C tools/testing/selftests TARGETS=net run_tests
make -C tools/testing/selftests TARGETS=bpf run_tests

# Covers: syscalls, memory management, networking, BPF, filesystems, ...
```

## Syzkaller: Fuzzing

<svg viewBox="0 0 720 260" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,'Source Han Sans CN','Microsoft YaHei',sans-serif" role="img" aria-label="Syzkaller fuzzing process: randomly generating syscall sequences to discover kernel bugs">
  <defs><marker id="sk-arrow" markerWidth="10" markerHeight="8" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6 Z" fill="#475569"/></marker></defs>
  <rect width="720" height="260" fill="#ffffff"/>
  <text x="360" y="28" text-anchor="middle" font-size="17" font-weight="700" fill="#1f2933">Syzkaller: Coverage-guided Unsupervised Kernel Fuzzer</text>
  <rect x="20" y="70" width="140" height="64" rx="6" fill="#e2e8f0"/>
  <text x="90" y="98" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">Generate Random Syscall Sequences</text>
  <text x="90" y="116" text-anchor="middle" font-size="10" fill="#475569">(Coverage-guided)</text>
  <line x1="160" y1="102" x2="196" y2="102" stroke="#475569" stroke-width="1.6" marker-end="url(#sk-arrow)"/>
  <rect x="200" y="70" width="140" height="64" rx="6" fill="#e2e8f0"/>
  <text x="270" y="106" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">Execute in QEMU VM</text>
  <line x1="340" y1="102" x2="376" y2="102" stroke="#475569" stroke-width="1.6" marker-end="url(#sk-arrow)"/>
  <rect x="380" y="70" width="140" height="64" rx="6" fill="#e2e8f0"/>
  <text x="450" y="106" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">Detect Crash / Hang</text>
  <line x1="520" y1="102" x2="556" y2="102" stroke="#475569" stroke-width="1.6" marker-end="url(#sk-arrow)"/>
  <rect x="560" y="70" width="140" height="64" rx="6" fill="#e2e8f0"/>
  <text x="630" y="98" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">Reproduce + Report</text>
  <text x="630" y="116" text-anchor="middle" font-size="10" fill="#475569">(Automatic Bisect)</text>
  <line x1="630" y1="134" x2="630" y2="160" stroke="#475569" stroke-width="1.6" marker-end="url(#sk-arrow)"/>
  <rect x="20" y="164" width="680" height="70" rx="8" fill="#dcfce7" stroke="#4ade80"/>
  <text x="36" y="192" font-size="13" font-weight="700" fill="#166534">Discovered 5000+ kernel bugs — Coverage-guided random syscall sequences continuously mining for crashes/hangs in QEMU</text>
  <text x="36" y="214" font-size="12" fill="#15803d">Deployment: github.com/google/syzkaller</text>
</svg>

## LTP (Linux Test Project)

```bash
# Largest kernel/glibc regression test suite (github.com/linux-test-project/ltp)
./runltp -f syscalls       # Syscall tests
./runltp -f mm             # Memory management
./runltp -f fs             # Filesystem
```

## Fault Injection

```bash
# Fault injection framework: Simulate various hardware/memory/IO faults
# Verify that error handling paths are correct

# Memory allocation failure:
echo 100 > /sys/kernel/debug/failslab/interval  # Fail 1 time per 100 slab allocs

# IO error:
echo 1 > /sys/kernel/debug/fail_make_request/times
```

## References

- **Source Code**: `kernel/printk/`, `kernel/debug/`, `lib/kunit/`, `tools/testing/selftests/`
- **Kernel Documentation**: `Documentation/dev-tools/`, `Documentation/admin-guide/dynamic-debug-howto.rst`
- **Tools**: syzkaller (github.com/google/syzkaller), LTP (github.com/linux-test-project/ltp)

*Keywords: printk, dynamic debug, KGDB, KUnit, kselftest, syzkaller, LTP, fault injection*