Debugging and Testing

Coverage: printk/dynamic debug → KGDB → KUnit → kselftest → LTP → syzkaller → fault injection → lockdep integration Kernel versions: 2.6 ~ 6.x

printk: Kernel Logging

// kernel/printk/printk.c
printk(KERN_ERR "error: %d\n", err);

// Log level filtering:
//   KERN_EMERG(0) KERN_ALERT(1) KERN_CRIT(2) KERN_ERR(3)
//   KERN_WARNING(4) KERN_NOTICE(5) KERN_INFO(6) KERN_DEBUG(7)

// Console output filtering:
echo 4 > /proc/sys/kernel/printk  // Only output ERROR and above to console

// 5.x+ Structured logging: dev_printk, pr_fmt

Dynamic Debug

# Enable debug logging by module/function/file/line (runtime, no recompilation needed)
echo "module nfs +p" > /sys/kernel/debug/dynamic_debug/control
echo "func tcp_rcv_established +p" > dynamic_debug/control
echo "file fs/nfs/* +p" > dynamic_debug/control
# p=printk, f=include func name, l=include line, t=include thread id

# View enabled entries
cat /sys/kernel/debug/dynamic_debug/control | grep "=p"

KGDB: Kernel-level GDB

# Remote kernel debugging via serial port (similar to gdbserver)
# boot: kgdboc=ttyS0,115200 kgdbwait
# gdb vmlinux → target remote /dev/ttyS0 → set breakpoints, step through
# SysRq-g to enter KGDB

echo g > /proc/sysrq-trigger  # Trigger entry into KGDB

KUnit: Kernel Unit Testing

// lib/kunit/
// Similar to user-space unit testing frameworks, executes entirely within the kernel
#include <kunit/test.h>

static void my_test(struct kunit *test) {
    KUNIT_EXPECT_EQ(test, my_function(42), 84);
}

static struct kunit_case my_test_cases[] = {
    KUNIT_CASE(my_test),
    {},
};

// Run: Built-in at compile time → Automatically executed at boot → Results in dmesg

kselftest: Kernel Self-Test Suite

# Kernel self-contained tests (tools/testing/selftests/)
make -C tools/testing/selftests TARGETS=net run_tests
make -C tools/testing/selftests TARGETS=bpf run_tests

# Covers: syscalls, memory management, networking, BPF, filesystems, ...

Syzkaller: Fuzzing

LTP (Linux Test Project)

# Largest kernel/glibc regression test suite (github.com/linux-test-project/ltp)
./runltp -f syscalls       # Syscall tests
./runltp -f mm             # Memory management
./runltp -f fs             # Filesystem

Fault Injection

# Fault injection framework: Simulate various hardware/memory/IO faults
# Verify that error handling paths are correct

# Memory allocation failure:
echo 100 > /sys/kernel/debug/failslab/interval  # Fail 1 time per 100 slab allocs

# IO error:
echo 1 > /sys/kernel/debug/fail_make_request/times

References

• Source Code: kernel/printk/, kernel/debug/, lib/kunit/, tools/testing/selftests/
• Kernel Documentation: Documentation/dev-tools/, Documentation/admin-guide/dynamic-debug-howto.rst
• Tools: syzkaller (github.com/google/syzkaller), LTP (github.com/linux-test-project/ltp)

Keywords: printk, dynamic debug, KGDB, KUnit, kselftest, syzkaller, LTP, fault injection