Debugging Device Drivers: Tools and Techniques

Application bugs usually reproduce under a debugger. Driver bugs often do not: they depend on interrupt timing, on what the hardware did a microsecond earlier, or on two threads reaching a lock in a particular order, and the evidence may vanish with the crash. Each operating system offers tools built for this, and they work best alongside instruments that watch the bus.

At a glance

ftrace
The Linux kernel tracer for function calls, events and latency, with little overhead when disabled.
KASAN
The Linux kernel address sanitiser, which finds out-of-bounds and use-after-free accesses.
Verifier
Windows Driver Verifier stresses a driver and stops at the first rule it breaks.
WinDbg
The Microsoft debugger for live kernel sessions and crash dump analysis.

The problem

The first instinct is to add print statements. They help, but printing from an interrupt handler changes timing and can hide the race being chased, and a message still buffered at the moment of a crash may never be seen.

The second instinct is to blame the hardware, or the software, without evidence either way. Without a record of what the driver did and what the bus carried, that argument can run for days.

Capture what the driver did and what the hardware saw at the same moment; most driver arguments end there.

Matching tool to symptom

For crashes, collect a dump and read the stack and state in WinDbg on Windows, or with kdump and the crash utility on Linux. For memory corruption, enable KASAN on a Linux debug kernel or the special pool option of Driver Verifier on Windows. For hangs and races, turn on lockdep on Linux or the Verifier deadlock checks on Windows, and trace with ftrace or Event Tracing for Windows.

For timing and protocol problems, put a logic analyser or protocol analyser on the bus and line its capture up with a software trace. On macOS, DriverKit drivers are user-space processes, so unified logging and crash reports cover much of the ground.

What good looks like

Build a debug configuration early: on Linux a kernel with KASAN, lockdep and dynamic debug enabled; on Windows a test machine with kernel debugging and Driver Verifier turned on.

Reproduce the failure reliably, then narrow it down with tracing rather than prints, keeping the capture running until the fault occurs.

Once fixed, add a stress or fault-injection test that would have caught it.

Tools by symptom
SymptomLinuxWindows
CrashOops log, kdump and crashCrash dump in WinDbg
Memory corruptionKASAN, KFENCEDriver Verifier special pool
Deadlock or racelockdep, KCSANDriver Verifier deadlock detection
Wrong behaviourftrace, dynamic debug, bpftraceWPP and ETW tracing
Live inspectionKGDB, JTAG debuggerWinDbg over network or serial
Main tools on each platform. Several need a debug kernel build or a test machine configured for kernel debugging.

A driver debugging loop

  1. Reproduce: Reliable trigger, ideally automated
  2. Capture: Software trace plus bus capture
  3. Isolate: Narrow to one path or event
  4. Fix: Smallest change that explains the evidence
  5. Guard: Regression or stress test added
The last step is the one most often skipped.

What it changes

Faster Diagnosis

Evidence from traces and dumps replaces guesswork about the cause.

Fewer Field Crashes

Memory and locking checkers find bugs that ordinary testing misses.

Clear Ownership

Bus captures settle whether a fault lies in hardware or software.

Lasting Quality

Each fixed bug leaves a test behind that stops it returning.

Where to go next

Design debug access into the hardware: a serial console, test points on the buses and, on development boards, a JTAG or SWD header.

Keep crash dump collection enabled on test fleets. A dump from the first occurrence is worth more than a description of the tenth.

Problems this prevents

  • Races hidden by print statements
  • Crashes with no dump to analyse
  • Memory corruption found only in the field
  • Hardware and software teams blaming each other

Discuss your project

Email support@aavico.com or use Get in Touch when the website finishes loading.