Source level debugging using Kgdb+Kmod which is a modified version of kgdb/gdbstubs which provides improved kernel module debug support including real-time modules. The target is debugged from a host by mean of serial cable which connects both machines.
An optional tracer module (Linux Trace Toolkit, LTT) can be included in the system. The tracer will register all the relevant system events and user-defined ones. LTT has also a graphical tool to visualize the logged events.
LTT provides developers with all of the information necessary to reconstruct a system's behavior over a certain period of time. Using LTT, one can graphically view the exact the dynamics of a system and find logical and temporal bugs of the system or the application.