Linux Tracing Technologies Guide

Tracing in the Linux kernel is a powerful mechanism that allows developers and system administrators to analyze and debug system behavior. This guide provides documentation on various tracing frameworks and tools available in the Linux kernel.

Introduction to Tracing

This section provides an overview of Linux tracing mechanisms and debugging approaches.

• Using the tracer for debugging
  • Introduction
  • Using trace_printk()
  • Debugging kernel crashes
  • Persistent buffers across boots
  • Using trace_printk() in the boot instance
• Using the Linux Kernel Tracepoints
  • Purpose of tracepoints
  • Usage
• Notes on Analysing Behaviour Using Events and Tracepoints
  • 1. Introduction
  • 2. Listing Available Events
  • 3. Enabling Events
  • 4. Event Filtering
  • 5. Analysing Event Variances with PCL
  • 6. Higher-Level Analysis with Helper Scripts
  • 7. Lower-Level Analysis with PCL
• Tracefs ring-buffer memory mapping
  • Overview
  • Memory mapping setup
  • Limitations
  • Example

Core Tracing Frameworks

The following are the primary tracing frameworks integrated into the Linux kernel.

• ftrace - Function Tracer
• Function Tracer Design
• Using ftrace to hook to functions
• Kernel Probes (Kprobes)
• Kprobe-based Event Tracing
• Fprobe-based Event Tracing
• Fprobe - Function entry/exit probe
• Lockless Ring Buffer Design

Event Tracing and Analysis

A detailed explanation of event tracing mechanisms and their applications.

• Event Tracing
• Subsystem Trace Points: kmem
• Subsystem Trace Points: power
• NMI Trace Events
• MSR Trace Events
• Subsystem Trace Points: PCI
• Boot-time tracing
• Event Histograms
• Histogram Design Notes

Hardware and Performance Tracing

This section covers tracing features that monitor hardware interactions and system performance.

• Intel(R) Trace Hub (TH)
• System Trace Module
• MIPI SyS-T over STP
• CoreSight - ARM Hardware Trace
• Runtime Verification
• HiSilicon PCIe Tune and Trace device
• In-kernel memory-mapped I/O tracing
• Hardware Latency Detector
• OSNOISE Tracer
• Timerlat tracer

User-Space Tracing

These tools allow tracing user-space applications and interactions.

• user_events: User-based Event Tracing
• Uprobe-tracer: Uprobe-based Event Tracing

Additional Resources

For more details, refer to the respective documentation of each tracing tool and framework.