Qemu Debug

Timeline

Timeline

2025-11-23

1. init

This article introduces various methods for debugging QEMU, including using gdb to directly debug QEMU source code, remote debugging via the built-in gdbserver, using the flexible logging system to observe guest state, and focusing on QEMU's tracing tool. The article details the quick start approach for tracing, including how to dynamically enable trace-events via startup parameters or the monitor, configure output files, and introduces the definition format of trace-events, steps to add them, and usage considerations such as type support and portability requirements. In addition, it outlines the design of the trace backend and support for multiple backends, providing practical guidance for QEMU source debugging and performance analysis.

Environment

wget https://download.qemu.org/qemu-10.1.2.tar.xz
tar xvJf qemu-10.1.2.tar.xz
cd qemu-10.1.2
mkdir -p output
./configure --prefix=$PWD/output --target-list=aarch64-softmmu,riscv64-softmmu --enable-debug
bear -- make -j$(nproc)

Create .clangd

CompileFlags:
  Add: -Wno-unknown-warning-option
  Remove: [-m*, -f*]

Source Code Debugging

Debugging QEMU source code does not require starting it first and then attaching with gdb like debugging the kernel. QEMU itself can be debugged directly locally. (Note: enable-debug must be enabled during compilation.)

gdb

You can directly use gdb for debugging.

$ gdb --args ./build/riscv64-softmmu/qemu-system-riscv64 \
    -M virt -kernel Image -nographic

vscode

.vscode/launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "qemu-system-riscv",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/build/qemu-system-riscv64",
      "args": [
        "-device", "edu,id=edu1"
      ],
      "stopAtEntry": true,
      "cwd": "${fileDirname}",
      "environment": [],
      "externalConsole": false,
      "MIMode": "gdb",
      "setupCommands": [
          {
              "description": "Enable pretty-printing for gdb",
              "text": "-enable-pretty-printing",
              "ignoreFailures": true
          },
          {
              "description": "Set Disassembly Flavor to Intel",
              "text": "-gdb-set disassembly-flavor intel",
              "ignoreFailures": true
          }
      ]
    },
  ]
}

Remote debugging

QEMU has a built-in gdbserver that can control the guest processor. Users can connect via any gdb client.

The specific commands are as follows:

$QEMU $QEMU_ARGS -s -S

• -s: Start the gdbstub and set the port number to 1234.
• -S: Make QEMU stop at the first instruction of the guest and wait for a gdb client connection.

To specify a debug port number, you can use the following command:

$QEMU $QEMU_ARGS -gdb tcp::<your-port> -S

Then use the gdb for the corresponding architecture to connect to the QEMU gdbstub:

$ARCH-gdb $BINARY -ex "target remote localhost:1234"

Log Debugging

QEMU has a flexible logging system that makes it easy to observe various states of the guest (instruction stream, interrupts, exceptions, system calls).

The basic command format is given below:

$QEMU $QEMU_ARGS -d <log-type,...> -D <log-file-name>

• -d: Specify the log type; multiple types can be specified, using,to separate.
• -D: Specify the file path for log output. If this parameter is not added, output defaults to the command-line terminal.

You can use the following command to view the currently supported log types:

$QEMU -d ?
Log items (comma separated):
out_asm         show generated host assembly code for each compiled TB
in_asm          show target assembly code for each compiled TB
op              show micro ops for each compiled TB
op_opt          show micro ops after optimization
op_ind          show micro ops before indirect lowering
op_plugin       show micro ops before plugin injection
int             show interrupts/exceptions in short format
exec            show trace before each executed TB (lots of logs)
cpu             show CPU registers before entering a TB (lots of logs)
fpu             include FPU registers in the 'cpu' logging
mmu             log MMU-related activities
pcall           x86 only: show protected mode far calls/returns/exceptions
cpu_reset       show CPU state before CPU resets
unimp           log unimplemented functionality
guest_errors    log when the guest OS does something invalid (eg accessing a
non-existent register)
page            dump pages at beginning of user mode emulation
nochain         do not chain compiled TBs so that "exec" and "cpu" show
complete traces
plugin          output from TCG plugins
strace          log every user-mode syscall, its input, and its result
tid             open a separate log file per thread; filename must contain '%d'
vpu             include VPU registers in the 'cpu' logging
invalid_mem     log invalid memory accesses
trace:PATTERN   enable trace events

Use "-d trace:help" to get a list of trace events.

We list some commonly used combinations.

• If we want to observe how TCG translates instructions, we can use the following command:

$QEMU $QEMU_ARGS -d in_asm,op,out_asm -D tcg.log

• If we want to observe the CPU state (register values, interrupts/exceptions), we can use the following command:

$QEMU $QEMU_ARGS -d exec,cpu,int -D cpu.log

• If you want to obtain the precise instruction stream executed by the CPU, you need to set each TB to contain only one instruction. You can use the following command:

$QEMU $QEMU_ARGS --accel tcg,one-insn-per-tb=on -d exec,cpu,int -D cpu.log

Trace Events

Reference:

Qemu Docs tracing

https://www.qemu.org/docs/master/devel/tracing.html

QEMU has a very useful debugging tool called tracing, which can be used to track the execution of QEMU internal functions and for performance tuning. For example, to trace the memory access of a guest program, you can print out the read/write records of QEMU’s memory_region, as long as the corresponding trace-event is registered.

Quick Start

In QEMU’s startup options, add the trace parameter to specify the events to trace. Here, we take tracing memory region access events as an example:

$ qemu-system-riscv64 -M virt --trace "memory_region_ops_*" # The * symbol represents the preceding character as the matching object.
...
719585@1608130130.441188:memory_region_ops_read cpu 0 mr 0x562fdfbb3820 addr 0x3cc value 0x67 size 1
719585@1608130130.441190:memory_region_ops_write cpu 0 mr 0x562fdfbd2f00 addr 0x3d4 value 0x70e size 2

We can find the trace-events related to mr in the system/trace-events file of the QEMU source code:

# memory.c
memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"
memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"

If you want to enable multiple trace-events, simply append them to the startup options.--trace <name>。

To avoid lengthy parameters, you can record the trace-events to be traced in a configuration file and then load it:

echo "memory_region_ops_*" >/tmp/events
echo "kvm_*" >>/tmp/events
qemu-system-riscv64 -M virt --trace events=/tmp/events ...

Tracing also supports output to a file, so we modify the QEMU command above:

qemu-system-riscv64 -M virt --trace events=/tmp/events,file=/tmp/event.log ...

If you don’t want to enable it in the QEMU startup options, you can also enable it dynamically in the QEMU monitor, which is more flexible. The procedure is as follows:

$ qemu-system-riscv64 -M virt -monitor stdio -S -display none
(qemu) trace-event memory_region_ops_read on
(qemu) c
...
memory_region_ops_write cpu 0 mr 0x55a289a24d80 addr 0x10000000 value 0x78 size 1 name 'serial'

Another benefit of using tracing in the monitor is that you can use the Tab key to complete commands, without having to manually look up the trace-events supported by each component from the source code.

You can also useinfo trace-eventsCommand to query supported trace events.

or usetrace-fileCommand to output the trace log to a file.

Tracing supports multiple backends; by default, QEMU’s log is used as the backend.

Introduction to trace-events

In every directory level of the QEMU source code, you can add a trace-events file. Simply declare its relative path in the top-level meson.build file, and you can add custom trace-events to it:

if have_system
  trace_events_subdirs += [
    'accel/kvm',
    'backends/tpm',
    'ebpf',
    'hw/arm',
    ...
  ]
endif

The trace-events files in these subdirectories need to be processed by tracetool to automatically generate trace code.

During the QEMU build process, each trace-events file will be processed by the tracetool script, which automaticallybuild/trace/generates trace-related code under the path, mainly including the following files:

- trace-<子目录名>.c
- trace-<subdir>.h
- trace-dtrace-<subdir>.h
- trace-dtrace-<subdir>.dtrace
- trace-dtrace-<subdir>.o
- trace-ust-<subdir>.h

Here, the<subdirectory name>means replacing ‘/’ in the subdirectory path with ‘_’.

Example

accel/kvm/trace-events

trace/trace-accel_kvm.c
trace/trace-accel_kvm.h
trace/trace-dtrace-accel_kvm.h
trace/trace-dtrace-accel_kvm.dtrace
trace/trace-dtrace-accel_kvm.o
trace/trace-ust-accel_kvm.h

The various trace-events files are merged into a trace-events-all file,trace/trace-events-all

This merged file will be used by the simpletrace.py script provided by QEMU for subsequent analysis of the binary trace records of the simple backend.

In the source directory, the generated files are not directly included; instead, the local trace.h file is referenced via #include, without any subdirectory path prefix.。

For example, io/channel-buffer.c references it like this:

#include "trace.h"

In addition, we must manually create the io/trace.h file,

and include the correspondingtrace/trace-<subdirectory name>.hfile, which is generated by tracetool at build time:

echo '#include "trace/trace-io.h"' > io/trace.h

It is worth noting: although it is possible to include trace.h from outside the subdirectory where the source file resides, this is generally not recommended.

It is strongly recommended that all trace-events be declared directly in the subdirectory where they are used. The only exception is that some shared tracing events are defined in the top-level trace-events file.

The tracing files generated in the top-level directory are prefixed with trace/trace-root rather than just trace, to avoid ambiguity between trace.h in the current directory and the file in the top-level directory.

Example

src/
│
├── io/
│   ├── channel.c
│   ├── trace-events   <-- 你写的
│   └── trace.h        <-- 你创建，引用 trace-io.h
│
├── hw/arm/
│   ├── xxxx.c
│   ├── trace-events
│   └── trace.h
│
└── meson.build        <-- 声明哪些目录包含 trace-events

After build:

build/trace/
    trace-io.c
    trace-io.h
    trace-hw_arm.c
    trace-hw_arm.h
    trace-events-all     <-- 所有 trace-events 合并

Adding a new trace-event

Adding a new trace-event requires only two steps:

• Declare the trace-event in the trace-events file in the corresponding directory.
• Add a function call for this event in the target source code that needs debugging.

Taking tracing QEMU memory allocation and release as an example, the trace-event format is as follows:

qemu_vmalloc(size_t size, void *ptr) "size %zu ptr %p"
qemu_vfree(void *ptr) "ptr %p"

Each event declaration begins with the event name, followed by parameters, and finally a format string for pretty printing.。

The format string should reflect the types defined in the trace event. Tracing only supports basic scalar types (char, int, long), and does not support floating-point types (float, double).

Pay special attention to int64_t and uint64_t types use PRId64 and PRIu64 respectively, which ensures portability between 32-bit and 64-bit platforms.

The format string must not end with a newline character. The backend is responsible for adjusting line endings to achieve correct logging.

Once the trace-event is defined, call it directly from the target source code, as shown in the following example:

#include "trace.h"  /* needed for trace event prototype */
void *qemu_vmalloc(size_t size)
{
    void *ptr;
    size_t align = QEMU_VMALLOC_ALIGN;
    if (size < align) {
        align = getpagesize();
    }
    ptr = qemu_memalign(align, size);
    /* Insert the trace-event in the format: trace_<event-name> */
    trace_qemu_vmalloc(size, ptr);
    return ptr;
}

If there are multiple trace events in a function, you shouldadd a unique identifier at the end of the nameto distinguish them.

In some cases, it may be necessary to perform relatively complex calculations to generate values used only as arguments to the trace function. In such cases, the following function can be used to guard this calculation logic:

trace_event_get_state_backends()

When the event is disabled at compile time or runtime, the associated calculations will be skipped. If the event is disabled at compile time, this check will incur no performance overhead.

Example code is as follows.

#include "trace.h"  /* needed for trace event prototype */
void *qemu_vmalloc(size_t size)
{
    void *ptr;
    size_t align = QEMU_VMALLOC_ALIGN;
    if (size < align) {
        align = getpagesize();
    }
    ptr = qemu_memalign(align, size);
    if (trace_event_get_state_backends(TRACE_QEMU_VMALLOC)) {
        void *complex;
        /* some complex computations to produce the 'complex' value */
        trace_qemu_vmalloc(size, ptr, complex);
    }
    return ptr;
}

The following situations are well-suited for debugging with tracing:

1. Trace state changes in code. Key points in code often involve state changes, such as start, stop, allocate, release, etc. State changes are ideal trace events because they help understand the system execution process.
2. Trace guest operations. Guest I/O accesses (such as reading device registers) are good trace events and can be used to analyze guest interaction behavior.
3. Use correlation fields to understand the context of a single trace line output. For example, trace the pointer returned by malloc and its use as an argument to free, so that malloc and free operations can be matched. Trace events lacking context have limited practical value.

Introduction to trace backends

QEMU’s tracing uses a front-end/back-end separated design and supports multiple backends. In addition to the log backend mentioned above, it also supports the lighter-weight simple backend, as well as ftrace and dtrace.

We can also add more backend support through the tracetool script.

To enable different backends, you can use the following QEMU build commands:

./configure --enable-trace-backends=simple,dtrace

By running./configure --helpView all supported backends. If no backend is explicitly selected, the configuration will use the log backend by default (equivalent to--enable-trace-backends=log）。

Analyzing trace files

We take the simple backend as an example. This backend writes binary trace logs to a file through a separate thread, and has lower overhead compared to the log backend.

Meanwhile, the QEMU source repository provides a Python script for offline trace file analysis. Although its functionality may not be as powerful as platform-specific or third-party tracing backends, it is portable and requires no special library dependencies.

To format using the simpletrace.py script, you need the trace-events-all file and the binary trace file:

./scripts/simpletrace.py <trace-events-all> <trace-log>

You must ensure that the trace-events-all file used is the same as the one generated when building QEMU; otherwise, trace event declarations may have changed, resulting in inconsistent output.

Reference:

Learning Qemu Docs

https://gevico.github.io/learning-qemu-docs/