Qemu Debug

Timeline

Timeline

2025-11-23

1. init

This article introduces various debugging techniques for QEMU, including source code debugging using gdb and vscode, remote debugging using the built-in gdbserver, and observing guest states through the logging system. Furthermore, this article discusses in detail the usage of the tracing tool, the declaration and addition process of trace-events, and the tracing architecture supporting multiple backends, providing comprehensive guidance for tracking QEMU internal function execution and performance tuning.

Environment

wget https://download.qemu.org/qemu-10.1.2.tar.xz
tar xvJf qemu-10.1.2.tar.xz
cd qemu-10.1.2
mkdir -p output
./configure --prefix=$PWD/output --target-list=aarch64-softmmu,riscv64-softmmu --enable-debug
bear -- make -j$(nproc)

Create .clangd

CompileFlags:
  Add: -Wno-unknown-warning-option
  Remove: [-m*, -f*]

Source code debugging

Debugging QEMU source code does not require the same process as debugging the kernel, where you need to start it first and then attach via gdb; QEMU itself can be executed directly locally. (Note: enable-debug is required during compilation)

gdb

You can debug directly using gdb

$ gdb --args ./build/riscv64-softmmu/qemu-system-riscv64 \
    -M virt -kernel Image -nographic

vscode

.vscode/launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "qemu-system-riscv",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/build/qemu-system-riscv64",
      "args": [
        "-device", "edu,id=edu1"
      ],
      "stopAtEntry": true,
      "cwd": "${fileDirname}",
      "environment": [],
      "externalConsole": false,
      "MIMode": "gdb",
      "setupCommands": [
          {
              "description": "Enable pretty-printing for gdb",
              "text": "-enable-pretty-printing",
              "ignoreFailures": true
          },
          {
              "description": "Set Disassembly Flavor to Intel",
              "text": "-gdb-set disassembly-flavor intel",
              "ignoreFailures": true
          }
      ]
    },
  ]
}

Remote debugging

QEMU has a built-in gdbserver that can control the guest processor. Users can connect via any gdb client.

The specific operation commands are given below:

$QEMU $QEMU_ARGS -s -S

• -s: Start the gdbstub and set the port number to 1234
• -S: Make QEMU stop at the first instruction of the guest and wait for the gdb client to connect.

If you want to specify the debugging port number, you can use the following command:

$QEMU $QEMU_ARGS -gdb tcp::<your-port> -S

Then use the gdb corresponding to the architecture to connect to the QEMU gdbstub:

$ARCH-gdb $BINARY -ex "target remote localhost:1234"

Log debugging

QEMU has a flexible logging system that makes it very convenient to observe various states of the guest (instruction flow, interrupts, exceptions, system calls).

The basic command format is given below:

$QEMU $QEMU_ARGS -d <log-type,...> -D <log-file-name>

• -d: Specify the type of log, multiple can be used, separated by,commas
• -D: Specify the file path for outputting the log; if this parameter is not added, it defaults to outputting to the command line terminal

You can use the following command to view the currently supported log types:

$QEMU -d ?
Log items (comma separated):
out_asm         show generated host assembly code for each compiled TB
in_asm          show target assembly code for each compiled TB
op              show micro ops for each compiled TB
op_opt          show micro ops after optimization
op_ind          show micro ops before indirect lowering
op_plugin       show micro ops before plugin injection
int             show interrupts/exceptions in short format
exec            show trace before each executed TB (lots of logs)
cpu             show CPU registers before entering a TB (lots of logs)
fpu             include FPU registers in the 'cpu' logging
mmu             log MMU-related activities
pcall           x86 only: show protected mode far calls/returns/exceptions
cpu_reset       show CPU state before CPU resets
unimp           log unimplemented functionality
guest_errors    log when the guest OS does something invalid (eg accessing a
non-existent register)
page            dump pages at beginning of user mode emulation
nochain         do not chain compiled TBs so that "exec" and "cpu" show
complete traces
plugin          output from TCG plugins
strace          log every user-mode syscall, its input, and its result
tid             open a separate log file per thread; filename must contain '%d'
vpu             include VPU registers in the 'cpu' logging
invalid_mem     log invalid memory accesses
trace:PATTERN   enable trace events

Use "-d trace:help" to get a list of trace events.

We list some commonly used combinations.

• If we want to observe how TCG translates instructions, we can use the following command:

$QEMU $QEMU_ARGS -d in_asm,op,out_asm -D tcg.log

• If we want to observe the state of the CPU (register values, interrupts/exceptions), we can use the following command:

$QEMU $QEMU_ARGS -d exec,cpu,int -D cpu.log

• If you want to obtain the precise instruction flow executed by the CPU, you need to set each TB to contain only one instruction, you can use the following command:

$QEMU $QEMU_ARGS --accel tcg,one-insn-per-tb=on -d exec,cpu,int -D cpu.log

Tracing events

Reference:

Qemu Docs tracing

https://www.qemu.org/docs/master/devel/tracing.html

QEMU has a very useful debugging tool called tracing, which can be used to track the execution of QEMU internal functions, as well as for performance tuning. For example, to track the memory access of a guest program, the read and write records of QEMU’s memory_region can be printed out, as long as the corresponding trace-event is registered.

Quick start

In QEMU’s startup options, add the trace parameter to specify the events to be tracked. Here, tracking memory region access events is taken as an example:

$ qemu-system-riscv64 -M virt --trace "memory_region_ops_*" # * represents the preceding character as the matching object
...
719585@1608130130.441188:memory_region_ops_read cpu 0 mr 0x562fdfbb3820 addr 0x3cc value 0x67 size 1
719585@1608130130.441190:memory_region_ops_write cpu 0 mr 0x562fdfbd2f00 addr 0x3d4 value 0x70e size 2

We can find the mr-related trace-event in the system/trace-events file of the QEMU source code:

# memory.c
memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"
memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"

If you want to enable multiple trace-events, you just need to append them in the startup options--trace <name>。

To avoid lengthy parameters, you can record the trace-events to be tracked in a configuration file and then load it:

echo "memory_region_ops_*" >/tmp/events
echo "kvm_*" >>/tmp/events
qemu-system-riscv64 -M --trace events=/tmp/events ...

At the same time, tracing also supports outputting to a file. Let’s modify the QEMU command above:

qemu-system-riscv64 -M --trace events=/tmp/events,file=/tmp/event.log ...

If you don’t want to enable it in the QEMU startup options, we can also dynamically enable it in the QEMU monitor, which is more flexible. The operation is as follows:

$ qemu-system-riscv64 -M virt -monitor stdio -S -display none
(qemu) trace-event memory_region_ops_read on
(qemu) c
...
memory_region_ops_write cpu 0 mr 0x55a289a24d80 addr 0x10000000 value 0x78 size 1 name 'serial'

Another benefit of using tracing in the monitor is that you can use the Tab key to complete commands, so you don’t have to painstakingly look up the trace-events supported by each component from the source code.

can also useinfo trace-eventscommand queries the supported trace events.

Or usetrace-filecommand outputs the trace log to a file.

tracing supports multiple backends, and uses QEMU’s log as the backend by default.

Introduction to trace-events

In every level of the QEMU source code directory, you can add a trace-events file. You just need to declare its relative path in the top-level meson.build directory, and then you can add custom trace-events to it:

if have_system
  trace_events_subdirs += [
    'accel/kvm',
    'backends/tpm',
    'ebpf',
    'hw/arm',
    ...
  ]
endif

These trace-events files in the subdirectories need to be processed by tracetool to automatically generate trace code.

During the QEMU build process, each trace-events file will be processed by the tracetool script, automatically generating trace-related code under the /trace/ path, which mainly includes the following files:

- trace-<子目录名>.c
- trace-<subdir>.h
- trace-dtrace-<subdir>.h
- trace-dtrace-<subdir>.dtrace
- trace-dtrace-<subdir>.o
- trace-ust-<subdir>.h

Here represents replacing ‘/’ in the subdirectory path with ‘_’ .

Example

accel/kvm/trace-events

trace/trace-accel_kvm.c
trace/trace-accel_kvm.h
trace/trace-dtrace-accel_kvm.h
trace/trace-dtrace-accel_kvm.dtrace
trace/trace-dtrace-accel_kvm.o
trace/trace-ust-accel_kvm.h

The various trace-events files will be merged into a single trace-events-all file,trace/trace-events-all

This file will also be installed to the /usr/share/qemu directory. This merged file will be used by the simpletrace.py script provided by QEMU to subsequently analyze trace records in the simple trace data format.

In the source code directory, the generated files are not directly included; instead, the local trace.h file is referenced via #include, without any subdirectory path prefix。

For example, io/channel-buffer.c would reference it like this:

#include "trace.h"

Additionally, we must manually create the io/trace.h file,

and include the corresponding trace/trace-.h file in it, which is generated in :

echo '#include "trace/trace-io.h"' > io/trace.h

It is worth noting: although trace.h can be introduced from outside the subdirectory where the source file is located, it is generally not recommended to do so.

It is strongly recommended that all trace-events be declared directly in the subdirectory where they are used. The only exception is that some shared trace events are defined in the trace-events file in the top-level directory.

The trace files generated in the top-level directory will have the trace/trace-root prefix, rather than just trace , to avoid ambiguity between the trace.h in the current directory and the files in the top-level directory.

Example

src/
│
├── io/
│   ├── channel.c
│   ├── trace-events   <-- 你写的
│   └── trace.h        <-- 你创建，引用 trace-io.h
│
├── hw/arm/
│   ├── xxxx.c
│   ├── trace-events
│   └── trace.h
│
└── meson.build        <-- 声明哪些目录包含 trace-events

After building:

build/trace/
    trace-io.c
    trace-io.h
    trace-hw_arm.c
    trace-hw_arm.h
    trace-events-all     <-- 所有 trace-events 合并

Add a new trace-event

Adding a new trace-event only requires two steps:

• Declare the trace-event in the trace-events file of the corresponding directory
• Add the function call for this event in the target source code that needs to be debugged.

Taking tracing QEMU memory allocation and deallocation as an example, the format of a trace-event is as follows:

qemu_vmalloc(size_t size, void *ptr) "size %zu ptr %p"
qemu_vfree(void *ptr) "ptr %p"

Each event declaration will start with the event name, followed by parameters, and finally a format string for pretty printing。

The format string should reflect the types defined in the trace event. tracing only supports basic scalar types (char, int, long), and does not support floating-point types (float, double)

Pay special attention to int64_t and uint64_t types use PRId64 and PRIu64 respectively, which ensures portability between 32-bit and 64-bit platforms.

The format string must not end with a newline character. It is the backend’s responsibility to adjust the line ending for correct logging.

Once the trace-event is defined, call it directly from the target source code, as shown in the example below:

#include "trace.h"  /* needed for trace event prototype */
void *qemu_vmalloc(size_t size)
{
    void *ptr;
    size_t align = QEMU_VMALLOC_ALIGN;
    if (size < align) {
        align = getpagesize();
    }
    ptr = qemu_memalign(align, size);
    /* Insert the trace-event, the format is: trace_<event-name> */
    trace_qemu_vmalloc(size, ptr);
    return ptr;
}

If there are multiple trace events in a function, you shouldadd a unique identifier at the end of the nameto distinguish them.

In some cases, it may be necessary to perform relatively complex calculations to generate values that are only used as parameters for trace functions. In such cases, you can use the following function to protect this calculation logic:

trace_event_get_state_backends()

When an event is disabled at compile time or runtime, the related calculations will be skipped. If the event is disabled at the compile stage, this check will not incur any performance overhead.

The example code is as follows.

#include "trace.h"  /* needed for trace event prototype */
void *qemu_vmalloc(size_t size)
{
    void *ptr;
    size_t align = QEMU_VMALLOC_ALIGN;
    if (size < align) {
        align = getpagesize();
    }
    ptr = qemu_memalign(align, size);
    if (trace_event_get_state_backends(TRACE_QEMU_VMALLOC)) {
        void *complex;
        /* some complex computations to produce the 'complex' value */
        trace_qemu_vmalloc(size, ptr, complex);
    }
    return ptr;
}

The following situations are very suitable for debugging using tracing:

1. Tracking state changes in the code. Key points in the code usually involve state changes, such as start, stop, allocation, release, etc. State changes are ideal trace events because they help understand the system execution process.
2. Tracking guest operations. Guest I/O accesses (such as reading device registers) are good trace events and can be used to analyze guest interaction behavior.
3. Use associated fields to understand the context of a single line of trace output. For example, track the pointer returned by malloc and its usage as a parameter for free, so that malloc and free operations can be matched. Trace events without context have limited practical value.

Introduction to trace backends

QEMU’s tracing adopts a front-end/back-end separated design, supporting multiple backends. In addition to the log mentioned above, it also supports the lighter-weight simple, as well as ftrace and dtrace.

We can also add more backend support through the tracetool script.

To enable different backends, you can use the following QEMU compilation command:

./configure --enable-trace-backends=simple,dtrace

By running./configure --helpto view all supported backends. If no backend is explicitly selected, the configuration will default to the log backend.

Analyzing trace files

Let’s take the simple backend as an example. This backend writes binary trace logs to a file through an independent thread, which has lower overhead compared to the log backend.

At the same time, the QEMU source repository provides Python scripts for offline trace file analysis. Although the functionality may not be as powerful as specific platform or third-party trace backends, it is portable and requires no special library dependencies.

To format using the simpletrace.py script, you need the trace-events-all file and the binary trace file:

./scripts/simpletrace.py <trace-events-all> <trace-log>

You must ensure that the trace-events-all file used is the same as the one generated when building QEMU, otherwise the trace event declarations may have changed, leading to inconsistent output.

Reference:

Learning Qemu Docs

https://gevico.github.io/learning-qemu-docs/