Cover image for QEMU Address Space Abstraction

QEMU Address Space Abstraction

Created:Updated:
virtualization
Words 2.7k
Views
Visitors
Timeline

Timeline

2025-11-21

  1. init
This article introduces the address space abstraction mechanism in QEMU, discussing in detail the CPU memory access flow and the two core concepts, address-space and memory-region, provided by QEMU to simulate memory and peripheral behaviors. The article further summarizes the address space layout, the address overlap and alias mechanisms of memory-region, the structural definitions and initialization flows of AddressSpace and MemoryRegion, and elucidates how QEMU uses these mechanisms to achieve discrete mapping and remapping of addresses, thereby constructing the physical address space view of the virtual machine.

Environment

Source code

1
2
3
4
5
6
wget https://download.qemu.org/qemu-10.1.2.tar.xz
tar xvJf qemu-10.1.2.tar.xz
cd qemu-10.1.2
mkdir -p output
./configure --prefix=$PWD/output --target-list=aarch64-softmmu,riscv64-softmmu --enable-debug
bear -- make -j$(nproc)

Create .clangd

1
2
3
CompileFlags:
  Add: -Wno-unknown-warning-option
  Remove: [-m*, -f*]

gdb

1
gdb -args ./build/qemu-system-riscv64 -M virt -device edu,id=edu1 -nographic

Basic Introduction

From the CPU’s perspective, all memory access operations are performed on addresses (load/store). The CPU does not care what device corresponds to the address, as long as it can read and write the correct results.

CPU Memory Access Flow

  1. After calculating the target address (via the Arithmetic Logic Unit ALU), the CPU sends it to the address bus, while also providing read/write control signals;
  2. **The device corresponding to the address could be a regular memory block, or an I/O device (**specifically referring to peripherals here), which will respond to the signals on the address bus;
  3. If it is a read operation, the data corresponding to that address is transmitted back via the bus according to the bit width specified by the CPU, generally stored in the register specified by the CPU’s memory access instruction;
  4. If it is a write operation, the data transmitted from the bus is written into the specified address according to the bit width specified by the CPU. If it is an I/O device, it generally updates the register corresponding to that address and may produce side effects.

QEMU Simulating Memory and Peripherals

To simulate the behavior of memory/peripherals, QEMU must at least implement the following mechanisms:

  1. Basic address space management,The ability to distinguish what device it is based on the address delivered by the CPU；
  2. Implementing discrete address mapping, as the addresses of some peripherals are not necessarily contiguous；
  3. Implementing address remapping, for example, the RAM and XRAM of the MCS-51 both start from address 0;

To this end, QEMU provides two concepts,address-space and memory-region(referred to as mr below), the former is used to describe the mapping relationship of the entire address space (different components may see different address spaces), and the latter is used to describe the mapping rules for a certain address range within the address space.

Address Space Layout

Execution:

1
$ ./build/qemu-system-riscv64 -M virt -monitor stdio -s -S -display none

Then we inputinfo mtreethe command to see the layout of the address space:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
(qemu) info mtree
address-space: cpu-memory-0
address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000001000-000000000000ffff (prio 0, rom): riscv_virt_board.mrom
    0000000000100000-0000000000100fff (prio 0, i/o): riscv.sifive.test
    0000000000101000-0000000000101023 (prio 0, i/o): goldfish_rtc
    0000000002000000-0000000002003fff (prio 0, i/o): riscv.aclint.swi
    0000000002004000-000000000200bfff (prio 0, i/o): riscv.aclint.mtimer
    0000000003000000-000000000300ffff (prio 0, i/o): gpex_ioport_window
      0000000003000000-000000000300ffff (prio 0, i/o): gpex_ioport
    0000000004000000-0000000005ffffff (prio 0, i/o): platform bus
    000000000c000000-000000000c5fffff (prio 0, i/o): riscv.sifive.plic
    0000000010000000-0000000010000007 (prio 0, i/o): serial
    0000000010001000-00000000100011ff (prio 0, i/o): virtio-mmio
    0000000010002000-00000000100021ff (prio 0, i/o): virtio-mmio
    0000000010003000-00000000100031ff (prio 0, i/o): virtio-mmio
    0000000010004000-00000000100041ff (prio 0, i/o): virtio-mmio
    0000000010005000-00000000100051ff (prio 0, i/o): virtio-mmio
    0000000010006000-00000000100061ff (prio 0, i/o): virtio-mmio
    0000000010007000-00000000100071ff (prio 0, i/o): virtio-mmio
    0000000010008000-00000000100081ff (prio 0, i/o): virtio-mmio
    0000000010100000-0000000010100007 (prio 0, i/o): fwcfg.data
    0000000010100008-0000000010100009 (prio 0, i/o): fwcfg.ctl
    0000000010100010-0000000010100017 (prio 0, i/o): fwcfg.dma
    0000000020000000-0000000021ffffff (prio 0, romd): virt.flash0
    0000000022000000-0000000023ffffff (prio 0, romd): virt.flash1
    0000000030000000-000000003fffffff (prio 0, i/o): alias pcie-ecam @pcie-mmcfg-mmio 0000000000000000-000000000fffffff
    0000000040000000-000000007fffffff (prio 0, i/o): alias pcie-mmio @gpex_mmio_window 0000000040000000-000000007fffffff
    0000000080000000-0000000087ffffff (prio 0, ram): riscv_virt_board.ram
    0000000400000000-00000007ffffffff (prio 0, i/o): alias pcie-mmio-high @gpex_mmio_window 0000000400000000-00000007ffffffff

address-space: gpex-root
  0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container

address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io

memory-region: pcie-mmcfg-mmio
  0000000000000000-000000000fffffff (prio 0, i/o): pcie-mmcfg-mmio

memory-region: gpex_mmio_window
  0000000000000000-ffffffffffffffff (prio 0, i/o): gpex_mmio_window
    0000000000000000-ffffffffffffffff (prio 0, i/o): gpex_mmio

memory-region: system
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000001000-000000000000ffff (prio 0, rom): riscv_virt_board.mrom
    0000000000100000-0000000000100fff (prio 0, i/o): riscv.sifive.test
    0000000000101000-0000000000101023 (prio 0, i/o): goldfish_rtc
    0000000002000000-0000000002003fff (prio 0, i/o): riscv.aclint.swi
    0000000002004000-000000000200bfff (prio 0, i/o): riscv.aclint.mtimer
    0000000003000000-000000000300ffff (prio 0, i/o): gpex_ioport_window
      0000000003000000-000000000300ffff (prio 0, i/o): gpex_ioport
    0000000004000000-0000000005ffffff (prio 0, i/o): platform bus
    000000000c000000-000000000c5fffff (prio 0, i/o): riscv.sifive.plic
    0000000010000000-0000000010000007 (prio 0, i/o): serial
    0000000010001000-00000000100011ff (prio 0, i/o): virtio-mmio
    0000000010002000-00000000100021ff (prio 0, i/o): virtio-mmio
    0000000010003000-00000000100031ff (prio 0, i/o): virtio-mmio
    0000000010004000-00000000100041ff (prio 0, i/o): virtio-mmio
    0000000010005000-00000000100051ff (prio 0, i/o): virtio-mmio
    0000000010006000-00000000100061ff (prio 0, i/o): virtio-mmio
    0000000010007000-00000000100071ff (prio 0, i/o): virtio-mmio
    0000000010008000-00000000100081ff (prio 0, i/o): virtio-mmio
    0000000010100000-0000000010100007 (prio 0, i/o): fwcfg.data
    0000000010100008-0000000010100009 (prio 0, i/o): fwcfg.ctl
    0000000010100010-0000000010100017 (prio 0, i/o): fwcfg.dma
    0000000020000000-0000000021ffffff (prio 0, romd): virt.flash0
    0000000022000000-0000000023ffffff (prio 0, romd): virt.flash1
    0000000030000000-000000003fffffff (prio 0, i/o): alias pcie-ecam @pcie-mmcfg-mmio 0000000000000000-000000000fffffff
    0000000040000000-000000007fffffff (prio 0, i/o): alias pcie-mmio @gpex_mmio_window 0000000040000000-000000007fffffff
    0000000080000000-0000000087ffffff (prio 0, ram): riscv_virt_board.ram
    0000000400000000-00000007ffffffff (prio 0, i/o): alias pcie-mmio-high @gpex_mmio_window 0000000400000000-00000007ffffffff

It can be seen that:

Address RangeTypeCorresponding Device
0x00001000~0xFFFFROMOnboard Firmware
0x00100000~0x00101023I/OTest device + RTC
0x02000000~0x020BFFFFI/OCLINT (Software/Timer Interrupt)
0x03000000~0x030FFFFFI/OPCI I/O window
0x04000000~0x05FFFFFFI/OPlatform Bus device
0x0C000000~0x0C5FFFFFI/OPLIC
0x10000000~0x100081FFI/OUART + Virtio-MMIO
0x20000000~0x23FFFFFFROMDflash0 / flash1
0x80000000~0x87FFFFFFRAMGuest memory
0x300000000~0x7FFFFFFFFI/OPCI MMIO alias
  • A Guest (representing the emulated object, here referring to the virt machine) can have multiple address-spaces，The address mapping relationships described by each address-space are not necessarily the same, typically I/O and memory。
  • Each address-space corresponds to an mr tree, for example, the root node of the mr corresponding to address-space: memory is system, and the child nodes are arranged in order of address size.

Since mr describes the mapping rules within a specific address range, thus making it very convenient to implement discrete mapping of devices.

Example:

1
2
3
4
5
address-space: cpu-memory-0
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000001000-000000000000ffff (prio 0, rom): riscv_virt_board.mrom
    0000000000100000-0000000000100fff (prio 0, i/o): riscv.sifive.test
    ...
  • cpu-memory-0is address-space
  • systemis the top-level memory-region (the container of the entire virtual system)
  • riscv_virt_board.mrom、sifive.testetc. are all sub memory-region

Each memory-region is attached under the address-space and provides an access handler

memory_region address overlap

mr supports address range overlap at the same level, the overlapping parts are presented according to priority, with the high-priority overlapping part serving as the access target. In (prio 0, type), the prio followed by the priority, there is no address overlap between virt peripherals, so the priority is all 0.

Here is an example:

1
2
3
4
5
6
0x8000   0x70000  0x60000  0x50000  0x40000  0x30000  0x20000  0x10000    0
  |--------|--------|--------|--------|--------|--------|--------|--------|
A:[-----------------------------------------------------------------------] prio:0
B:[-----------------------------------------------------] prio:1
C:[-----------------------------------] prio:2
D:[-----------------] prio:3

For mr A, its address range can be viewed as:

1
2
3
0x8000   0x70000  0x60000  0x50000  0x40000  0x30000  0x20000  0x10000    0
  |--------|--------|--------|--------|--------|--------|--------|--------|
A:[DDDDDDDDDDDDDDDDD|CCCCCCCCCCCCCCCCC|BBBBBBBBBBBBBBBBB|AAAAAAAAAAAAAAAAA]

To implement the above mechanism,QEMU uses alias to describe the overlapping parts in mr. Using alias can place a part of one mr onto another mr, thereby simplifying the complexity of memory simulation (can be analogized to mmap)。

alias example

1
0000000030000000-000000003fffffff (prio 0, i/o): alias pcie-ecam @pcie-mmcfg-mmio 0000000000000000-000000000fffffff
  • alias memory-region maps a region to another address range of the address-space
  • Facilitates different buses accessing the same physical device
  • An alias is also a memory-region, but it internally references another region

AddressSpace

  • Represents The complete address space seen by the CPU or bus
  • Includes:
    • All mapped memory-region
    • Priority (prio) of each region
    • Type (RAM / ROM / I/O / alias)
  • Can be understood as The “physical address space view” of the virtual machine

A CPU can have multiple address-spaces (for example, a risc-v CPU hascpu-memory-0, and also other I/O spaces or PCI bus spaces).

include/system/memory.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
/**
 * struct AddressSpace: describes a mapping of addresses to #MemoryRegion objects
 */
struct AddressSpace {
    /* private: */
    struct rcu_head rcu;
    char *name;
    MemoryRegion *root;

    /* Accessed via RCU.  */
    struct FlatView *current_map;

    int ioeventfd_nb;
    int ioeventfd_notifiers;
    struct MemoryRegionIoeventfd *ioeventfds;
    QTAILQ_HEAD(, MemoryListener) listeners;
    QTAILQ_ENTRY(AddressSpace) address_spaces_link;

    /*
     * Maximum DMA bounce buffer size used for indirect memory map requests.
     * This limits the total size of bounce buffer allocations made for
     * DMA requests to indirect memory regions within this AddressSpace. DMA
     * requests that exceed the limit (e.g. due to overly large requested size
     * or concurrent DMA requests having claimed too much buffer space) will be
     * rejected and left to the caller to handle.
     */
    size_t max_bounce_buffer_size;
    /* Total size of bounce buffers currently allocated, atomically accessed */
    size_t bounce_buffer_size;
    /* List of callbacks to invoke when buffers free up */
    QemuMutex map_client_list_lock;
    QLIST_HEAD(, AddressSpaceMapClient) map_client_list;
};

MemoryRegion

  • Memory or I/O Specific block
  • Includes:
    • Starting address range (offset relative to the address-space)
    • Size
    • Type (RAM / ROM / I/O / alias / container)
    • Sub memory-region (supports nesting)
    • Corresponding read/write handler or object pointer

Can be understood as A “single block” in the address-space, which can be physical memory, device registers, PCI BARs, etc.

include/system/memory.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
/** MemoryRegion:
 *
 * A struct representing a memory region.
 */
struct MemoryRegion {
    Object parent_obj;

    /* private: */

    /* The following fields should fit in a cache line */
    bool romd_mode;
    bool ram;
    bool subpage;
    bool readonly; /* For RAM regions */
    bool nonvolatile;
    bool rom_device;
    bool flush_coalesced_mmio;
    bool unmergeable;
    uint8_t dirty_log_mask;
    bool is_iommu;
    RAMBlock *ram_block;
    Object *owner;
    /* owner as TYPE_DEVICE. Used for re-entrancy checks in MR access hotpath */
    DeviceState *dev;

    const MemoryRegionOps *ops;
    void *opaque;
    MemoryRegion *container;
    int mapped_via_alias; /* Mapped via an alias, container might be NULL */
    Int128 size;
    hwaddr addr;
    void (*destructor)(MemoryRegion *mr);
    uint64_t align;
    bool terminates;
    bool ram_device;
    bool enabled;
    uint8_t vga_logging_count;
    MemoryRegion *alias;
    hwaddr alias_offset;
    int32_t priority;
    QTAILQ_HEAD(, MemoryRegion) subregions;
    QTAILQ_ENTRY(MemoryRegion) subregions_link;
    QTAILQ_HEAD(, CoalescedMemoryRange) coalesced;
    const char *name;
    unsigned ioeventfd_nb;
    MemoryRegionIoeventfd *ioeventfds;
    RamDiscardManager *rdm; /* Only for RAM */

    /* For devices designed to perform re-entrant IO into their own IO MRs */
    bool disable_reentrancy_guard;
};

Initialization process

Let’s understand the relationship between mr and address-space from the QEMU initialization process:

1
2
3
4
5
6
7
8
9
10
main() // system/main.c
|--qemu_init(argc, argv) // system/vlc.c
|  |--cpu_exec_init_all() // system/physmem.c
|  |  |--io_mem_init()
|  |  |  |--memory_region_init_io(&io_mem_unassigned, NULL, &unassigned_mem_ops, NULL, NULL, UINT64_MAX)
|  |  |--memory_map_init()
|  |  |  |--memory_region_init(system_memory, NULL, "system", UINT64_MAX)
|  |  |  |--address_space_init(&address_space_memory, system_memory, "memory")
|  |  |  |--memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io", 65536)
|  |  |  |--address_space_init(&address_space_io, system_io, "I/O")

memory_map_init

system/physmem.c

1
2
3
4
5
6
7
8
9
10
11
12
static void memory_map_init(void)
{
    system_memory = g_malloc(sizeof(*system_memory));

    memory_region_init(system_memory, NULL, "system", UINT64_MAX);
    address_space_init(&address_space_memory, system_memory, "memory");

    system_io = g_malloc(sizeof(*system_io));
    memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io",
                          65536);
    address_space_init(&address_space_io, system_io, "I/O");
}

Formemory_region_init(), eventually callingmemory_region_do_init()：

system/memory.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
void memory_region_init(MemoryRegion *mr,
                        Object *owner,
                        const char *name,
                        uint64_t size)
{
    object_initialize(mr, sizeof(*mr), TYPE_MEMORY_REGION);
    memory_region_do_init(mr, owner, name, size);
}

static void memory_region_do_init(MemoryRegion *mr,
                                  Object *owner,
                                  const char *name,
                                  uint64_t size)
{
    mr->size = int128_make64(size);
    if (size == UINT64_MAX) {
        mr->size = int128_2_64();
    }
    mr->name = g_strdup(name);
    mr->owner = owner;
    mr->dev = (DeviceState *) object_dynamic_cast(mr->owner, TYPE_DEVICE);
    mr->ram_block = NULL;

    if (name) {
        char *escaped_name = memory_region_escape_name(name);
        char *name_array = g_strdup_printf("%s[*]", escaped_name);

        if (!owner) {
            owner = machine_get_container("unattached");
        }

        object_property_add_child(owner, name_array, OBJECT(mr));
        object_unref(OBJECT(mr));
        g_free(name_array);
        g_free(escaped_name);
    }
}

This piece of code will complete the initialization of some key fields of the mr.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/** MemoryRegion:
 *
 * A struct representing a memory region.
 */
struct MemoryRegion {
    Object parent_obj;

    /* private: */
    Object *owner;
    const MemoryRegionOps *ops;
    Int128 size;
    QTAILQ_HEAD(, MemoryRegion) subregions;
    QTAILQ_ENTRY(MemoryRegion) subregions_link;
    ...
};
  • ops points to the actual interface for mr memory access;
  • subregions points to other mrs, through subregions, all associated mrs can be linked together.

Some of this initialization code consists of registered function callbacks, making it inconvenient to clarify the intermediate logic through static code review; gdb can be used to assist.

system_memory is a global variable pointer，pointing to the root node of the mr, we can set watchpoints on system_memory->ops and system_memory->subregions to see in which function they are initialized.

First, observe system_memory->ops, the commands and process are as follows:

watch system_memory->ops
watch system_memory->ops

The first and second hits on the watchpoint are reset operations on ops, and the third hit is where the actual initialization takes place; we can observe the call stack:

ops initialization
ops initialization

It can be seen thatmemory_region_initfn() is in theobject_init_with_type() is called, this is QEMU’s QOM module, which canbe simply understood as the initialization of the mr object, thisThe initialization method is a registered function pointer。

By analogy, we can find out where system_memory->subregions is initialized:

system_memory->subregions
system_memory->subregions

Node relationships

Further watchsystem_memory->subregions

how other subregions are added
how other subregions are added

memory_region_update_container_subregions()the process is very simple, the final execution result is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                      struct MemoryRegion
                    +------------------------+
                    |subregions              |
                    |    QTAILQ_HEAD()       |
                    +------------------------+
                                |
            +-------------------+---------------------+
            |                                         |
            |                                         |
    struct MemoryRegion                      struct MemoryRegion
+------------------------+                +------------------------+
|subregions              |                |subregions              |
|    QTAILQ_HEAD()       |                |    QTAILQ_HEAD()       |
+------------------------+                +------------------------+
      ...                                            ...

Doesn’t it look like a tree structure? In fact, this isRed-black tree。

There is a root field inside the address-space, pointing to the root node of the memory-region, thus implementing one address-space corresponding to one memory-region tree, as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
     AddressSpace
+-------------------------+
|name                     |
|   (char *)              |
|                         |     MemoryRegion(system_memory/system_io)
+-------------------------+          +------------------------+
|root                     |          |subregions              |
|   (MemoryRegion *)      | -------->|    QTAILQ_HEAD()       |
+-------------------------+          +------------------------+
                                                  |
                                                  |
                              +-------------------+---------------------+
                              |                                         |
                   struct MemoryRegion                         struct MemoryRegion
                   +------------------------+                  +------------------------+
                   |subregions              |                  |subregions              |
                   |    QTAILQ_HEAD()       |                  |    QTAILQ_HEAD()       |
                   +------------------------+                  +------------------------+

Each mr corresponds to a specific memory block RAMBlock，This memory block is allocated from the Host, serving as storage for Guest peripheral devices.

mr provides some types for describing storage devices, common ones include RAM, ROM, IOMMU, container.

Back in the QEMU interactive terminal, using the following command, we can print the memory-region distribution of virt and the corresponding peripherals:

qom-tree
qom-tree

memory-region (mr) all represent an accessible address block, such as:

  • RAM/ROM → stored content
  • I/O → device registers
  • alias → address mapping
  • container → ‘container’, managing sub-regions

Each mr has starting offset(relative to the parent region) and Size。

For memory-region container type, which contains other mrs and records the offset of each mr.

container is a special type of memory-region

Itself does not store data, nor does it have direct read/write handlers

Function:

  1. manages sub-memory-regions
  2. provides address offset mapping
  3. establishes a hierarchical structure

In practical application scenarios, wecan use mr containers to create different address hierarchical relationships, which can clearly describe the relationships between different subsystems at the address space level, which is very helpful for achieving modularity.

Reference:

Learning Qemu Docs

https://gevico.github.io/learning-qemu-docs/