Cover image for Qemu TCG

Qemu TCG

(Updated: )
virtualization
Timeline

Timeline

2025-11-23

  1. init
This article introduces the core principles and working mechanisms of the dynamic binary translation engine TCG (Tiny Code Generator) in QEMU, discussing in detail the differences between Guest and Target architectures, the role of TCG IR, as well as the process of translating basic blocks into translation blocks, variable classification, caching mechanisms, and performance optimization techniques such as direct block chaining.

Environment

1
2
3
4
5
6
wget https://download.qemu.org/qemu-10.1.2.tar.xz
tar xvJf qemu-10.1.2.tar.xz
cd qemu-10.1.2
mkdir -p output
./configure --prefix=$PWD/output --target-list=aarch64-softmmu,riscv64-softmmu --enable-debug
bear -- make -j$(nproc)

Create .clangd

1
2
3
CompileFlags:
  Add: -Wno-unknown-warning-option
  Remove: [-m*, -f*]

gdb

1
gdb -args ./build/qemu-system-riscv64 -M virt -device edu,id=edu1 -nographic

QEMU supports multiple accelerators, which can be broadly divided into two types: instruction simulation technology (TCG) and virtualization technology (KVM, HVF), etc.

Common translation techniques:

  • Interpreter: Parses and executes one Guest instruction at a time, repeating the cycle.
  • Static Binary Translation: Translates before program execution. No translation overhead at runtime, but limited optimization scope.
  • Dynamic Binary Translation: Translates dynamically during program execution. Typically translates based on program traces, not the entire code, and can deeply optimize hot code.

TCG (Tiny Code Generator) was originally a C language compiler backend, later evolving into QEMU’sbinary dynamic compilation (translation) engine。

Target & Guest

  • Guest (virtual machine/simulated architecture)

    • Refers tothe CPU architecture simulated by QEMU, which is the CPU type of the system you are running.

    • For example:

      • riscv64(RISC-V 64-bit)
      • aarch64(ARM 64-bit)
      • i386(x86 32-bit)

    • Specify the Guest architecture when starting QEMU, for example:

      1
      qemu-system-riscv64 -machine virt -kernel kernel.elf

    • The Guest determines the instruction set, registers, etc., emulated by QEMU.

  • TCG Target

    • TCG (Tiny Code Generator) is QEMU’s dynamic translator, used to convertGuest instructionsintonative instructions executable by the Host CPU。
    • The TCG Target isthe architecture of the host CPU, which determines on which CPU the machine code generated by QEMU will run.
    • For example:
      • Host =x86_64
      • Guest =riscv64
      • TCG Target =x86_64(because QEMU ultimately generates x86_64 machine code to execute on the host)

TCR Translation

TCG IR

Similar to LLVM, QEMU also defines its own IR, with the process as follows:

1
2
3
4
5
6
7
+---------------+      +----------------+      +---------------+
|               |      |                |      |               |
| Source binary | ---> |    QEMU IR     | ---> | Target binary |
|     code      |      |                |      |     code      |
|               |      |                |      |               |
+---------------+      +----------------+      +---------------+
      Guest                                          Host

Translate guest instructions intoQEMU IR, this is the frontend; the backend translatesQEMU IRinto host instructions

Advantages：

  • Good extensibility: supporting a new frontend (Guest) only requires implementing source -> IR;
  • Easy to pipeline: similar to LLVM, various passes can be introduced to optimize different stages.

Disadvantages：

  • Generally lower performance (relatively speaking)

TCG Translation Process

Basic Process

TCG’s binary translation uses Basic Blocks as the fundamental unit, and the translation output is a Translation Block.

Rules for dividing Basic Blocks:Branch instruction；Privileged instruction/exception；Code segment crossing page。

  1. Guest PC points to a Basic Block
    • **Basic Block（BB）**is a contiguous sequence of instructions without branches.
    • For example, a sequence of sequentially executed instructions in x86/RISC-V.
  2. Check TB Cache, translation cache
    • TB（Translated Block）is the result of a Basic Block being translated by DBT intohost executable code.
    • DBT maintains a hash table or cache, with the key typically being the Guest PC.
  3. Hit TB Cache (Y)
    • If the TB corresponding to the Guest PC has already been generated, directly jump toExec TBfor execution.
    • No need to translate again, improving efficiency.
  4. TB Cache Miss (N)
    • Calltranslationmodule to convert the Guest’s Basic Block into a Translation Block (TB).
    • After translation, the TB is saved to the cache for fast execution next time.
  5. Direct Block Chaining
    • Each TB storesits corresponding Guest PC, and usually also recordsthe next Guest PC(the jump target of the last instruction)
    • After executing a TB, if the next Guest PC already has a TB, it directly jumps to execute the next TB without returning toCheck TB Cache。
    • This is achieved by generating a direct jump instruction at the end of the TB.
    • This optimizes execution efficiency by avoiding returning to the interpreter loop each time, especially for loop-intensive code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
                              +---------------+                    
       +----------------------| Do something  |-------------------+
       |                      +---------------+                   |
       v                                                          |
+--------------+       +----------------+ Y       +---------+     |
|   Guest PC   +------>| Check TB Cache +-------->| Exec TB +-----+
+--------------+       +------+---------+         +---------+      
                              | N                      ^           
                              v                        |           
                        +-------------+                |           
                        | translation |                |           
                        +-----+-------+                |           
                              v                        |           
                       +-----------------+             |           
                       |Save TB to Cache +-------------+           
                       +-----------------+
  • Once a Basic Block is converted to a TB by DBT, the next time the same Basic Block is executed, the TB can be directly fetched from the cache for execution without needing conversion again:

Translation Block

TCG’s binary translation uses code blocks (Basic Blocks) as the basic unit, and the product of translation is a Translation Block.

TCG has three types of variables: temporary, local temporary, and global.

  • globalGeneral system registers are global variables, which are further divided into two types: register and memory. Memory typically defines registers within the CPU.

For the memory type, during backend translation, the guest CPU register values are first loaded into host registers, and after computation, they are stored back into the guest CPU structure. The simulation process involves memory access operations.

For the register type of TCGv, during backend translation, it is directly mapped to host registers, allowing direct access each time. It is typically used to store the pointer to the guest CPU environment.

  • temporaryThe variable’s lifetime is only within a single BB.

  • local temporaryThe variable’s lifetime is within a TB and can span across BBs.

Rules for dividing Basic Blocks:Branch instructions；Privileged instructions/exceptions；Code segment crossing a page。

A BB starts from the end of the previous BB or from a set_label instruction. A BB ends with a branch instruction (brcond_xxx)、goto_tb and exit_tb ends,

  • Prologue：

    • Save Guest CPU state (registers, flags, etc.) to host registers or memory

    • Set up execution environment

  • Epilogue：

    • Restore Guest CPU state

    • Jump back to Dispatcher or directly to the next TB

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
                        +---------------------+                                     
         1)             |                     |                                     
       +----------------+   QEMU TCG engine   +---------------+                     
       |          +---->|                     |<---+          |                     
       |          |     +----------+---^------+    |          |                     
       |          |                |   |   4)      |          | 5)                  
       |          |            3)  |   +------+    |          |                     
       v          |2)              v          |    | 6)       v                     
+---------------+ |        +---------------+  |    |  +---------------+             
|   prologue    | |        |   prologue    |  |    |  |   prologue    |             
+---------------+ |        +---------------+  |    |  +---------------+             
|               | |        |               |  |    |  |               |             
|  Translation  | |        |  Translation  |  |    |  |  Translation  |             
|     Block1    | |        |     Block2    |  |    |  |     Block3    |             
|               | |        |               |  |    |  |               |             
+---------------+ |        +---------------+- |    |  +---------------+             
|   epilogue    | |        |   epilogue    |  |    |  |   epilogue    |             
+------+--------+ |        +-------+-------+  |    |  +------+--------+             
       +----------+                +----------+    +---------+

Direct block chaining

Taking the x86_64 platform as an example, each context switch requires executing about 20 instructions (instructions also perform memory reads and writes), so one of the optimization measures of DBT is toreduce context switches, achieving direct linking between TBs: (for example, direct jump instructions can directly connect two TB blocks, but indirect jump instructions cannot because they depend on runtime computation)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
            1)          +---------------------+                                 
       +----------------+   QEMU TCG engine   +---------------------------+     
       |                +---------------------+                           |     
       v                                                                  |     
+---------------+          +---------------+          +---------------+   |     
|   prologue    |          |   prologue    |   3)     |   prologue    |   |     
+---------------+ +------> +---------------+  +-----> +---------------+   |     
|               | |        |               |  |       |               |   | 5)  
|  Translation  | |        |  Translation  |  |       |  Translation  |   |     
|     Block1    | |        |     Block2    |  |       |     Block3    |   |     
|               | |2)      |               |  |       |               |   |     
+---------------+-+        +---------------+--+       +---------------+---+     
|   epilogue    |          |   epilogue    |          |   epilogue    |         
+------+--------+          +-------+-------+          +------+--------+

PS: The Guest instructions corresponding to two chained TBs need to be on the same Guest page.

Code Buffer

  • code_bufferis the contiguous memory area on the host machine where TCG (Tiny Code Generator) storestranslated Guest instructions (TB).
  • All**Translation Block（TB）**are generated, stored, and executed in this buffer.
  • SubsequentPrologue / TB.code / Epilogueare all located in code_buffer.
1
2
3
4
5
6
7
8
9
10
11
12
13
code_buffer = mmap()                                               
|                                             TCGContext.code_ptr  
v                                              v                   
+-----------+----------+-------------+---------+------------------+
|           |          |             |         |                  |
|  prologue | epilogue |  TB.struct  | TB.code |     ...          | size = Host / dynamic_code_size
|           |          |             |         |                  |
+-----------+----------+-------------+---------+------------------+
^           ^                        ^                             
|           |                        |                             
|           tcg_code_gen_epilogue    |                             
|                                    tb.tc.ptr                     
tcg_qemu_tb_exec

tcg_code_gen_epilogue：

  • Generate the epilogue part of TB and updateTB.tc.ptr

tcg_qemu_tb_exec：

  • Obtain TB.code according to TB.struct and jump to TB.code

TB.struct：

  • Record the meta-information of TB, for example:
    • Guest PC (entry address)
    • TB length
    • Pointer to TB.code
    • Next TB (Direct Block Chaining)

TB.code：

  • Host machine code generated after TCG translation
  • Execution starts here in practice

Return to the QEMU world after epilogue execution

In the early stage of QEMU startup, a function called tcg is executed_init_machine, completing the allocation and initialization of code_buffer.

accel/tcg/tcg-all.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
static int tcg_init_machine(AccelState *as, MachineState *ms)
{
    TCGState *s = TCG_STATE(as);
    unsigned max_threads = 1;

#ifndef CONFIG_USER_ONLY
    CPUClass *cc = CPU_CLASS(object_class_by_name(target_cpu_type()));
    bool mttcg_supported = cc->tcg_ops->mttcg_supported;

    switch (s->mttcg_enabled) {
    case ON_OFF_AUTO_AUTO:
        /*
         * We default to false if we know other options have been enabled
         * which are currently incompatible with MTTCG. Otherwise when each
         * guest (target) has been updated to support:
         *   - atomic instructions
         *   - memory ordering primitives (barriers)
         * they can set the appropriate CONFIG flags in ${target}-softmmu.mak
         *
         * Once a guest architecture has been converted to the new primitives
         * there is one remaining limitation to check:
         *   - The guest can't be oversized (e.g. 64 bit guest on 32 bit host)
         */
        if (mttcg_supported && !icount_enabled()) {
            s->mttcg_enabled = ON_OFF_AUTO_ON;
            max_threads = ms->smp.max_cpus;
        } else {
            s->mttcg_enabled = ON_OFF_AUTO_OFF;
        }
        break;
    case ON_OFF_AUTO_ON:
        if (!mttcg_supported) {
            warn_report("Guest not yet converted to MTTCG - "
                        "you may get unexpected results");
        }
        max_threads = ms->smp.max_cpus;
        break;
    case ON_OFF_AUTO_OFF:
        break;
    default:
        g_assert_not_reached();
    }
#endif

    tcg_allowed = true;

    page_init();
    tb_htable_init();
    tcg_init(s->tb_size * MiB, s->splitwx_enabled, max_threads);

#if defined(CONFIG_SOFTMMU)
    /*
     * There's no guest base to take into account, so go ahead and
     * initialize the prologue now.
     */
    tcg_prologue_init();
#endif

#ifdef CONFIG_USER_ONLY
    qdev_create_fake_machine();
#endif

    return 0;
}

  • All subsequent code translation and execution work revolves around code_buffer
  • The backend management work of TCGContext is also carried out around code_buffer

DecodeTree

1
2
3
4
5
6
7
+---------------+      +----------------+
|               |      |                | 
| Source binary | ---> |    QEMU IR     |
|     code      |      |                |   
|               |      |                |  
+---------------+      +----------------+  
      Guest

Decodetree is a mechanism proposed by Bastian Koppelmann in 2017 when porting RISC-V QEMU. It was proposed mainly because previous instruction decoders (e.g., ARM) used a bunch of switch-case statements for judgment, which were not only hard to read but also difficult to maintain.

Therefore, Bastian Koppelmann proposed the Decodetree mechanism, where developers only need todefine the format of each instruction using Decodetree’s syntax, and then Decodetree can dynamically generate the corresponding instruction decoder.c containing switch-case statements.

Decodetree is essentially a Python script that takes a file defining the instruction format of the architecture as input and outputs the instruction decoder source code file.

1
2
3
4
+-----------+           +-----------+            +-------------------+
| arch-insn |   input   |  scripts/ |   output   | decode-@BASENAME@ |
|  .decode  +---------->| decode.py +----------->|       .c.in       |
+-----------+           +-----------+            +-------------------+
  • input: Instruction encoding format file defined by the architecture
  • output: Source code of the instruction decoder (participates in QEMU compilation)

Decodetree Syntax

Decodetree’s syntax is divided into four parts: Fields, Argument Sets, Formats, and Patterns.

  • Fields, describing fields such as registers and immediates in the instruction encoding;
  • Argument Sets, describing the storage of values extracted from each field in the instruction;
  • Formats, describing the instruction format and generating the corresponding decode function;
  • Pattern, describing the decode method of an instruction.

Decodetree Field

Field defines how to extract the values of each field (e.g., rd, rs1, rs2, imm) from an instruction.

1
2
field_def     := '%' identifier ( unnamed_field )* ( !function=identifier )?
unnamed_field := number ':' ( 's' ) number
  • %identifier
    • The name of the field is defined by the developer.
    • Example:%rd,%rs1,%imm
  • unnamed_field
    • Specifies the bit position of the field in the instruction
    • Format:high_bit : low_bit
    • Optionalsindicates sign extension
    • Example:7:5indicates instruction bits[7:5]
    • Example:31:s20indicates bits[31:20] and requires sign extension
  • !function=identifier
    • After extracting the field value from the instruction, call a function for further processing
    • For example, immediates require sign extension, bit concatenation, or address translation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
以 RISC-V 的 U-type 指令为例：
31                              12  11                 7  6    0
+----------------------------------+--------------------+------+
|            imm[31:12]            |         rd         |opcode| U-type
+----------------------------------+--------------------+------+

可以声明为：
%rd       7:5
%imm_u    12:s20                 !function=ex_shift_12

最后会生成如下的代码：
static void decode_insn32_extract_u(DisasContext *ctx, arg_u *a, uint32_t insn)
{
    a->imm = ex_shift_12(ctx, sextract32(insn, 12, 20)); // is obtained from insn[31:12] and sign-extended, and then calls ex_shift_12() to left-shift by 12 bits
    a->rd = extract32(insn, 7, 5); // Obtained from insn[11:7]
}

Decodetree Argument Sets

Argument Set DefinitionUsed to store the values of each field extracted from the instruction。

1
2
args_def    := '&' identifier ( args_elt )+ ( !extern )?
args_elt    := identifier
  • &identifier
    • Define a name for the Argument Set, customized by the developer
    • Example:&regs,&loadstore
  • args_elt
    • Elements contained in the Argument Set, usually previously defined fields
    • Example:rd,rs1,imm
    • Meaning: Save the extracted values of these fields into this Argument Set
  • !extern
    • Indicates that this Argument Set has already been defined in another Decoder; if this field is present, the corresponding argument set struct will not be generated again
    • Avoid generating duplicate structs
1
2
3
4
5
6
7
8
// Example of U-type instruction format
// &u    imm rd

// Generate the following code
typedef struct {
    int imm;
    int rd;
} arg_u;

Decodetree Format

Format Defines the instruction format(such as R, I, S, B, U, J-type in RISC-V),and will generate the corresponding decode function。

1
2
3
4
5
6
fmt_def      := '@' identifier ( fmt_elt )+
fmt_elt      := fixedbit_elt | field_elt | field_ref | args_ref
fixedbit_elt := [01.-]+
field_elt    := identifier ':' 's'? number
field_ref    := '%' identifier | identifier '=' '%' identifier
args_ref     := '&' identifier

  • The identifier can be customized by the developer, e.g., opr, opi, etc.

  • fmt_elt can use the following different syntaxes:

    • fixedbit_elt contains one or more0、1、.、-，each representing 1 bit in the instruction。
      • .indicates that the bit can be represented by 0 or 1.
      • -indicates that the bit is completely ignored.
    • Declared using Field syntax, e.g., ra:5, rb:5, lit:8

  • field_ref has the following two formats (the examples below refer to the Fields defined above):

    • '%' identifier: directly references a defined Field.

      • For example:%rd, generates:

        1
        a->rd = extract32(insn, 7, 5);

    • identifier '=' '%' identifier: directlyreferences a defined Field, but renames its corresponding argument name via the first identifier. This method can be used to specify different argument names referencing the same Field

      • For example:my_rd=%rd, generates:

        1
        a->my_rd = extract32(insn, 7, 5)

  • args_ref specifies the Argument Set passed to the decode function. If args is not specified_For ref, Decodetree will base on field_elt or field_ref to automatically generate an Argument Set. Additionally,A Format can contain at most one args_ref

When fixedbit_elt or field_ref is defined, all bits of the Format must be defined (can be defined viafixedbit_eltor.to define each bit, spaces are ignored).

1
@opi    ...... ra:5 lit:8    1 ....... rc:5
  • insn[31:26] can be 0 or 1
  • insn[25:21] is ra
  • insn[20:13] is lit
  • insn[12] is fixed to 1
  • insn[11:5] can be 0 or 1
  • insn[4:0] is rc

This Format will generate the following decode function:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Since we did not specify args_ref, Decodetree based on the field_definition of elt, automatically generated arg_decode_the Argument Set insn320
typedef struct {
    int lit;
    int ra;
    int rc;
} arg_decode_insn320;

static void decode_insn32_extract_opi(DisasContext *ctx, arg_decode_insn320 *a, uint32_t insn)
{
    a->ra = extract32(insn, 21, 5);
    a->lit = extract32(insn, 13, 8);
    a->rc = extract32(insn, 0, 5);
}

Take the RISC-V I-type instruction as an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
31           20 19    15 14     12  11                 7  6    0
+--------------+--------+----------+--------------------+------+
|   imm[11:0]  |  rs1   |  funct3  |         rd         |opcode| I-type
+--------------+--------+----------+--------------------+------+

# Fields:
%rs1       15:5
%rd        7:5

# immediates:
%imm_i    20:s12

# Argment sets:
&i    imm rs1 rd

@i       ........ ........ ........ ........ &i      imm=%imm_i     %rs1 %rd

This example will generate the following decode function:

1
2
3
4
5
6
7
8
9
10
11
12
typedef struct {
    int imm;
    int rd;
    int rs1;
} arg_i;

static void decode_insn32extract_i(DisasContext *ctx, arg_i *a, uint32_t insn)
{
    a->imm = sextract32(insn, 20, 12); 
    a->rs1 = extract32(insn, 15, 5);
    a->rd = extract32(insn, 7, 5);
}

Returning to the previous RISC-V U-type instruction, we can define its format like the I-type instruction:

1
2
3
4
5
6
7
8
9
10
# Fields:
%rd        7:5

# immediates:
%imm_u    12:s20                 !function=ex_shift_12

# Argument sets:
&u    imm rd

@u       ....................      ..... ....... &u      imm=%imm_u          %rd

will generate the following decode function:

1
2
3
4
5
6
7
8
9
10
typedef struct {
    int imm;
    int rd;
} arg_u;

static void decode_insn32_extract_u(DisasContext *ctx, arg_u *a, uint32_t insn)
{
    a->imm = ex_shift_12(ctx, sextract32(insn, 12, 20));
    a->rd = extract32(insn, 7, 5);
}

Decodetree Pattern

Pattern actually defines the decode method of an instruction. Decodetree will dynamically generate the corresponding switch-case decode decision branches based on the definition of Patterns.

1
2
3
4
pat_def      := identifier ( pat_elt )+
pat_elt      := fixedbit_elt | field_elt | field_ref | args_ref | fmt_ref | const_elt
fmt_ref      := '@' identifier
const_elt    := identifier '=' number
  • The identifier can be customized by the developer, such as: addl_r, addli, etc.
  • pat_elt can use the following different syntaxes:
    • fixedbit_elt and fixedbit in Format_have the same definition as elt.
    • field_elt is the same as field in Format_The definition of elt is the same.
    • field_ref is the same as field in Format_The definition of ref is the same.
    • args_ref is the same as args in Format_The definition of ref is the same.
    • fmt_ref directly references a previously defined Format.
    • const_elt can directly specify the value of a certain argument.

Pattern example:

1
addl_i   010000 ..... ..... .... 0000000 ..... @opi

Defines the Pattern for the addl_i instruction, where:

  • insn[31:26] is 010000.
  • insn[11:5] is 0000000.
  • References the @opi Format defined in the Format example.
  • Since all bits of the Pattern must beexplicitly defined, therefore @opi must include the format definitions of the remaining insn[25:12] and insn[4:0], otherwise Decodetree will report an error.

Finally addl_the decoder of i will also call trans_addl_i() this translator function

Example

Design a RISC-V arithmetic instruction cube, with the instruction encoding format following R-type, and the semantics as:rd = [rs1] * [rs1] * [rs1]. (Implemented via helper)

1
2
3
4
5
6
7
8
31      25 24  20 19    15 14     12  11                7 6     0
+---------+--------+--------+----------+-------------------+-------+
|  func7  |  rs2   |  rs1   |  funct3  |         rd        | opcode| R-type
+---------+--------+--------+----------+-------------------+-------+
     6                         6                            0x7b
+---------+--------+--------+----------+-------------------+-------+
| 000110  | 00000  |  rs1   |    110   |         rd        |1111011| cube
+---------+--------+--------+----------+-------------------+-------+

The TCG flow of QEMU is:

  1. Generate TB (Translation Block) from Guest instructions
  2. Translate Guest instructions to host instruction sequence

Some instructions are complex, or QEMU does not have ready-made TCG opcodes to generate directly. In this case, it is necessary tohelper：

  • Function: Implement the instruction semantics using host-executable C code
  • TCG translation phase, when encountering this instruction, generate a code snippet that calls the helper
  • execution phase, directly execute the helper implementation

Example C code:

1
2
3
4
5
6
7
8
9
static int custom_cube(uintptr_t addr)
{
    int cube;
    asm volatile (
       ".insn r 0x7b, 6, 6, %0, %1, x0"
        :"=r"(cube)  // store the result in variable cube
        :"r"(addr)); // take the value of variable addr as input
    return cube; 
}

Add instruction decoding for cube in QEMU:

1
2
3
// target/riscv/insn32.decode
@r_cube  ....... ..... .....    ... ..... ....... %rs1 %rd
cube     0000110 00000 .....    110 ..... 1111011 @r_cube

  • DEF_HELPER_3indicates this is a helper with 3 parameters

  • Parameters:

    • env→ CPU state structure (CPURISCVState *env)

    • tl→ instruction operand (e.g., rd)

    • tl→ instruction operand (e.g., rs1)

Add instruction simulation logic for cube (implemented using helper):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// target/riscv/helper.h
DEF_HELPER_3(cube, void, env, tl, tl)

// target/riscv/op_helper.c
void helper_cube(CPURISCVState *env, target_ulong rd, target_ulong rs1)
{
    MemOpIdx oi = make_memop_idx(MO_TEUQ, 0);
    target_ulong val = cpu_ldq_mmu(env, env->gpr[rs1], oi, GETPC());
    env->gpr[rd] = val * val * val;
}

// target/riscv/insn_trans/trans_rvi.c.inc
static bool trans_cube(DisasContext *ctx, arg_cube *a)
{
    //gen_helper_cube(tcg_env, tcg_constant_tl(a->rd), tcg_constant_tl(a->rs1)); // a->rs1 is just a register number
    gen_helper_cube(tcg_env, get_gpr(a->rd), get_gpr(a->rs1));//get general purpose register
    // tcg_env is a global variable
    return true;
}

DEF_HELPER_3 indicates there are three parameters

PositionMeaning
cubehelper name → the corresponding function name ishelper_cube
voidReturn type of helper
envFirst parameter type:CPURISCVState *env(i.e., CPU state)
tlSecond parameter type:target_ulong
tlThird parameter type:target_ulong

env->gpr[rs1]→ Read the value of rs1 from the register file

cpu_ldq_mmu(...)→ Simulate loading the value at the address of rs1 from memory

env->gpr[rd] = val * val * val;→ Perform cube calculation and write the result to rd

arg_cubeis generatedArgument Sets

Write a simple example program:

1
2
3
4
5
6
7
8
9
10
11
int main(void) {
    int a = 3;
    int ret = 0;
    ret = custom_cube((uintptr_t)&a);
    if (ret == a * a * a) {
        printf("ok!\n");
    } else {
        printf("err! ret=%d\n", ret);
    }
    return 0;
}

Compile and run the test:

1
2
3
$ riscv64-linux-musl-gcc main.c -o cube_demo --static
$ qemu-riscv64 cube_demo
$ ok!

TCG IR

Earlier we discussed how to use QEMU’s helper functions to simulate instruction functionality, but generally, helpers are mainly used when IR implementation is inconvenient.

For better performance, it is recommended to use IR for implementation.

The frontend of TCG is responsible for converting instructions of the target architecture into TCG ops, while the backend of TCG is responsible for converting TCG ops into instructions of the target architecture.

Here we mainly focus on the frontend of TCG and discuss the usage of common TCG ops.

Recommended reading:

Documentation/TCG/frontend-ops

https://wiki.qemu.org/Documentation/TCG/frontend-ops

The basic format of a TCG op is as follows:

1
2
3
4
5
6
tcg_gen_<op>[i]_<reg_size>(TCGv<reg_size> args, ...)

op: 操作类型
i: 操作数数量
reg_size: 寄存器大小 (32/64/tl)
args: 操作数列表

Registers

1
TCGv reg = tcg_global_mem_new(TCG_AREG0, offsetof(CPUState, reg), "reg");

Temporaries

1
2
3
4
5
6
7
8
9
10
// Create a new temporary register
TCGv tmp = tcg_temp_new();

// Create a local temporary register.
// Simple temporary register cannot carry its value across jump/brcond,
// only local temporary can.
TCGv tmpl = tcg_temp_local_new();

// Free a temporary register
tcg_temp_free(tmp);

labels

1
2
3
4
5
// Create a new label
int l = gen_new_label();

// Label the current location.
gen_set_label(l);

Ops

Operating on a single register:

1
2
3
4
5
6
7
// ret = arg1
// Assignment_(mathematical_logic): Assign one register to another
tcg_gen_mov_tl(ret, arg1);

// ret = - arg1
// Negation: Negate the sign of a register
tcg_gen_neg_tl(ret, arg1);

Operating on two registers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// ret = arg1 + arg2
// Addition: Add two registers
tcg_gen_add_tl(ret, arg1, arg2);

// ret = arg1 - arg2
// Subtraction: Subtract two registers
tcg_gen_sub_tl(ret, arg1, arg2);

// ret = arg1 * arg2
// Multiplication: Multiply two signed registers and return the result
tcg_gen_mul_tl(ret, arg1, arg2);

// ret = arg1 * arg2
// Multiplication: Multiply two unsigned registers and return the result
tcg_gen_mulu_tl(ret, arg1, arg2);

// ret = arg1 / arg2
// Division_(mathematics): Divide two signed registers and return the result
tcg_gen_div_tl(ret, arg1, arg2);

// ret = arg1 / arg2
// Division_(mathematics): Divide two unsigned registers and return the result
tcg_gen_divu_tl(ret, arg1, arg2);

// ret = arg1 % arg2
// Division_(mathematics): Divide two signed registers and return the remainder
tcg_gen_rem_tl(ret, arg1, arg2);

// ret = arg1 % arg2
// Division_(mathematics) Divide two unsigned registers and return the remainder
tcg_gen_remu_tl(ret, arg1, arg2);

Bit Operations

Logic operations on a single register:

1
2
3
// ret = !arg1
// Negation: Logical NOT an register
tcg_gen_not_tl(ret, arg1);

Logic operations on two registers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// ret = arg1 & arg2
// Logical_conjunction: Logical AND two registers
tcg_gen_and_tl(ret, arg1, arg2);

// ret = arg1 arg2
// Logical_disjunction: Logical OR two registers
tcg_gen_or_tl(ret, arg1, arg2);

// ret = arg1 ^ arg2
// Exclusive_or: Logical XOR two registers
tcg_gen_xor_tl(ret, arg1, arg2);

// ret = arg1 ↑ arg2
// Logical_NAND: Logical NAND two registers
tcg_gen_nand_tl(ret, arg1, arg2);

// ret = arg1 ↓ arg2
// Logical_NOR Logical NOR two registers
tcg_gen_nor_tl(ret, arg1, arg2);

// ret = !(arg1 ^ arg2)
// Logical_equivalence: Compute logical equivalent of two registers
tcg_gen_eqv_tl(ret, arg1, arg2);

// ret = arg1 & ~arg2
// Logical AND one register with the complement of another
tcg_gen_andc_tl(ret, arg1, arg2);

// ret = arg1 ~arg2
// Logical OR one register with the complement of another
tcg_gen_orc_tl(ret, arg1, arg2);

Shift

1
2
3
4
5
6
7
8
9
10
11
// ret = arg1 >> arg2 /* Sign fills vacant bits */
// Arithmetic shift right one operand by magnitude of another
tcg_gen_sar_tl(ret, arg1, arg2);

// ret = arg1 << arg2
// Logical_shift Logical shift left one registerby magnitude of another
tcg_gen_shl_tl(ret, arg1, arg2);

// ret = arg1 >> arg2
// Logical_shift Logical shift right one register by magnitude of another
tcg_gen_shr_tl(ret, arg1, arg2);

Rotation

1
2
3
4
5
6
7
// ret = arg1 rotl arg2
// Circular_shift: Rotate left one register by magnitude of another
tcg_gen_rotl_tl(ret, arg1, arg2);

// ret = arg1 rotr arg2
// Circular_shift Rotate right one register by magnitude of another
tcg_gen_rotr_tl(ret, arg1, arg2);

Byte

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// ret = ((arg1 & 0xff00) >> 8) // ((arg1 & 0xff) << 8)
// Endianness Byte swap a 16bit register
tcg_gen_bswap16_tl(ret, arg1);

// ret = ...see bswap16 and extend to 32bits...
// Endianness Byte swap a 32bit register
tcg_gen_bswap32_tl(ret, arg1);


// ret = ...see bswap32 and extend to 64bits...
// Endianness Byte swap a 64bit register
tcg_gen_bswap64_tl(ret, arg1);

// ret = (int8_t)arg1
// Sign extend an 8bit register
tcg_gen_ext8s_tl(ret, arg1);

// ret = (uint8_t)arg1
// Zero extend an 8bit register
tcg_gen_ext8u_tl(ret, arg1);

// ret = (int16_t)arg1
// Sign extend an 16bit register
tcg_gen_ext16s_tl(ret, arg1);

// ret = (uint16_t)arg1
// Zero extend an 16bit register
tcg_gen_ext16u_tl(ret, arg1);

// ret = (int32_t)arg1
// Sign extend an 32bit register
tcg_gen_ext32s_tl(ret, arg1);

// ret = (uint32_t)arg1
// Zero extend an 32bit register
tcg_gen_ext32u_tl(ret, arg1);

Load/Store

These are for moving data between registers and arbitrary host memory.

Typically used for funky CPU state that is not represented by dedicated registers already and thus infrequently used.

These are not for accessing the target’s memory space;

see the QEMU_XX helpers below for that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// Load an 8bit quantity from host memory and sign extend
tcg_gen_ld8s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load an 8bit quantity from host memory and zero extend
tcg_gen_ld8u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 16bit quantity from host memory and sign extend
tcg_gen_ld16s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 16bit quantity from host memory and zero extend
tcg_gen_ld16u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 32bit quantity from host memory and sign extend
tcg_gen_ld32s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 32bit quantity from host memory and zero extend
tcg_gen_ld32u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 64bit quantity from host memory
tcg_gen_ld64_tl(reg, cpu_env, offsetof(CPUState, reg));

// Alias to target native sized load
tcg_gen_ld_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 8bit quantity to host memory
tcg_gen_st8_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 16bit quantity to host memory
tcg_gen_st16_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 32bit quantity to host memory
tcg_gen_st32_tl(reg, cpu_env, offsetof(CPUState, reg));

// Alias to target native sized store
tcg_gen_st_tl(reg, cpu_env, offsetof(CPUState, reg));

These are for moving data between registers and arbitrary target memory.

The address to load/store via is always the second argument while the first argument is always the value to be loaded/stored.

The third argument (memory index) only makes sense for system targets; user targets will simply specify 0 all the time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
// ret = *(int8_t *)addr
// Load an 8bit quantity from target memory and sign extend
tcg_gen_qemu_ld8s(ret, addr, mem_idx);

// ret = *(uint8_t *)addr
// Load an 8bit quantity from target memory and zero extend
tcg_gen_qemu_ld8u(ret, addr, mem_idx);

// ret = *(int8_t *)addr
// Load a 16bit quantity from target memory and sign extend
tcg_gen_qemu_ld16s(ret, addr, mem_idx);

// ret = *(uint8_t *)addr
// Load a 16bit quantity from target memory and zero extend
tcg_gen_qemu_ld16u(ret, addr, mem_idx);

// ret = *(int8_t *)addr
// Load a 32bit quantity from target memory and sign extend
tcg_gen_qemu_ld32s(ret, addr, mem_idx);

// ret = *(uint8_t *)addr
// Load a 32bit quantity from target memory and zero extend
tcg_gen_qemu_ld32u(ret, addr, mem_idx);

// ret = *(uint64_t *)addr
// Load a 64bit quantity from target memory
tcg_gen_qemu_ld64(ret, addr, mem_idx);

// *(uint8_t *)addr = arg
// Store an 8bit quantity to target memory
tcg_gen_qemu_st8(arg, addr, mem_idx);

// *(uint16_t *)addr = arg
// Store a 16bit quantity to target memory
tcg_gen_qemu_st16(arg, addr, mem_idx);

// *(uint32_t *)addr = arg
// Store a 32bit quantity to target memory
tcg_gen_qemu_st32(arg, addr, mem_idx);

// *(uint64_t *)addr = arg
// Store a 64bit quantity to target memory
tcg_gen_qemu_st64(arg, addr, mem_idx);

Code Flow

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// if (arg1 <condition> arg2) goto label
// Test two operands and conditionally branch to a label
tcg_gen_brcond_tl(TCG_COND_XXX, arg1, arg2, label);

// Goto translation block (TB chaining)
// Every TB can goto_tb to max two other different destinations. There are
// two jump slots. tcg_gen_goto_tb takes a jump slot index as an arg,
// 0 or 1. These jumps will only take place if the TB's get chained,
// you need to tcg_gen_exit_tb with (tb // index) for that to ever happen.
// tcg_gen_goto_tb may be issued at most once with each slot index per TB.
tcg_gen_goto_tb(num);

// Exit translation block
// num may be 0 or TB address ORed with the index of the taken jump slot.
// If you tcg_gen_exit_tb(0), chaining will not happen and a new TB
// will be looked up based on the CPU state.
tcg_gen_exit_tb(num);

// ret = arg1 <condition> arg2
// Compare two operands
tcg_gen_setcond_tl(TCG_COND_XXX, ret, arg1, arg2);

Example

We use IR to implement the cube instruction:

1
2
3
4
5
6
7
8
9
10
11
12
13
// target/riscv/insn_trans/trans_rvi.c.inc
static bool trans_cube(DisasContext *ctx, arg_cube *a)
{
    TCGv dest = tcg_temp_new(); // Allocate a temporary variable
    TCGv rd = get_gpr(ctx, a->rd, EXT_NONE); // Get the rd register
    // Read the value from the memory pointed to by the rs1 register and store it in dest
    tcg_gen_qemu_ld_tl(dest, get_gpr(ctx, a->rs1, EXT_NONE), ctx->mem_idx, MO_TEUQ);
    // Compute the cube and store it in the rd register
    tcg_gen_mul_tl(rd, dest, dest); // rd = dest * dest
    tcg_gen_mul_tl(rd, rd, dest); // rd = rd * dest
    gen_set_gpr(ctx, a->rd, rd);
    return true;
}
1
$ ./build/qemu-system-riscv64 -M virt -d in_asm,op,out_asm -nographic -D cpu.log
  • -d indicates outputting logs
    • in_asm indicates input assembly
    • op indicates intermediate IR
    • out_asm indicates output assembly
  • -D indicates output to file or terminal

Reference:

Learning Qemu Docs

https://gevico.github.io/learning-qemu-docs/