Qemu TCG

Timeline

Timeline

2025-11-23

1. init

This article introduces the basic principles and working mechanisms of QEMU's TCG (Tiny Code Generator) dynamic binary translation engine. The article first distinguishes between instruction simulation technology and virtualization technology, and outlines three types of translation techniques: interpreters, static binary translation, and dynamic binary translation, emphasizing that TCG belongs to the latter. It then elaborates on the concepts of Guest (the simulated CPU architecture) and TCG Target (the host CPU architecture), and explains that TCG uses an LLVM-like intermediate representation (TCG IR) process: the frontend translates Guest instructions into TCG IR, and the backend then translates TCG IR into host instructions. The article focuses on analyzing TCG's translation process: using Basic Blocks as the basic unit, translating Guest code into Translation Blocks (TB), and improving reuse efficiency through TB caching; it also introduces the division rules of Basic Blocks (branch instructions, privileged instructions/exceptions, cross-page), TCG's variable types (temporary, local temporary, global

Environment

wget https://download.qemu.org/qemu-10.1.2.tar.xz
tar xvJf qemu-10.1.2.tar.xz
cd qemu-10.1.2
mkdir -p output
./configure --prefix=$PWD/output --target-list=aarch64-softmmu,riscv64-softmmu --enable-debug
bear -- make -j$(nproc)

Create .clangd

CompileFlags:
  Add: -Wno-unknown-warning-option
  Remove: [-m*, -f*]

gdb

gdb -args ./build/qemu-system-riscv64 -M virt -device edu,id=edu1 -nographic

QEMU supports multiple accel, but they can generally be divided into two types: instruction simulation technology (TCG), virtualization technology (KVM, HVF), etc.

Common translation techniques:

• Interpreter: parses and executes one Guest instruction at a time, repeatedly.
• Static Binary Translation: translation is performed before the program runs. There is no translation overhead at runtime, but the optimization scope is limited.
• Dynamic Binary Translation: dynamically translates while the program is running. It generally translates according to program traces, does not translate everything, and can deeply optimize hot code.

TCG (Tiny Code Generator) was originally a compiler backend for the C language, and later evolved into QEMU’sdynamic binary translation engine。

Target & Guest

• Guest (virtual machine / simulated architecture)

  • refers to The CPU architecture simulated by QEMU, that is, the CPU type of the system you are running.

  • For example:

    • riscv64(RISC-V 64-bit)
    • aarch64(ARM 64-bit)
    • i386(x86 32-bit)

  • Specify the Guest architecture when starting QEMU, for example:

    qemu-system-riscv64 -machine virt -kernel kernel.elf

  • The Guest determines the instruction set, registers, etc. that QEMU simulates.

• TCG Target (TCG target architecture)

  • TCG (Tiny Code Generator) is QEMU’s dynamic translator, used to convert Guest instructions into native instructions executable by the Host CPU。
  • TCG Target is the architecture of the host CPU, which determines on what CPU the machine code generated by QEMU will run.
  • For example:
    • Host =x86_64
    • Guest =riscv64
    • TCG Target =x86_64(because QEMU ultimately generates x86_64 machine code to execute on the host)

TCG Translation

TCG IR

Similar to LLVM, QEMU also defines its own intermediate representation (IR), with the following flow:

+---------------+      +----------------+      +---------------+
|               |      |                |      |               |
| Source binary | ---> |    QEMU IR     | ---> | Target binary |
|     code      |      |                |      |     code      |
|               |      |                |      |               |
+---------------+      +----------------+      +---------------+
      Guest                                          Host       

The frontend translates Guest instructions into TCG IR, and the backend then translates TCG IR into host instructions.

Advantages：

• Good extensibility: to support a new Guest architecture, you only need to implement the frontend from Guest instructions to TCG IR;
• Easy to pipeline: similar to LLVM, various passes can be introduced to optimize different stages.

Disadvantages：

• Performance is generally not high (relatively speaking)

TCG translation flow

Basic flow

TCG’s binary translation uses Basic Blocks as the basic unit, and the translation product is a Translation Block.

Rules for dividing Basic Blocks:Branch instructions；Privileged instructions/exceptions；Code segment spans pages。

1. Guest PC points to a Basic Block
   • Basic Block（BB） It is a contiguous sequence of instructions without branches.
   • For example, a sequence of instructions executed in order in x86/RISC-V.
2. Check TB Cache, the translation cache
   • TB（Translated Block） is a Basic Block translated by DBT into host executable code , the result after.
   • DBT maintains a hash table or cache, with the key typically being the Guest PC.
3. Hit TB Cache (Y)
   • If the TB corresponding to the Guest PC has already been generated, directly jump to Exec TB execution.
   • No need to translate again, improving efficiency.
4. TB Cache miss (N)
   • Call translation module, converting the Guest’s Basic Block into a Translation Block (TB).
   • After translation is complete, the TB is saved to the cache for quick execution next time.
5. Direct Block Chaining
   • Each TB stores its corresponding Guest PC, and usually also records the next Guest PC(the jump target of the last instruction)
   • After executing a TB, if the next Guest PC already has a TB, it directly jumps to execute the next TB, instead of returning toCheck TB Cache。
   • This is achieved by generating a direct jump instruction at the end of the TB.
   • This optimizes execution efficiency, avoiding returning to the interpreter loop every time, especially for loop-intensive code.

                              +---------------+                    
       +----------------------| Do something  |-------------------+
       |                      +---------------+                   |
       v                                                          |
+--------------+       +----------------+ Y       +---------+     |
|   Guest PC   +------>| Check TB Cache +-------->| Exec TB +-----+
+--------------+       +------+---------+         +---------+      
                              | N                      ^           
                              v                        |           
                        +-------------+                |           
                        | translation |                |           
                        +-----+-------+                |           
                              v                        |           
                       +-----------------+             |           
                       |Save TB to Cache +-------------+           
                       +-----------------+                         

• After a Basic Block is converted to a TB by DBT, the next time the same Basic Block is executed, the TB can be directly fetched from the cache and executed without going through translation again:

Translation Block

TCG’s binary translation uses Basic Blocks as the basic unit, and the translation product is a Translation Block.

TCG has three types of variables: temporary, local temporary, and global

• global: Generally, system registers are global variables, and global variables are further divided into two types: register and memory.

  • register type: During backend translation, it is directly mapped to a host register and can be accessed directly, generally used to store the pointer to the guest CPU env.
  • memory type: It generally defines registers in the CPU. During backend translation, the guest CPU structure fields are loaded into host registers, and after computation, stored back into the guest CPU structure. The simulation process involves memory access behavior.

• temporary: The variable’s lifetime is only within one BB and cannot cross branches.

• local temporary: The variable’s lifetime is within one TB and can cross BBs.

Rules for dividing Basic Blocks:Branch instructions；Privileged instructions/exceptions；Code segment spans pages。

A BB starts from the end of the previous BB or a set_label instruction, and ends with a branch instruction (brcond_xxx）、goto_TB and exit_TB ends.

• Prologue (preceding code)：

  • Save Guest CPU state (registers, flags, etc.) to host registers or memory

  • Set up the execution environment

• Epilogue (following code)：

  • Restore Guest CPU state

  • Jump back to the Dispatcher or directly to the next TB

                        +---------------------+                                     
         1)             |                     |                                     
       +----------------+   QEMU TCG engine   +---------------+                     
       |          +---->|                     |<---+          |                     
       |          |     +----------+---^------+    |          |                     
       |          |                |   |   4)      |          | 5)                  
       |          |            3)  |   +------+    |          |                     
       v          |2)              v          |    | 6)       v                     
+---------------+ |        +---------------+  |    |  +---------------+             
|   prologue    | |        |   prologue    |  |    |  |   prologue    |             
+---------------+ |        +---------------+  |    |  +---------------+             
|               | |        |               |  |    |  |               |             
|  Translation  | |        |  Translation  |  |    |  |  Translation  |             
|     Block1    | |        |     Block2    |  |    |  |     Block3    |             
|               | |        |               |  |    |  |               |             
+---------------+ |        +---------------+- |    |  +---------------+             
|   epilogue    | |        |   epilogue    |  |    |  |   epilogue    |             
+------+--------+ |        +-------+-------+  |    |  +------+--------+             
       +----------+                +----------+    +---------+                      

Direct block chaining

Taking the x86_64 platform as an example, each context switch requires executing about 20 instructions (which also read and write memory), so one of the optimization measures of DBT isreducing context switches, achieving direct linking between TBs (direct jump instructions can directly connect two TB blocks, but indirect jump instructions cannot, because they depend on runtime computation):

            1)          +---------------------+                                 
       +----------------+   QEMU TCG engine   +---------------------------+     
       |                +---------------------+                           |     
       v                                                                  |     
+---------------+          +---------------+          +---------------+   |     
|   prologue    |          |   prologue    |   3)     |   prologue    |   |     
+---------------+ +------> +---------------+  +-----> +---------------+   |     
|               | |        |               |  |       |               |   | 5)  
|  Translation  | |        |  Translation  |  |       |  Translation  |   |     
|     Block1    | |        |     Block2    |  |       |     Block3    |   |     
|               | |2)      |               |  |       |               |   |     
+---------------+-+        +---------------+--+       +---------------+---+     
|   epilogue    |          |   epilogue    |          |   epilogue    |         
+------+--------+          +-------+-------+          +------+--------+         

PS: The Guest instructions corresponding to two chained TBs need to be in the same Guest page.

Code Buffer

• code_buffer is where TCG (Tiny Code Generator) stores on the host machine translated Guest instructions (TB) contiguous memory area.
• All Translation Block（TB） are all generated, stored, and executed in this buffer.
• Subsequent Prologue / TB.code / Epilogue are all located in code_buffer.

code_buffer = mmap()                                               
|                                             TCGContext.code_ptr  
v                                              v                   
+-----------+----------+-------------+---------+------------------+
|           |          |             |         |                  |
|  prologue | epilogue |  TB.struct  | TB.code |     ...          | size = Host / dynamic_code_size
|           |          |             |         |                  |
+-----------+----------+-------------+---------+------------------+
^           ^                        ^                             
|           |                        |                             
|           tcg_code_gen_epilogue    |                             
|                                    tb.tc.ptr                     
tcg_qemu_tb_exec                                                   

tcg_code_gen_epilogue：

• Generate the epilogue part of the TB, and updateTB.tc.ptr

tcg_qemu_tb_exec：

• Obtain TB.code from TB.struct, and jump to TB.code

TB.struct：

• Record TB metadata, for example:
  • Guest PC (entry address)
  • TB length
  • Pointer to TB.code
  • Next TB (Direct Block Chaining)

TB.code：

• Host machine code generated by TCG translation
• Execution starts from here at runtime

After the epilogue executes, it returns to QEMU’s main loop (dispatcher).

Early in QEMU startup, a function called tcg is executed_init_machine, completing the allocation and initialization of code_buffer.

accel/tcg/tcg-all.c

static int tcg_init_machine(AccelState *as, MachineState *ms)
{
    TCGState *s = TCG_STATE(as);
    unsigned max_threads = 1;

#ifndef CONFIG_USER_ONLY
    CPUClass *cc = CPU_CLASS(object_class_by_name(target_cpu_type()));
    bool mttcg_supported = cc->tcg_ops->mttcg_supported;

    switch (s->mttcg_enabled) {
    case ON_OFF_AUTO_AUTO:
        /*
         * We default to false if we know other options have been enabled
         * which are currently incompatible with MTTCG. Otherwise when each
         * guest (target) has been updated to support:
         *   - atomic instructions
         *   - memory ordering primitives (barriers)
         * they can set the appropriate CONFIG flags in ${target}-softmmu.mak
         *
         * Once a guest architecture has been converted to the new primitives
         * there is one remaining limitation to check:
         *   - The guest can't be oversized (e.g. 64 bit guest on 32 bit host)
         */
        if (mttcg_supported && !icount_enabled()) {
            s->mttcg_enabled = ON_OFF_AUTO_ON;
            max_threads = ms->smp.max_cpus;
        } else {
            s->mttcg_enabled = ON_OFF_AUTO_OFF;
        }
        break;
    case ON_OFF_AUTO_ON:
        if (!mttcg_supported) {
            warn_report("Guest not yet converted to MTTCG - "
                        "you may get unexpected results");
        }
        max_threads = ms->smp.max_cpus;
        break;
    case ON_OFF_AUTO_OFF:
        break;
    default:
        g_assert_not_reached();
    }
#endif

    tcg_allowed = true;

    page_init();
    tb_htable_init();
    tcg_init(s->tb_size * MiB, s->splitwx_enabled, max_threads);

#if defined(CONFIG_SOFTMMU)
    /*
     * There's no guest base to take into account, so go ahead and
     * initialize the prologue now.
     */
    tcg_prologue_init();
#endif

#ifdef CONFIG_USER_ONLY
    qdev_create_fake_machine();
#endif

    return 0;
}

• All subsequent code translation and execution work revolves around code_buffer.
• The backend management of TCGContext is also carried out around code_buffer.

DecodeTree

+---------------+      +----------------+
|               |      |                | 
| Source binary | ---> |    QEMU IR     |
|     code      |      |                |   
|               |      |                |  
+---------------+      +----------------+  
      Guest                                 

Decodetree is a mechanism proposed by Bastian Koppelmann in 2017 when porting RISC-V QEMU. The mechanism was proposed mainly because previous instruction decoders (e.g., ARM) used a bunch of switch-case statements for judgment. They were not only hard to read but also hard to maintain.

Therefore Bastian Koppelmann proposed the Decodetree mechanism, where developers only need todefine the format of each instruction using Decodetree syntax, and then Decodetree can dynamically generate the corresponding instruction decoder.c containing switch cases.

Decodetree is essentially a Python script that takes a file defining the architecture’s instruction format as input and outputs the instruction decoder source file.

+-----------+           +-----------+            +-------------------+
| arch-insn |   input   |  scripts/ |   output   | decode-@BASENAME@ |
|  .decode  +---------->| decode.py +----------->|       .c.in       |
+-----------+           +-----------+            +-------------------+

• input: The instruction encoding format file defined by the architecture
• output: The source code of the instruction decoder (participates in QEMU compilation)

Decodetree syntax

Decodetree syntax is divided into four parts: Fields, Argument Sets, Formats, and Patterns.

• Fields, describing fields such as registers and immediates in the instruction encoding;
• Argument Sets, describing the values used to store the fields extracted from the instruction;
• Formats, describing the format of the instruction and generating the corresponding decode function;
• Pattern, describing the decode method of an instruction.

Decodetree Field

Field defines how to extract the values of each field (e.g., rd, rs1, rs2, imm) in an instruction.

field_def     := '%' identifier ( unnamed_field )* ( !function=identifier )?
unnamed_field := number ':' ( 's' ) number

• %identifier
  • The name of the field is defined by the developer.
  • Example:%rd,%rs1,%imm
• unnamed_field
  • Specifies the bit position of the field in the instruction
  • Format:high_bit : low_bit
  • OptionalsIndicates sign extension
  • Example:7:5Represents bit[7:5] of the instruction
  • Example:31:s20Represents bit[31:20] and requires sign extension
• !function=identifier
  • After extracting the field value from the instruction, call a function for further processing
  • For example, immediates may require sign extension, bit concatenation, or address translation

以 RISC-V 的 U-type 指令为例：
31                              12  11                 7  6    0
+----------------------------------+--------------------+------+
|            imm[31:12]            |         rd         |opcode| U-type
+----------------------------------+--------------------+------+

可以声明为：
%rd       7:5
%imm_u    12:s20                 !function=ex_shift_12

最后会生成如下的代码：
static void decode_insn32_extract_u(DisasContext *ctx, arg_u *a, uint32_t insn)
{
    a->imm = ex_shift_12(ctx, sextract32(insn, 12, 20)); // It is obtained from insn[31:12], sign-extended, and then calls ex_shift_12() to left shift by 12 bits
    a->rd = extract32(insn, 7, 5); // Obtained from insn[11:7]
}

Decodetree Argument Sets

Argument Set definitionUsed to store the values of the fields extracted from the instruction。

args_def    := '&' identifier ( args_elt )+ ( !extern )?
args_elt    := identifier

• &identifier
  • Defines the name of an Argument Set, customized by the developer
  • Example:&regs,&loadstore
• args_elt
  • The elements contained in an Argument Set, usually previously defined fields
  • Example:rd,rs1,imm
  • Meaning: save the values extracted from these fields into this Argument Set
• !extern
  • Indicates that this Argument Set has already been defined in another Decoder. If this field is present, the corresponding argument set struct will not be generated again
  • Avoid generating duplicate structs

// U-type instruction format example
// &u    imm rd

// Generates the following code
typedef struct {
    int imm;
    int rd;
} arg_u;

Decodetree Format

Format Defines the instruction format (such as R, I, S, B, U, J-type in RISC-V),and generates the corresponding decode function。

fmt_def      := '@' identifier ( fmt_elt )+
fmt_elt      := fixedbit_elt | field_elt | field_ref | args_ref
fixedbit_elt := [01.-]+
field_elt    := identifier ':' 's'? number
field_ref    := '%' identifier | identifier '=' '%' identifier
args_ref     := '&' identifier

• The identifier can be customized by the developer, e.g., opr, opi, etc.

• fmt_elt can use the following different syntaxes:

  • fixedbit_elt contains one or more0、1、.、-，Each represents 1 bit in the instruction。
    • .Indicates that this bit can be represented by either 0 or 1.
    • -Indicates that this bit is completely ignored.
  • Declared using Field syntax, e.g., ra:5, rb:5, lit:8

• field_ref has the following two formats (the examples below refer to the Fields defined above):

  • '%' identifier: directly references a defined Field.

    • For example:%rd, it will generate:

      a->rd = extract32(insn, 7, 5);

  • identifier '=' '%' identifier: directlyreference a defined Field, but uses the first identifier to rename its corresponding argument name. This method can be used to specify different argument names to refer to the same Field

    • For example:my_rd=%rd, it will generate:

      a->my_rd = extract32(insn, 7, 5)

• args_ref specifies the Argument Set passed to the decode function. If no args_ref is specified, Decodetree will use field_elt or field_ref to automatically generate an Argument Set. Additionally,A Format can contain at most one args_ref.

When fixedbit_elt or field_ref is defined, all bits of the Format must be defined (can be defined viafixedbit_eltor.to define each bit; spaces are ignored).

@opi    ...... ra:5 lit:8    1 ....... rc:5

• insn[31:26] can be 0 or 1
• insn[25:21] is ra
• insn[20:13] is lit
• insn[12] is fixed to 1
• insn[11:5] can be 0 or 1
• insn[4:0] is rc

This Format will generate the following decode function:

// Since we did not specify args_ref, therefore Decodetree based on the field_the definition of elt, automatically generated arg_decode_the Argument Set insn320
typedef struct {
    int lit;
    int ra;
    int rc;
} arg_decode_insn320;

static void decode_insn32_extract_opi(DisasContext *ctx, arg_decode_insn320 *a, uint32_t insn)
{
    a->ra = extract32(insn, 21, 5);
    a->lit = extract32(insn, 13, 8);
    a->rc = extract32(insn, 0, 5);
}

Take the RISC-V I-type instruction as an example:

31           20 19    15 14     12  11                 7  6    0
+--------------+--------+----------+--------------------+------+
|   imm[11:0]  |  rs1   |  funct3  |         rd         |opcode| I-type
+--------------+--------+----------+--------------------+------+

# Fields:
%rs1       15:5
%rd        7:5

# immediates:
%imm_i    20:s12

# Argment sets:
&i    imm rs1 rd

@i       ........ ........ ........ ........ &i      imm=%imm_i     %rs1 %rd

This example will generate the following decode function:

typedef struct {
    int imm;
    int rd;
    int rs1;
} arg_i;

static void decode_insn32extract_i(DisasContext *ctx, arg_i *a, uint32_t insn)
{
    a->imm = sextract32(insn, 20, 12); 
    a->rs1 = extract32(insn, 15, 5);
    a->rd = extract32(insn, 7, 5);
}

Returning to the earlier RISC-V U-type instruction, we can define its format just like the I-type instruction:

# Fields:
%rd        7:5

# immediates:
%imm_u    12:s20                 !function=ex_shift_12

# Argument sets:
&u    imm rd

@u       ....................      ..... ....... &u      imm=%imm_u          %rd

will generate the following decode function:

typedef struct {
    int imm;
    int rd;
} arg_u;

static void decode_insn32_extract_u(DisasContext *ctx, arg_u *a, uint32_t insn)
{
    a->imm = ex_shift_12(ctx, sextract32(insn, 12, 20));
    a->rd = extract32(insn, 7, 5);
}

Decodetree Pattern

Pattern actually defines how an instruction is decoded. Decodetree will dynamically generate the corresponding switch-case decode decision branches based on the definition of Patterns.

pat_def      := identifier ( pat_elt )+
pat_elt      := fixedbit_elt | field_elt | field_ref | args_ref | fmt_ref | const_elt
fmt_ref      := '@' identifier
const_elt    := identifier '=' number

• The identifier can be customized by the developer, such as: addl_r, addli, etc.
• pat_elt can use the following different syntaxes:
  • fixedbit_elt and the fixedbit in Format_have the same definition.
  • field_elt and the field in Format_have the same definition.
  • field_ref and the field in Format_have the same definition.
  • args_ref and the args in Format_have the same definition.
  • fmt_ref directly references a defined Format.
  • const_elt can directly specify the value of an argument.

Pattern example:

addl_i   010000 ..... ..... .... 0000000 ..... @opi

Defines the Pattern for the instruction addl_i, where:

• insn[31:26] is 010000.
• insn[11:5] is 0000000.
• It references the @opi Format defined in the Format example.
• Since all bits of the Pattern must beexplicitly defined, therefore @opi must include the format definitions for the remaining insn[25:12] and insn[4:0], otherwise Decodetree will report an error.

Finally, addl_i’s decoder will also call trans_addl_i() this translator function

example

Design a RISC-V arithmetic instruction cube, whose instruction encoding format follows R-type, with the semantics:rd = [rs1] * [rs1] * [rs1](implemented via a helper).

31      25 24  20 19    15 14     12  11                7 6     0
+---------+--------+--------+----------+-------------------+-------+
|  func7  |  rs2   |  rs1   |  funct3  |         rd        | opcode| R-type
+---------+--------+--------+----------+-------------------+-------+
     6                         6                            0x7b
+---------+--------+--------+----------+-------------------+-------+
| 000110  | 00000  |  rs1   |    110   |         rd        |1111011| cube
+---------+--------+--------+----------+-------------------+-------+

QEMU’s TCG flow is:

1. Generate TB (Translation Block) from Guest instructions
2. Translate Guest instructions into host instruction sequences

Some instructions are relatively complex, or QEMU does not have an existing TCG opcode to generate directly. In this case, it is necessary to helper：

• Function: implement the instruction semantics in host-executable C code
• TCG translation phase, when this instruction is encountered, generate a code snippet that calls the helper
• Execution phase, directly execute the helper’s implementation

Example C code:

static int custom_cube(uintptr_t addr)
{
    int cube;
    asm volatile (
       ".insn r 0x7b, 6, 6, %0, %1, x0"
        :"=r"(cube)  // Store the result in the variable cube
        :"r"(addr)); // Take the value of variable addr as input
    return cube; 
}

Add instruction decoding for cube in QEMU:

// target/riscv/insn32.decode
@r_cube  ....... ..... .....    ... ..... ....... %rs1 %rd
cube     0000110 00000 .....    110 ..... 1111011 @r_cube

• DEF_HELPER_3Indicates that this is a helper with 3 parameters

• Parameters:

  • env→ CPU state structure (CPURISCVState *env)

  • tl→ Instruction operand (e.g., rd)

  • tl→ Instruction operand (e.g., rs1)

Add instruction simulation logic for cube (implemented using a helper):

// target/riscv/helper.h
DEF_HELPER_3(cube, void, env, tl, tl)

// target/riscv/op_helper.c
void helper_cube(CPURISCVState *env, target_ulong rd, target_ulong rs1)
{
    MemOpIdx oi = make_memop_idx(MO_TEUQ, 0);
    target_ulong val = cpu_ldq_mmu(env, env->gpr[rs1], oi, GETPC());
    env->gpr[rd] = val * val * val;
}

// target/riscv/insn_trans/trans_rvi.c.inc
static bool trans_cube(DisasContext *ctx, arg_cube *a)
{
    //gen_helper_cube(tcg_env, tcg_constant_tl(a->rd), tcg_constant_tl(a->rs1)); // a->rs1 is just a register number
    gen_helper_cube(tcg_env, get_gpr(a->rd), get_gpr(a->rs1)); // get general purpose register
    // tcg_env is a global variable
    return true;
}

DEF_HELPER_3Indicates that this helper has 3 parameters:

Position	Meaning
cube	helper name → the corresponding final function name ishelper_cube
void	The return type of the helper
env	The type of the first parameter:CPURISCVState *env(i.e., CPU state)
tl	The type of the second parameter:target_ulong
tl	The type of the third parameter:target_ulong

env->gpr[rs1]→ Read the value of rs1 from the register file

cpu_ldq_mmu(...)→ Simulate loading the value at the address in rs1 from memory

env->gpr[rd] = val * val * val;→ Perform the cube calculation and write the result to rd

arg_cubeIt is the Argument Set structure automatically generated by Decodetree based on the Format definition.

Write a simple example program:

int main(void) {
    int a = 3;
    int ret = 0;
    ret = custom_cube((uintptr_t)&a);
    if (ret == a * a * a) {
        printf("ok!\n");
    } else {
        printf("err! ret=%d\n", ret);
    }
    return 0;
}

Compile, run, and test:

$ riscv64-linux-musl-gcc main.c -o cube_demo --static
$ qemu-riscv64 cube_demo
$ ok!

TCG IR

Earlier we discussed how to use QEMU’s helper functions to simulate instruction functionality, but in general, helpers are mainly used when IR implementation is inconvenient.

If you want better performance, it is recommended to use IR for implementation.

The TCG frontend is responsible for converting target architecture instructions into TCG ops, while the TCG backend is responsible for converting TCG ops into target architecture instructions.

Here we mainly focus on the TCG frontend and discuss the usage of common TCG ops.

Recommended reading:

Documentation/TCG/frontend-ops

https://wiki.qemu.org/Documentation/TCG/frontend-ops

The basic format of TCG op is as follows:

tcg_gen_<op>[i]_<reg_size>(TCGv<reg_size> args, ...)

op: 操作类型
i: 操作数数量
reg_size: 寄存器大小 (32/64/tl)
args: 操作数列表

Registers

TCGv reg = tcg_global_mem_new(TCG_AREG0, offsetof(CPUState, reg), "reg");

Temporaries

// Create a new temporary register
TCGv tmp = tcg_temp_new();

// Create a local temporary register.
// Simple temporary register cannot carry its value across jump/brcond,
// only local temporary can.
TCGv tmpl = tcg_temp_local_new();

// Free a temporary register
tcg_temp_free(tmp);

labels

// Create a new label
int l = gen_new_label();

// Label the current location.
gen_set_label(l);

Ops

Operating on a single register:

// ret = arg1
// Assignment_(mathematical_logic): Assign one register to another
tcg_gen_mov_tl(ret, arg1);

// ret = - arg1
// Negation: Negate the sign of a register
tcg_gen_neg_tl(ret, arg1);

Operating on two registers:

// ret = arg1 + arg2
// Addition: Add two registers
tcg_gen_add_tl(ret, arg1, arg2);

// ret = arg1 - arg2
// Subtraction: Subtract two registers
tcg_gen_sub_tl(ret, arg1, arg2);

// ret = arg1 * arg2
// Multiplication: Multiply two signed registers and return the result
tcg_gen_mul_tl(ret, arg1, arg2);

// ret = arg1 * arg2
// Multiplication: Multiply two unsigned registers and return the result
tcg_gen_mulu_tl(ret, arg1, arg2);

// ret = arg1 / arg2
// Division_(mathematics): Divide two signed registers and return the result
tcg_gen_div_tl(ret, arg1, arg2);

// ret = arg1 / arg2
// Division_(mathematics): Divide two unsigned registers and return the result
tcg_gen_divu_tl(ret, arg1, arg2);

// ret = arg1 % arg2
// Division_(mathematics): Divide two signed registers and return the remainder
tcg_gen_rem_tl(ret, arg1, arg2);

// ret = arg1 % arg2
// Division_(mathematics) Divide two unsigned registers and return the remainder
tcg_gen_remu_tl(ret, arg1, arg2);

Bit Operations

Logic operations on a single register:

// ret = !arg1
// Negation: Logical NOT an register
tcg_gen_not_tl(ret, arg1);

Logic operations on two registers:

// ret = arg1 & arg2
// Logical_conjunction: Logical AND two registers
tcg_gen_and_tl(ret, arg1, arg2);

// ret = arg1 | arg2
// Logical_disjunction: Logical OR two registers
tcg_gen_or_tl(ret, arg1, arg2);

// ret = arg1 ^ arg2
// Exclusive_or: Logical XOR two registers
tcg_gen_xor_tl(ret, arg1, arg2);

// ret = arg1 ↑ arg2
// Logical_NAND: Logical NAND two registers
tcg_gen_nand_tl(ret, arg1, arg2);

// ret = arg1 ↓ arg2
// Logical_NOR: Logical NOR two registers
tcg_gen_nor_tl(ret, arg1, arg2);

// ret = !(arg1 ^ arg2)
// Logical_equivalence: Compute logical equivalent of two registers
tcg_gen_eqv_tl(ret, arg1, arg2);

// ret = arg1 & ~arg2
// Logical AND one register with the complement of another
tcg_gen_andc_tl(ret, arg1, arg2);

// ret = arg1 | ~arg2
// Logical OR one register with the complement of another
tcg_gen_orc_tl(ret, arg1, arg2);

Shift

// ret = arg1 >> arg2 /* Sign fills vacant bits */
// Arithmetic shift right one operand by magnitude of another
tcg_gen_sar_tl(ret, arg1, arg2);

// ret = arg1 << arg2
// Logical_shift: Logical shift left one register by magnitude of another
tcg_gen_shl_tl(ret, arg1, arg2);

// ret = arg1 >> arg2
// Logical_shift Logical shift right one register by magnitude of another
tcg_gen_shr_tl(ret, arg1, arg2);

Rotation

// ret = arg1 rotl arg2
// Circular_shift: Rotate left one register by magnitude of another
tcg_gen_rotl_tl(ret, arg1, arg2);

// ret = arg1 rotr arg2
// Circular_shift Rotate right one register by magnitude of another
tcg_gen_rotr_tl(ret, arg1, arg2);

Byte

// ret = ((arg1 & 0xff00) >> 8) | ((arg1 & 0xff) << 8)
// Endianness: Byte swap a 16bit register
tcg_gen_bswap16_tl(ret, arg1);

// ret = byte-swapped arg1 (32-bit)
// Endianness: Byte swap a 32bit register
tcg_gen_bswap32_tl(ret, arg1);

// ret = byte-swapped arg1 (64-bit)
// Endianness: Byte swap a 64bit register
tcg_gen_bswap64_tl(ret, arg1);

// ret = (int8_t)arg1
// Sign extend an 8bit register
tcg_gen_ext8s_tl(ret, arg1);

// ret = (uint8_t)arg1
// Zero extend an 8bit register
tcg_gen_ext8u_tl(ret, arg1);

// ret = (int16_t)arg1
// Sign extend an 16bit register
tcg_gen_ext16s_tl(ret, arg1);

// ret = (uint16_t)arg1
// Zero extend an 16bit register
tcg_gen_ext16u_tl(ret, arg1);

// ret = (int32_t)arg1
// Sign extend an 32bit register
tcg_gen_ext32s_tl(ret, arg1);

// ret = (uint32_t)arg1
// Zero extend an 32bit register
tcg_gen_ext32u_tl(ret, arg1);

Load/Store

These are for moving data between registers and arbitrary host memory.

Typically used for funky CPU state that is not represented by dedicated registers already and thus infrequently used.

These are not for accessing the target’s memory space;

see the QEMU_XX helpers below for that.

// Load an 8bit quantity from host memory and sign extend
tcg_gen_ld8s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load an 8bit quantity from host memory and zero extend
tcg_gen_ld8u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 16bit quantity from host memory and sign extend
tcg_gen_ld16s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 16bit quantity from host memory and zero extend
tcg_gen_ld16u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 32bit quantity from host memory and sign extend
tcg_gen_ld32s_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 32bit quantity from host memory and zero extend
tcg_gen_ld32u_tl(reg, cpu_env, offsetof(CPUState, reg));

// Load a 64bit quantity from host memory
tcg_gen_ld64_tl(reg, cpu_env, offsetof(CPUState, reg));

// Alias to target native sized load
tcg_gen_ld_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 8bit quantity to host memory
tcg_gen_st8_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 16bit quantity to host memory
tcg_gen_st16_tl(reg, cpu_env, offsetof(CPUState, reg));

// Store a 32bit quantity to host memory
tcg_gen_st32_tl(reg, cpu_env, offsetof(CPUState, reg));

// Alias to target native sized store
tcg_gen_st_tl(reg, cpu_env, offsetof(CPUState, reg));

These are for moving data between registers and arbitrary target memory.

The address to load/store via is always the second argument while the first argument is always the value to be loaded/stored.

The third argument (memory index) only makes sense for system targets; user targets will simply specify 0 all the time.

// ret = *(int8_t *)addr
// Load an 8bit quantity from target memory and sign extend
tcg_gen_qemu_ld8s(ret, addr, mem_idx);

// ret = *(uint8_t *)addr
// Load an 8bit quantity from target memory and zero extend
tcg_gen_qemu_ld8u(ret, addr, mem_idx);

// ret = *(int16_t *)addr
// Load a 16bit quantity from target memory and sign extend
tcg_gen_qemu_ld16s(ret, addr, mem_idx);

// ret = *(uint16_t *)addr
// Load a 16bit quantity from target memory and zero extend
tcg_gen_qemu_ld16u(ret, addr, mem_idx);

// ret = *(int32_t *)addr
// Load a 32bit quantity from target memory and sign extend
tcg_gen_qemu_ld32s(ret, addr, mem_idx);

// ret = *(uint32_t *)addr
// Load a 32bit quantity from target memory and zero extend
tcg_gen_qemu_ld32u(ret, addr, mem_idx);

// ret = *(uint64_t *)addr
// Load a 64bit quantity from target memory
tcg_gen_qemu_ld64(ret, addr, mem_idx);

// *(uint8_t *)addr = arg
// Store an 8bit quantity to target memory
tcg_gen_qemu_st8(arg, addr, mem_idx);

// *(uint16_t *)addr = arg
// Store a 16bit quantity to target memory
tcg_gen_qemu_st16(arg, addr, mem_idx);

// *(uint32_t *)addr = arg
// Store a 32bit quantity to target memory
tcg_gen_qemu_st32(arg, addr, mem_idx);

// *(uint64_t *)addr = arg
// Store a 64bit quantity to target memory
tcg_gen_qemu_st64(arg, addr, mem_idx);

Code Flow

// if (arg1 <condition> arg2) goto label
// Test two operands and conditionally branch to a label
tcg_gen_brcond_tl(TCG_COND_XXX, arg1, arg2, label);

// Goto translation block (TB chaining)
// Every TB can goto_tb to max two other different destinations. There are
// two jump slots. tcg_gen_goto_tb takes a jump slot index as an arg,
// 0 or 1. These jumps will only take place if the TB gets chained,
// you need to tcg_gen_exit_tb with (tb | index) for that to ever happen.
// tcg_gen_goto_tb may be issued at most once with each slot index per TB.
tcg_gen_goto_tb(num);

// Exit translation block
// num may be 0 or TB address ORed with the index of the taken jump slot.
// If you tcg_gen_exit_tb(0), chaining will not happen and a new TB
// will be looked up based on the CPU state.
tcg_gen_exit_tb(num);

// ret = arg1 <condition> arg2
// Compare two operands
tcg_gen_setcond_tl(TCG_COND_XXX, ret, arg1, arg2);

example

We use IR to implement the cube instruction:

// target/riscv/insn_trans/trans_rvi.c.inc
static bool trans_cube(DisasContext *ctx, arg_cube *a)
{
    TCGv dest = tcg_temp_new(); // Allocate a temporary variable
    TCGv rd = get_gpr(ctx, a->rd, EXT_NONE); // Get the rd register
    // Read the value in memory pointed to by the value of the rs1 register, and store it into dest
    tcg_gen_qemu_ld_tl(dest, get_gpr(ctx, a->rs1, EXT_NONE), ctx->mem_idx, MO_TEUQ);
    // Compute cube and store it into the rd register
    tcg_gen_mul_tl(rd, dest, dest); // rd = dest * dest
    tcg_gen_mul_tl(rd, rd, dest); // rd = rd * dest
    gen_set_gpr(ctx, a->rd, rd);
    return true;
}

Print IR

$ ./build/qemu-system-riscv64 -M virt -d in_asm,op,out_asm -nographic -D cpu.log

• -dIndicates output log
  • in_asmIndicates input assembly
  • opIndicates intermediate IR
  • out_asmIndicates output assembly
• -DIndicates output to file or terminal

Reference:

Learning Qemu Docs

https://gevico.github.io/learning-qemu-docs/