1. Chip Manual
  2. Introduction to Virtualization
    1. Virtualization Technology
    2. Virtualization Application Scenarios
    3. Important Theoretical Foundations of Virtualization Technology
    4. Hardware-Assisted Virtualization Technology
    5. Full Virtualization and Paravirtualization
    6. Classification of Virtual Machines
    7. Memory Virtualization
    8. I/O virtualization
  3. CPU virtualization
    1. vCPU and VM concepts
    2. Creating Multiple vCPUs
    3. ARMv8/v9 Exception Handling
    4. Hypervisor enters VM
    5. Exit from VM to Hypervisor
      1. HVC system call
      2. Exception Syndrome Register ESR_EL2
  4. Memory Virtualization
    1. ARM64 Page Table Mapping
    2. 2-stage page table translation
      1. Stage 1 page table mapping
      2. Stage 2 page table mapping
      3. S2 concatenated page table
      4. S2 page table attributes
        1. MemAttr
        2. S2AP attribute
      5. Registers related to S2 page table
    3. Experiment 3: Creating S2 Mapping with 4-Level Page Table
    4. Experiment 4: Creating S2 Concatenated Page Table with 43 Bits IPA
    5. Experiment 5: Creating S2 Concatenated Page Table with 40 Bits IPA
    6. Experiment 6: Creating Two-Stage Address Mapping in a Virtual Machine
  5. I/O device virtualization
    1. Three mainstream approaches
    2. Experiment 7: Simulating a serial port device in the hypervisor.
  6. Timer virtualization.
    1. Method 1 for calculating virtual time in a VM.
    2. Method 2 for calculating virtual time in VM
    3. Two ways for Timer to trigger an interrupt
    4. Experiment 8: Enable hypervisor (EL2) physical timer
    5. Experiment 9: Enable virtual timer in VM
  7. Interrupt virtualization
    1. Routing of interrupt handling
    2. Virtual interrupt
    3. GICv2 interrupt controller
      1. GIC interrupt types
      2. GICv2 Registers
      3. Access GIC-400 registers
      4. Virtualization support on GIC v2
      5. GIC v2 Interrupt Virtualization Flow
      6. Two new sets of registers added for GIC v2 virtualization
      7. MMIO in GIC Virtualization
      8. GICH_LRn register implements interrupt injection
    4. Experiment 10: Injecting Interrupts into a VM
    5. Points to note when simulating vDistributor
  8. Virtualization Host Extensions
Cover image for ARM64 Virtualization

ARM64 Virtualization

(Updated: )
virtualization
,
Timeline

Timeline

2025-12-05

  1. init
This article introduces virtualization technology under the ARM64 architecture, discussing in detail its theoretical foundation, hardware-assisted virtualization mechanisms, and the classification of Hypervisors. At the same time, it summarizes the implementation principles of CPU virtualization based on exception levels, memory virtualization using two-stage page tables, and I/O virtualization such as device passthrough under the ARMv8/v9 architecture.

Chip Manual

  • Arm Architecture Reference Manual Armv8, for Armv8-A architecture profile, v8.6
    • Content on Virtualization: D1.5
    • Content on Exceptions:
      • D1.1 Exception levels
      • D1.2 Exception terminology
      • D1.10 Exception entry
      • D1.11 Exception return
      • D1.12 Synchronous exception types, routing and priorities
      • D1.13 Asynchronous exception types, routing, masking and priorities
      • D13.2.37 ESR_EL2, Exception Syndrome Register(EL2)
      • D13.2.138 VBAR_EL2, Vector Base Address Register(EL2)
      • D13.2.47 HCR_EL2, Hypervisor Configuration Register
      • FAR_EL2 Fault Address Register(EL2)
      • D13.2.53 HPFAR_EL2, Hypervisor IPA Fault Address Register
    • Content on Memory Management
      • D5.2.3 Controlling address translation stages
      • D5.2.6 Overview of the VMSAv8-64 address translation stages
      • D5.2.7 The VMSAv8-64 translation table format
      • D5.3 VMSAv8-64 translation table format descriptors
      • D13.2.61 ID_AA64MMFR0_EL1, AArch64 Memory Model Feature Register 0
      • D13.2.146 VTTBR_EL2, Virtualization Translation Table Base Register
      • D13.2.145 VTCR_EL2, Virtualization Translation Control Register
  • Armv8-A virtualization
  • ARM Generic Interrupt Controller Architecture version 2.0, Architecture Specification

Introduction to Virtualization

Virtualization Technology

Virtualization technology, as a resource management technique, can abstract and combine various physical resources of a computer (such as CPU, memory, I/O devices, etc.) and allocate them to multiple virtual machines.

  • Platform Virtualization（platform virtualization): Virtualization for computers and operating systems, such as KVM, etc.
  • Resource Virtualization（resource virtualization): Virtualization for specific system resources, including memory, storage, network resources, etc., such as container technology.
  • Application Virtualization（application virtualization): Including simulation, emulation, interpretation technologies, such as Java Virtual Machine.

Virtualization Application Scenarios

  • Desktop Virtualization
  • Server Virtualization
  • Embedded Virtualization

Important Theoretical Foundations of Virtualization Technology

1974: Paper “Formal Requirements for Virtualizable Third Generation Architectures”

Three Essential Elements for Achieving Virtualization:

  • Resource control: The VMM must be able to manage all system resources.
  • Equivalence: The guest’s runtime behavior is consistent with that on bare metal.
  • Efficiency: Programs running on the guest are not interfered with by the VMM.

The x86 architecture encountered some challenges in implementing virtualization, particularly failing to meet the second condition above. Computer architectures contain two types of instructions.

  • Sensitive instructions: Instructions that operate on certain privileged resources, such as those accessing or modifying virtual machine mode or machine state.
  • Privileged instructions: Instructions with special privileges. These are used only by the operating system or other system software and are generally not directly available to users.

Hardware-Assisted Virtualization Technology

  • In 2005, Intel began introducing hardware virtualization technology into CPUs, known as VT (Virtualization Technology).
  • The basic idea of VT is to create containers that can run virtual machines. In a VT-enabled CPU, there are two operation modes: VMX root mode and VMX non-root mode. Both modes support four privilege levels from Ring 0 to Ring 3, allowing both the hypervisor and virtual machines to freely choose their desired operating levels.

Intel Root Mode and Non-Root Mode
Intel Root Mode and Non-Root Mode

Intel Ring0-Ring3
Intel Ring0-Ring3

Full Virtualization and Paravirtualization

Full Virtualization and Paravirtualization
Full Virtualization and Paravirtualization

Currently, hardware-assisted virtualization technology is mainly used, and these two technologies are rarely used.

Classification of Virtual Machines

  • Hypervisor: A virtual machine monitor, also called VMM (Virtual Machine Manager), the hypervisor sits between the computer hardware and virtual machines, responsible for managing and allocating computer resources to each virtual machine.

    • Type 1: The first type of hypervisor is like a small operating system, designed solely to manage all virtual machines. Common virtualization software includes Xen, ACRN, etc.

    Type1 Hypervisor
    Type1 Hypervisor

    • Type 2: The second type of hypervisor relies on operating systems like Windows or Linux to allocate and manage scheduling resources. Common virtualization software includes VMware Player, KVM, and VirtualBox.

Type2 Hypervisor
Type2 Hypervisor

Memory Virtualization

  • Software Emulation: Shadow Page Tables

    • Low Efficiency

Shadow Page Tables
Shadow Page Tables

  • Hardware Memory Virtualization Technology—

    • Intel: Extended Page Table (EPT) technology
    • ARM: stage1 & stage2 page tables

  • Four types of addresses used in virtualization

    • GVA (Guest Virtual Address): Guest virtual address.
    • GPA (Guest Physical Address): Guest physical address.
    • HVA (Host Virtual Address): Host virtual address.
    • HPA (Host Physical Address): Host physical address

I/O virtualization

  • Software-emulated devicesFor example, with a disk, the hypervisor can create a file or a region on the actual disk to emulate a virtual disk and pass it to the guest.

  • Device pass-through(Device Pass Through). The hypervisor directly assigns a physical device to a specific virtual machine.

  • SR-IOV(Single Root I/O Virtualization) technology

CPU virtualization

vCPU and VM concepts

vCPU and VM Concepts
vCPU and VM Concepts

  • ARMv8/v9 CPU virtualization technology is built on different exception levels

ARMv8/v9 CPU virtualization technology is built on different exception levels
ARMv8/v9 CPU virtualization technology is built on different exception levels

Creating Multiple vCPUs

A physical CPU can create multiple vCPUs, leveraging the OS’s multi-process/multi-threading mechanism

Creating Multiple vCPUs
Creating Multiple vCPUs

A physical CPU can run multiple processes/threads, and the hypervisor can schedule them in a time-sharing manner. When a process switches from EL2 to EL1, a vCPU is created.

ARMv8/v9 Exception Handling

  • Non-Virtualization Scenario: Exception handling is concentrated at EL1, such as exception vector tables, exception handling, interrupt handling, system calls, etc.

Non-Virtualization Scenario
Non-Virtualization Scenario

  • Virtualization Scenario：

    Virtualization Scenario
    Virtualization Scenario

    • For the Hypervisor: It needs to handle exceptions from the hypervisor itself, exceptions from VMs, system calls from VMs, interrupts from VMs, etc.
    • For VM: The VM itself also needs to handle exceptions and system calls from the VM (GuestOS handles app exceptions and system calls)

Hypervisor enters VM

Enter the virtual machine
Enter the virtual machine

  1. Set HCR_EL2.RW to make the CPU run in AArch64
  2. Temporarily disable the vCPU’s MMU (optional)
  3. Set SPSR_EL2 so that after the eret instruction, the CPU jumps to EL1
  4. Set the ELR_EL2 register; after executing eret, jump to the vm_entry function to execute the eret instruction, perform processor mode switching, and make the CPU enter EL1
  5. Set the SP_EL1 stack to point the vCPU to a new stack space
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
.global jump_to_vm
jump_to_vm:
    /* Set EL1 execution state to AArch64 */
    ldr x0, =HCR_HOST_NVHE_FLAGS  //NVHE（Non-Virtualized Host Environment）
    msr hcr_el2, x0
    
    /*  Disable EL1's MMU */
    ldr x0, =SCTLR_VALUE_MMU_DISABLED
    msr sctlr_el1, x0
    
    /* Set SPSR_EL2 to enter EL1h (64-bit) upon return*/
    ldr x0, =SPSR_EL1
    msr spsr_el2, x0
    
    /* Set the virtual machine entry address (ELR_EL2)*/
    adr x0, vm_entry
    msr elr_el2, x0
    
    /* Set the virtual machine stack pointer (SP_EL1)*/
    adr x2, vm_sp
    add x2, x2, #4096
    msr sp_el1, x2
    
    eret

Exit from VM to Hypervisor

  • The guest OS of the VM actively calls HVC
  • An exception occurred in the VM (cannot be handled in EL1, such as a GPA page fault)
  • A hardware interrupt occurred

HVC system call

  • SVC system call: An app running at EL0 requests to enter the OS at EL1.

  • HVC system call: A VM running at EL1 requests to enter the hypervisor at EL2.

  • SMC system call: Software running in the hypervisor or VM requests to enter the secure firmware at EL3

Instruction format of HVC
Instruction format of HVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#define HVC_CALL(which, arg0, arg1, arg2) ({ \
    register unsigned long a0 asm ("x0") = (unsigned long)(arg0); \
    register unsigned long a1 asm ("x1") = (unsigned long)(arg1); \
    register unsigned long a2 asm ("x2") = (unsigned long)(arg2); \
    register unsigned long a8 asm ("x8") = (unsigned long)(which); \
    asm volatile ("hvc #0" \
                  : "+r" (a0), "r" (a1), "r" (a2), "r" (a8) \
                  : "memory"); \
    a0; \
})

#define HVC_CALL_0(which) HVC_CALL(which, 0, 0, 0)
#define HVC_CALL_1(which, arg0) HVC_CALL(which, arg0, 0, 0)
#define HVC_CALL_2(which, arg0, arg1) HVC_CALL(which, arg0, arg1, 0)

Follows the ARM64 standard function call convention

  • x0: Return value / First parameter
  • x1,x2: Parameter
  • x8: HVC function number (i.e.,which）

Exception Syndrome Register ESR_EL2

image-20251204212443966
image-20251204212443966

The ESR register contains a total of 4 fields, among which:

  • Bits 32 to 63 are reserved bits.
  • Bit 26~31，EC（Exception Class), this field indicates the type of exception that occurred and is also used to index the ISS field.
  • Bit 25，IL（Instruction Length for synchronous exceptions), indicates the instruction length of the synchronous exception.
  • Bit 0~24，ISS（Instruction Specific Syndrome) specific exception instruction encoding. This exception instruction encoding table depends on different exception types, and different exception types have different encoding formats.

Exception Class

  • Range Division：
    • 0x00–0x2C: Reserved forSynchronous Exception
    • 0x2D–0x3F: Reserved forSynchronous or Asynchronous Exception
  • Reserved：
    • 0x02,0x0A–0x0B,0x0F,0x10,0x14,0x1B,0x1D–0x1F,0x23,0x27,0x29–0x2E,0x36–0x37,0x39,0x3B,0x3D–0x3F
    • Programming these values will result inCONSTRAINED UNPREDICTABLEbehavior
EC Value (Binary)EC Value (Hexadecimal)Exception Type DescriptionDepends on AArch32 / Feature
0b0000000x00Unknown reason(Unknown reason)General
0b0000010x01Trapped WFI/WFE(Trapped WFI or WFE instruction)General
0b0000110x03MCR/MRC (coproc=0b1111)(AArch32 coprocessor access)Requires AArch32 support
0b0001000x04MCRR/MRRC (coproc=0b1111)Requires AArch32 support
0b0001010x05MCR/MRC (coproc=0b1110)Requires AArch32 support
0b0001100x06LDC/STC(Debug data transfer instruction)Requires AArch32 support
0b0001110x07FP/SVE disabled by CPACR/CPTx.FPEN(Floating-point/SVE access disabled)General (requires FP/SVE)
0b0010000x08VMRS（ID group trap）Requires AArch32 support
0b0010010x09Pointer Authentication instruction disabled(HCR_EL2.API=0, etc.)Requires ARMv8.3-PAuth
0b0011000x0CMRRC (coproc=0b1110)Requires AArch32 support
0b0011010x0DBranch Target Exception (BTI)Requires ARMv8.5-BTI
0b0011100x0EIllegal Execution state(Illegal execution state) or PC alignment faultGeneral
0b0100010x11SVC in AArch32(Reported to EL2 only when HCR_EL2.TGE=1)Requires AArch32 support
0b0100100x12HVC in AArch32(HVC not disabled)Requires AArch32 support
0b0100110x13SMC in AArch32(Reported to EL2 only when HCR_EL2.TSC=1)Requires AArch32 support
0b0101010x15SVC in AArch64Requires AArch64 support
0b0101100x16HVC in AArch64(HVC not disabled)Requires AArch64 support
0b0101110x17SMC in AArch64(Reported to EL2 only when HCR_EL2.TSC=1)Requires AArch64 support
0b0110000x18Trapped MSR/MRS/System instructions(AArch64 system register access)Requires AArch64 support
0b0110010x19SVE disabled by CPACR/CPTx.ZENRequires SVE support
0b0110100x1ATrapped ERET/ERETAA/ERETABRequires ARMv8.3-PAuth + NV
0b0111000x1CPointer Authentication failureRequires ARMv8.3-FPAC
0b1000000x20Instruction Abort from lower EL(Instruction fetch exception from a lower exception level)General
0b1000010x21Instruction Abort at same EL(Instruction fetch exception from the same exception level)General
0b1000100x22PC alignment fault(PC misaligned)General
0b1001000x24Data Abort from lower EL(Data abort from a lower Exception level)General
0b1001010x25Data Abort at same EL(Including nested virtualization VNCR related)General
0b1001100x26SP alignment fault(Stack pointer misalignment)General
0b1010000x28Trapped FP exception (AArch32)Requires AArch32 + FP trap support
0b1011000x2CTrapped FP exception (AArch64)Requires AArch64 + FP trap support
0b1011110x2FSError interrupt(Synchronous External Abort)General
0b1100000x30Breakpoint from lower ELGeneral
0b1100010x31Breakpoint at same ELGeneral
0b1100100x32Software Step from lower ELGeneral
0b1100110x33Software Step at same ELGeneral
0b1101000x34Watchpoint from lower ELGeneral
0b1101010x35Watchpoint at same EL(Including VNCR)General
0b1110000x38BKPT instruction (AArch32)Requires AArch32 support
0b1110100x3AVector Catch (AArch32)Requires AArch32 support
0b1111000x3CBRK instruction (AArch64)Requires AArch64 support
1
2
3
4
.global hvc_call
hvc_call:
	hvc 0x0
	ret

Exit to VM, HVC system call, save 10 to x0

1
hvc_call(10);

Trigger a synchronous exception in Lower EL using AArch64, below is the exception vector table

1
2
3
4
5
/ * Lower EL using AArch64 */
vtentry __do_vcpu_exit
vtentry __invalid
vtentry __invalid
vtentry __invalid

After triggering the exception, jump to the exception vector table__do_vcpu_exitFunction

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
_do_vcpu_exit:
    vm_exit 1                 
    mrs x25, esr_el2          // Read the ESR_EL2 register (Exception Syndrome Register), save to x25
    lsr x24, x25, #ESR_ELX_EC_SHIFT   // Right shift to get the EC field ([31:26]), i.e., Exception Class
    cmp x24, #ESR_ELX_EC_HVC64       // Compare if it is an HVC64 exception (EC == 0b010110)
    b.eq vm_hvc                // If equal, jump to vm_hvc to handle HVC

    mov x0, sp                 // Set parameter: x0 = current stack pointer
    mov x1, #BAD_SYNC          // Set error code: BAD_SYNC (indicating synchronization exception)
    mrs x2, esr_el2            // Re-read ESR_EL2 as context information
    bl bad_mode                // Jump to bad_mode to handle unknown/illegal exceptions

vm_hvc:
    mov x0, sp                 // Set parameter: x0 = stack pointer
    bl vm_hvc_handler         // Call HVC handler function
    vm_entry 1                // Return to continue running the virtual machine (enter VM)

vm_exitandvm_entryMacro implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
#define S_FRAME_SIZE 272 /* sizeof(struct pt_regs)	// */
#define S_X0 0 /* offsetof(struct pt_regs, regs[0])	// */
#define S_X1 8 /* offsetof(struct pt_regs, regs[1])	// */
#define S_X2 16 /* offsetof(struct pt_regs, regs[2])	// */
#define S_X3 24 /* offsetof(struct pt_regs, regs[3])	// */
#define S_X4 32 /* offsetof(struct pt_regs, regs[4])	// */
#define S_X5 40 /* offsetof(struct pt_regs, regs[5])	// */
#define S_X6 48 /* offsetof(struct pt_regs, regs[6])	// */
#define S_X7 56 /* offsetof(struct pt_regs, regs[7])	// */
#define S_X8 64 /* offsetof(struct pt_regs, regs[8])	// */
#define S_X10 80 /* offsetof(struct pt_regs, regs[10])	// */
#define S_X12 96 /* offsetof(struct pt_regs, regs[12])	// */
#define S_X14 112 /* offsetof(struct pt_regs, regs[14])	// */
#define S_X16 128 /* offsetof(struct pt_regs, regs[16])	// */
#define S_X18 144 /* offsetof(struct pt_regs, regs[18])	// */
#define S_X20 160 /* offsetof(struct pt_regs, regs[20])	// */
#define S_X22 176 /* offsetof(struct pt_regs, regs[22])	// */
#define S_X24 192 /* offsetof(struct pt_regs, regs[24])	// */
#define S_X26 208 /* offsetof(struct pt_regs, regs[26])	// */
#define S_X28 224 /* offsetof(struct pt_regs, regs[28])	// */
#define S_FP 232 /* offsetof(struct pt_regs, regs[29])	// */
#define S_LR 240 /* offsetof(struct pt_regs, regs[30])	// */
#define S_SP 248 /* offsetof(struct pt_regs, sp)	// */
#define S_PC 256 /* offsetof(struct pt_regs, pc)	// */
#define S_PSTATE 264 /* offsetof(struct pt_regs, pstate)	// */

#define BAD_SYNC  0
#define BAD_IRQ   1
#define BAD_FIQ   2
#define BAD_ERROR 3




.macro kernel_exit el
   // Load ELR, SPSR, restore the values of PC and pstate
   ldp x21, x22, [sp, #S_PC]
   .if \el == 1
   		ldr x23, [sp, #S_SP]
   		msr sp_el1, x23
   .endif
   
   msr elr_el2, x21
   msr spsr_el2, x22

   ldp x0, x1, [sp, #16 * 0]
   ldp x2, x3, [sp, #16 * 1]
   ldp x4, x5, [sp, #16 * 2]
   ldp x6, x7, [sp, #16 * 3]
   ldp x8, x9, [sp, #16 * 4]
   ldp x10, x11, [sp, #16 * 5]
   ldp x12, x13, [sp, #16 * 6]
   ldp x14, x15, [sp, #16 * 7]
   ldp x16, x17, [sp, #16 * 8]
   ldp x18, x19, [sp, #16 * 9]
   ldp x20, x21, [sp, #16 * 10]
   ldp x22, x23, [sp, #16 * 11]
   ldp x24, x25, [sp, #16 * 12]
   ldp x26, x27, [sp, #16 * 13]
   ldp x28, x29, [sp, #16 * 14]

   ldr lr, [sp, #S_LR]
   // restore sp
   add sp, sp, #S_FRAME_SIZE
   eret
.endm



.macro kernel_entry el
   sub sp, sp, #S_FRAME_SIZE
   stp x0, x1, [sp, #16 * 0]
   stp x2, x3, [sp, #16 * 1]
   stp x4, x5, [sp, #16 * 2]
   stp x6, x7, [sp, #16 * 3]
   stp x8, x9, [sp, #16 * 4]
   stp x10, x11, [sp, #16 * 5]
   stp x12, x13, [sp, #16 * 6]
   stp x14, x15, [sp, #16 * 7]
   stp x16, x17, [sp, #16 * 8]
   stp x18, x19, [sp, #16 * 9]
   stp x20, x21, [sp, #16 * 10]
   stp x22, x23, [sp, #16 * 11]
   stp x24, x25, [sp, #16 * 12]
   stp x26, x27, [sp, #16 * 13]
   stp x28, x29, [sp, #16 * 14]

   .if \el == 1
   		msr x21, sp_el1
   .else
      	// Position of x21 at the top of the stack
   		add x21, sp, #S_FRAME_SIZE
   .endif

   mrs x22, elr_el2
   mrs x23, spsr_el2
   // Save lr and sp to sp+#S_LR(pt_regs->regs[30]) and sp+#S_LR+8(pt_Position of regs->sp)
   stp lr, x21, [sp, #S_LR]
   // Save elr_eln and spsr_eln to sp+#S_PC(pt_regs->PC) and sp+#S_PC+8(pt_regs->pstate)
   // Here elr_eln is PC because when entering an exception, the hardware automatically saves the PC and PSTATE before the exception to ELR_ELn
   // and SPSR_ELn (The Saved Program Status Register)
   stp x22, x23, [sp, #S_PC]

.endm

vm_hvc_handlerImplementation

1
2
3
4
void vm_hvc_handler(struct pt_regs *regs){
    unsigned int el = read_sysreg(CurrentEL)>>2;
    printk("%s jump into hypervisor(el=%d), hvccall num: %d\n", __func__, el, regs->regs[0]);
}

Memory Virtualization

ARM64 Page Table Mapping

Non-virtualization Scenario
Non-virtualization Scenario

Only one-stage address mapping Stage 1, which translates virtual addresses to physical addresses

In a virtualization scenario
In a virtualization scenario

Two-stage address mapping:

  • Stage 1: GVA -> GPA within the VM
  • Stage 2: HVA to HPA within the hypervisor

Each stage of mapping goes through a 3-level or 4-level page table

From a system perspective

Virtualization’s 2-stage address translation

  1. VA (Virtual Address) -> IPA (Intermediate Physical Address)(ARM architecture manual terminology)
  2. IPA-> PA(Intermediate Physical Address)

Page table mapping from a system perspective
Page table mapping from a system perspective

The hypervisor itself also has memory access requirements, using TTBR0_EL2, which is another set of page tables, meaning there are three sets of page tables in total

2-stage page table translation

  • 2-stage requires more memory consumption
  • VA->IPA->PA translation: may involve up to 16 memory access operations
  • Both Stage 1 and Stage 2 can be cached in the TLB

2-stage page table translation
2-stage page table translation

Stage 1 page table mapping

  • Non-virtualization: VA -> PA

  • Virtualization: VA -> IPA

Major factors affecting page table mapping

  • Input address size: determined by TCR_ELx.TxSZ, specifies the VA range and the number of page table levels. Typically 48 bits
  • Output address size: determined by ID_AA64MMFR0_EL1.PARange, the maximum physical memory supported by the system, is filled into TCR_ELx.{I}PS
  • Page granularity: specified by TG0 and TG1 in TCR_ELx

Stage 2 page table mapping

  • Virtualization: IPA (GPA) -> PA

Major factors affecting page table mapping

  • Input address size: the maximum IPA is constrained by the system PA, which is determined by ID_AA64MMFR0_EL1.PARange.

The GPA in a VM cannot exceed the physical memory size.

  • Output address size: determined by ID_AA64MMFR0_EL1.PARange, the maximum physical memory supported by the system, is filled into VTCR_EL2.PS
  • Page granularity: specified by TG0 in VTCR_EL2

Stage 2 page table mapping supports 4-level or 3-level page table mapping

  • Input address size （PA size）
  • Page table granularity (4KB? 16KB? 64KB?)
  • At which level to start traversing the page table? (VTCR_EL2.SL0)

How to know the PA?

Taking the Cortex-A72 processor as an example, by consulting the Cortex-A72 Technical Reference Manual, it can be known thatThe Physical Address range supports up to 44 bits，16TB。
For newer IPs such as Cortex-A77, A78, X2, etc., the PA supports up to 40 bits, 1TB.

PA Size
PA Size

T0SZ is 64 -Support PA size

Taking 4KB page granularity as an example

  • When PA is 44 bits, it supports 4-level page table mapping
  • When PA is 42 bits, 40 bits, 36 bits, 32 bits, it supports 3-level page table mapping

When PA is 42 bits and 40 bits, S2 concatenated page tables can be used to compress the 4-level page table into 3 levels

S2 concatenated page table

Taking 40 bits + 4KB as an example

S2 concatenated page table: taking 40 bits + 4KB as an example
S2 concatenated page table: taking 40 bits + 4KB as an example

An extra bit 39 is used for the L0 page table, which has two page table entries, each pointing to an L1 page table

Concatenated page table
Concatenated page table

After using the concatenated page table, the two L1 page tables pointed to by the two page table entries of the original L0 page table are concatenated together, making the L1 page table equivalent to twice the original size.

  • The L1 index range also becomes twice the original.
  • The MMU indexes directly from the L1 page table, thus turning a 4-level page table into a 3-level page table.

Advantage: It can avoid the overhead caused by an additional level of translation.

  • The maximum number of concatenated tables supported is 16.
  • The VTCR_EL2.SL0 field is used to indicate: from which level of page table to start indexing.

VTCR_EL2.SL0
VTCR_EL2.SL0

  • IA: Input Address, IA[47:12]-IA[39:12] indicates using a 40 to 48-bit virtual address.
  • The first row ‘Tables’ indicates the number of tables that can be concatenated, with a maximum of 16; 1 means no concatenation, 2 means two page tables are concatenated.
  • The first column ‘Initial lookup level (SL0 value)’ indicates from which level of page table the hardware starts traversing, by setting VTCR_EL2.SL0.

S2 concatenated page table
S2 concatenated page table

Taking 43 bits + 4KB page granularity as an example, if using the S2 concatenated page table, the L1 page table will be formed by concatenating 16 tables.

S2 page table attributes

S2 page table attributes are slightly different from S1 page table attributes.

S2 page table attributes
S2 page table attributes

MemAttr

S2 page table attribute MemAttr
S2 page table attribute MemAttr

S2 page table attribute MemAttr
S2 page table attribute MemAttr

S2 page table attribute MemAttr
S2 page table attribute MemAttr

S2AP attribute

S2 page table attribute S2AP
S2 page table attribute S2AP

S2 page table attribute S2AP
S2 page table attribute S2AP

S2 page table attribute S2AP
S2 page table attribute S2AP

Registers related to S2 page table

  • Page table base address register:VTTBR_EL2

  • S2 page table translation control register:VTCR_EL2

    VTCR_EL2
    VTCR_EL2

    • PS indicates the maximum physical memory size

    • TG0 indicates page granularity: 4KB, 16KB, 64KB

    • SL0 indicates which level of page table the MMU starts searching from

    • T0SZ is used to specify the range of IPA, which is constrained by PA. It indicates the IPA range 264−T0SZ2^{64-\text{T0SZ}}

Hypervisor configuration register:HCR_EL2。

  • VM field: Enable S2 page table mapping

Experiment 3: Creating S2 Mapping with 4-Level Page Table

Experiment Objective: Understand the S2 page table of ARM64 architecture

Experiment Requirement: Based on Experiment 2, enable the HCR_EL2_VM field, enable the S2 MMU page table. What will happen? Please analyze

This experiment uses ‘IPA 44Bits + 4KB page + 4-level mapping’ to create the S2 page table. After enabling the S2 page table mapping, when jumping to the VM, executing the first instruction triggers a page fault exception because the first instruction has not established a mapping. This experiment uses page fault exception handling to build the S2 page table page by page

TODO

Experiment 4: Creating S2 Concatenated Page Table with 43 Bits IPA

Experiment Objective: Understand the S2 concatenated page table of ARM64 architecture

Experiment Requirement: Based on the previous experiment, this experiment creates an S2 concatenated page table with IPA bits set to 43 and a page granularity of 4KB. Draw a structural diagram of this mapping.

TODO

Experiment 5: Creating S2 Concatenated Page Table with 40 Bits IPA

Experiment Objective: Understand the S2 concatenated page table of ARM64 architecture
Experiment Requirement: Based on the previous experiment, this experiment creates an S2 concatenated page table with IPA bits set to 40 and a page granularity of 4KB. Draw a structural diagram of this mapping.

Experiment 6: Creating Two-Stage Address Mapping in a Virtual Machine

Experiment Objective: Understand the 2-stage page table mapping of ARM64 architecture, establish the second-stage address mapping, i.e., GPA to HPA mapping, using identity mapping for GPA to HPA.

  • Establish the first-stage address mapping, i.e., GVA to GPA mapping, using non-identity mapping.

Experiment Requirement:

  • Allocate a page in the Hypervisor. Since it is an identity mapping in the Hypervisor, gpa = hpa.
  • Write a value into this gpa, for example, 0x12345678.
  • When switching to the VM, pass this gpa to the VM.
  • Create a mapping from gva to gpa in the VM, assuming the gva address is 0x80000000.
  • Then read the value at the gva address in the VM to see if it is 0x12345678.

Please draw a diagram of all mappings in this system, including those inside the VM and Hypervisor.

TODO

Mapping diagram
Mapping diagram

Three sets of page tables
Three sets of page tables

I/O device virtualization

Three mainstream approaches

  1. Full software virtualization: Uses the trap-and-emulate approach to fully simulate device behavior.

  2. Paravirtualization: Uses efficient front-end and back-end drivers to reduce VM trap events and improve I/O device virtualization performance, such as VirtIO technology.

  3. Hardware-assisted virtualization: Hardware direct passthrough, IOMMU, and SR-IOV (Single Root I/O Virtualization)

The process of simulating MMIO register access (trap & emulate)

The process of simulating MMIO register access (trap & emulate)
The process of simulating MMIO register access (trap & emulate)

Parse the triggering instruction

  • VM accesses MMIO, triggering “Data Abort from a lower Exception level” exception.
  • No need to manually parse the exception instruction encoding; the hardware has already parsed it. Just read the ISS:

ISS
ISS

  • ISV: Indicates that the exception instruction parsing contained in ISS[23:14] is valid.
  • SAS: The size of the memory access when the exception occurs.
  • SSE: Whether the data requires sign extension.
  • SRT: Destination register Rt
  • SF: Whether it is a 32-bit or 64-bit load/store instruction
  • AR: Whether the instruction includes acquire/release barrier semantics
  • VNCR: Whether it is an access from nested virtualization
  • SET: Synchronous error type.
  • FnV: Whether the FAR register is valid.
  • WnR: Exception triggered by a read or write operation.

Flow of simulating reading the serial port MMIO register.

Flow of simulating reading the serial port MMIO register.
Flow of simulating reading the serial port MMIO register.

Flow of simulating writing to the serial port MMIO register.

Flow of simulating writing to the serial port MMIO register.
Flow of simulating writing to the serial port MMIO register.

Experiment 7: Simulating a serial port device in the hypervisor.

Experiment Objective: Simulate a serial port device using full software virtualization.

Experiment Requirements: Implement a serial port driver in the GuestOS and enable the GuestOS to print output via the serial port.

TODO

Timer virtualization.

  • System counter: Provides a system-wide, fixed-frequency system counter.
  • The system counter is broadcast to all cores.
    • The CNTPCT_EL0 register returns the value of the system counter.
    • The CNTFRQ_EL0 register is used to set the frequency corresponding to the system counter.
  • Each CPU core has a set of timers.

System counter and generic timer
System counter and generic timer

generic timer

Each ARMv8/v9 CPU core has a set of generic timers.

generic timer
generic timer

Take Cortex-A72 as an example.

Generic Timer Functional description
Generic Timer Functional description

Difference between physical timer and virtual timer.

The virtual counter canmeasure the passage of time on a virtual machine.。

The passage of time on a virtual machine is not equal to the passage of time on a real physical machine.

vCPU runtime is affected by VM scheduling.
vCPU runtime is affected by VM scheduling.

Assume two vCPUs run alternately, each running for 1ms.

vCPU scheduling
vCPU scheduling

If vCPU0 sets a timer at T=0 to trigger an interrupt after 3ms, has the interrupt been triggered?

Physical time vs virtual timePhysical time refers to wall-clock time, i.e., the actual elapsed time. Virtual time refers to the time experienced by the virtual processor (vCPU).

At physical time 4ms, vCPU0 has only run for 2ms. According to the passage of physical time, the time set by vCPU0’s comparator has not yet reached 3ms, so the interrupt will not be triggered at this point.

  • Virtual counter = physical counter - offset

  • CNTVCT_EL0 (virtual counter) = CNTPCT_EL0 (physical counter) - CNTVOFF_EL2 (virtual offset).

Method 1 for calculating virtual time in a VM.

Method 1 for calculating virtual time in VM
Method 1 for calculating virtual time in VM

Virtual time offset

offset=∑(Time_sched_in−Time_sched_out)\text{offset} = \sum(\text{Time\_sched\_in} - \text{Time\_sched\_out})

Where Time_sched_in is the time when vCPU is scheduled to run, Time_sched_out is the time when vCPU is scheduled to be suspended.

Method 1: The hypervisor updates the CNTVOFF_EL2 register each time the vCPU is scheduled to run.

Method 2 for calculating virtual time in VM

Method 2 for calculating virtual time in VM
Method 2 for calculating virtual time in VM

Method 2:

  • Set the virtual offset to a constant value, virtual counter = physical counter - virtual offset
  • When the vCPU is scheduled out, the hypervisor disables the virtual timer interrupt, meaning the vCPU’s jiffies temporarily stop incrementing.
  • When the vCPU is scheduled to run, the hypervisor enables the virtual timer interrupt, and the vCPU’s jiffies resume incrementing

Two ways for Timer to trigger an interrupt

  • Using the TimeValue method, i.e., assign an initial value to the timer and let it decrement. When it reaches 0, an interrupt is triggered, and the timer is reassigned in the interrupt handler

CNXX_TVAL_Elx register
CNXX_TVAL_Elx register

  • Using CmpValue method: assign a compare value CValue to the timer, and when the timer increments to Cvalue, an interrupt is triggered.

CNXX_CVAL_Elx register
CNXX_CVAL_Elx register

Experiment 8: Enable hypervisor (EL2) physical timer

Experiment objective: Familiarize with the usage of the generic timer in the ARM64 architecture

Experiment requirement: Enable the EL2 physical timer in BenOS

TODO

Experiment 9: Enable virtual timer in VM

Experiment objective: Familiarize with the usage of the generic timer in the ARM64 architecture

Experiment requirement: Enable virtual timer in BenOS. In this experiment, the print in the timer interrupt handler of the Guest OS cannot be seen. Because the virtual timer in the VM is routed to the hypervisor in EL2 for processing, and then injected into the VM. This process requires implementing GIC interrupt virtualization.

TODO

Interrupt virtualization

Routing of interrupt handling

IRQ routing
IRQ routing

For exceptions and interrupts, they can be routed to EL1, EL2, or EL3 for processing, requiring configuration of HCR and SCR related registers.

The HCR_EL2 register is the hypervisor configuration register, which contains the following fields related to exception handling routing

HCR_EL2 register
HCR_EL2 register

Example of interrupt routing
Example of interrupt routing

Virtual interrupt

  • There are two ways to send a virtual interrupt to a vCPU
    • Use system registers (the VI and VF fields in HCR_EL2) to configure vIRQ and vFIQ
    • Use GICv2 and GICv3 controllers
  • Using vIRQ and vFIQ requires emulating the interrupt controller in the hypervisor
    • Low efficiency, trapping and emulating
    • The GIC interrupt controller supports vCPUs without requiring software emulation

GIC interrupt controller
GIC interrupt controller

GICv2 interrupt controller

GICv2 interrupt controller
GICv2 interrupt controller

The Distributor registers (GICD_) Contains interrupt settings and configuration

The CPU Interface registers (GICC_) Contains CPU-specific special settings

GIC interrupt types

  • SGI: Software Generated Interrupt, a software-generated interrupt used to send interrupt signals to other CPU cores.
  • PPI: Private Peripheral Interrupt, a private peripheral interrupt that is unique to a specific CPU.
  • SPI: Shared Peripheral Interrupt, a shared peripheral interrupt that can be accessed by all CPUs.
  • LPI: Locality-specific Peripheral Interrupt, a new interrupt type added in GICv3. It is a message-based interrupt type.

GICv2 Interrupt Controller Interrupt Number Allocation
GICv2 Interrupt Controller Interrupt Number Allocation

Note: Interrupt numbers 1020 to 1023 are reserved.

GICv2
GICv2

GICv2 Registers

GICv2 registers are divided into two groups:

  • D: Registers of the Distributor
  • C: Registers of the CPU interface

Some registers are described by interrupt number, for example, using certain bits to describe the attributes of an interrupt number, and there can be n such registers.

For example,GICD_ISENABLERnThe register is used to enable a specific interrupt number. ‘n’ indicates there are n such registers.

Table 4-1 Distributor register map
Table 4-1 Distributor register map

It can be seen that this register ranges from 0x100 to 0x17c, all of which are this register.

GICD_ISENABLERn
GICD_ISENABLERn

Each bit in the register represents the enable status of an interrupt number.

GICD_ISENABLERn
GICD_ISENABLERn

  • ARM Core Interrupts:
    • Core n HP tiemr IRQ
    • Core n V timer IRQ
    • Legacy FIQn
    • Core n PS timer IRQ
    • Core n PNS timer IRQ (PPI ID 30)
    • Legacy IRQn
  • ARM Local Interrupts
    • ARM Mailbox IRQs
    • Core 0 PMU IRQ
    • Core 1 PMU IRQ
    • Core 2 PMU IRQ
    • Core 3 PMU IRQ
    • AXIERR IRQ
    • Local timer IRQ
  • 16 ARMC peripheral interrupts
  • 64 VC peripheral interrupts
  • 51 PCI-related peripheral interrupts

Access GIC-400 registers

  • Base address of GIC-400 on Raspberry Pi 4B

GIC-400 address
GIC-400 address

GIC-400 register map
GIC-400 register map

GIC-400 memory map
GIC-400 memory map

GIC-400 memory map
GIC-400 memory map

Access: Raspberry Pi GIC-400’s**Base address **+ GIC-400 memory map offset + Register offset

Virtualization support on GIC v2

Virtual CPU interface
Virtual CPU interface

GIC v2 to support virtualization:

  • Addedvirtual CPU interface, sending Vfiq/vIRQ to vCPU.
  • AddedGIC virtual interface control registers。
  • AddGIC virtual CPU interface registers

There is only one Distributor, and the hypervisor needs to simulate a vDistributor for the VM.

Virtual Distributor
Virtual Distributor

GIC v2 Interrupt Virtualization Flow

  1. Ensure all interrupts are routed to the hypervisor at EL2 for handling.。
  2. The hypervisor simulates a vDistributor for the VM, meaning that when the Guest OS accesses the MMIO registers of the GIC distributor, it traps to the hypervisor.。
  3. A mapping is established in the hypervisor, so that when the Guest OS accesses the MMIO region of the GIC CPU interface, it is mapped to the GIC virtual CPU interface.。
  4. When the hypervisor receives a physical interrupt, it must decide whether the interrupt should be handled by the hypervisor or injected into the VM.。
  5. If the interrupt needs to be injected into the VM, update the GICH_LRn register to fill in the interrupt-related information. When the CPU returns to the vCPU, the interrupt injection is complete.。
  6. When the VM is running, reading the GIC_IAR register yields the interrupt number.

An Example
An Example

The peripheral interrupt flow is as follows:

  1. It enters the Distributor hardware unit within the GIC controller, and is then dispatched to a physical CPU interface.
  2. The CPU responds to this interrupt.
  3. This interrupt is routed to EL2. The hypervisor reads the IAR register to obtain the physical interrupt number hwirq. The hypervisor writes to GICC_EOIR to inform the GIC that the physical interrupt handling is complete.
  4. The hypervisor sets the GICH_LRn register to register a virtual interrupt.
  5. After the Virtual CPU interface receives this interrupt, it determines whether this virtual interrupt can be sent to the vCPU.
  6. This interrupt enters the vCPU’s virtual exception vector table. The Guest OS reads the GICC_IAR register, obtains the virtual interrupt number vINTID, and then jumps to the interrupt handler for processing.
  7. After the Guest OS finishes processing the interrupt, it writes to the GICC_EOIR register to inform the GIC that this virtual interrupt has been completed.

Interrupt Injection
Interrupt Injection

The process is as follows:

  1. A peripheral triggers an IRQ interrupt, and the interrupt reaches the GIC.
  2. The GIC sends the interrupt to the CPU. Because HCR_EL2.IMQ is set to 1, the CPU routes it to EL2.
  3. The GIC configures the GIC List Register (GICH_LRn) to register a vIRQ.
  4. The Hypervisor switches to run the Guest OS, transitioning from EL2 to EL1/EL0, and the CPU runs on the vCPU.
  5. The vCPU receives the vIRQ interrupt from the GIC.

Two new sets of registers added for GIC v2 virtualization

Two new sets of registers added for GIC v2 virtualization
Two new sets of registers added for GIC v2 virtualization

MMIO in GIC Virtualization

According to one of the three necessary conditions for virtualization: equivalence. That is, the virtual machine is unaware that it is running in a virtualized environment.

  1. The VM accesses the GIC CPU interface as if it were real. The Hypervisor needs to map the GICC_interface mapped to GICV_on the interface. Base + 0x2000 —> (mapped to) Base + 0x6000

  2. GICC_interface and GICV_the interface is two sets of registers with the same format.

  3. Since there is only one Distributor inside GICv2, the Distributor seen by the VM is emulated by the hypervisor.

vDistributor and Distributor ultimately access the same hardware unit, so access conflicts need to be avoided

MMIO in GIC Virtualization
MMIO in GIC Virtualization

GIC virtual interface control register map
GIC virtual interface control register map

  • GICH_HCRGICH overall control register
  • GICH_ELSR0GICH_LR register status, determining whether the LR register is empty, etc.
  • GICH_LR0used to inject interrupts into the VM

GIC virtual CPU interface register map
GIC virtual CPU interface register map

GICH_LRn register implements interrupt injection

GICH_LRn
GICH_LRn

Writing to the GICH_LRn register can inject interrupts into the VM

Experiment 10: Injecting Interrupts into a VM

Experiment Objective: Familiarize with how GIC v2 injects interrupts

Experiment Requirements: Implement interrupt injection in BenOS, injecting the virtual timer interrupt into the VM. Steps are as follows:

  • Enable GIC virtualization, GICH_HCR.EN =1
  • Map the GICC in the VM_Interface access address to the hypervisor’s GICV_interface
  • Simulate the Distributor in the hypervisor
  • When the virtual timer in the VM triggers an interrupt, route it to the hypervisor at EL2 for handling
  • The hypervisor writes the interrupt information into the GICH_LRn register and injects the interrupt into the VM
  • EOI mode, handled by Hypervisor or inject vm
  • The GIC driver of the GuestOS in the VM reads the interrupt number and handles the interrupt

Points to note when simulating vDistributor

  • The Distributor seen by the VM is simulated by the hypervisor.
  • When simulating vDistributor, the hypervisor must consider concurrent access between the VM and the hypervisor.
  • Some registers of the Distributor are described by interrupt number, for example, using certain bits to describe the attributes of an interrupt number, so the same register can describe n interrupt sources

GICR_IPRIORITY
GICR_IPRIORITY

If vDistributor rashly writes directly to such registers via trap and emulate, it will corrupt the hypervisor’s original state

Methods to avoid conflicts:

GICD_IPRIORITY
GICD_IPRIORITY

Given reg_base，far_addr，field_width, far can be calculated_The starting start_hwirq number corresponding to addr.

1
2
3
4
for (start_hwirq ~ start_hwirq +4)
	if (irq owned by hypervisor)
		continue；
	update_reg_field();

Virtualization Host Extensions

TODO