Principles and Implementation of SR-IOV

even629

2025-05-22 (Updated: 2025-05-22)

Timeline

2025-05-22

init

This article introduces various implementation methods of I/O virtualization, discusses in detail the principles and performance bottlenecks of pure software sharing and direct assignment, and on this basis introduces the concepts of Single Root I/O Virtualization (SR-IOV) and natively shareable devices, aiming to solve the scalability limitations of device passthrough and enable multiple virtual machines to efficiently share underlying I/O hardware resources without going through the VMM.

I/O Virtualization (IOV)

**I/O Virtualization (IOV)**refers to sharing a single I/O resource among multiple virtual machines. Methods to implement IOV include:

pure software implementationsharing method,
hardware-supportedsharing method,
anda hybrid approach combining software and hardware。

device emulation (full virtualization)：
Device emulation mode mimics widely supported real hardware devices (e.g., Intel’s 1Gb network card), allowing the operating system in the virtual machine to continue using its originally supported drivers. The Virtual Machine Monitor (VMM) emulates this I/O device to ensure compatibility, processes these operations before actual I/O, and then forwards them to the actual physical device (which may differ from the emulated device).
⚠️ Problem: As a result, I/O operations must go through two layers of I/O stack—one inside the virtual machine and another in the hypervisor—which increases overhead and reduces performance.
the split-driver model (paravirtualization)：
This method is similar to device emulation, but itno longer emulates a traditional device. Instead, it adopts afront-end and back-end driver cooperation approach：
- The front-end driver runs in the guest operating system of the virtual machine;
- The back-end driver runs in the hypervisor.
  The two work together, specifically optimized for resource sharing. Compared to device emulation, the advantages of this method are:No need to emulate the entire device, resulting in better performance and lower overhead. The back-end driver communicates directly with the actual physical device.

Method	Advantages	Disadvantages
device emulation	Good compatibility, can use existing drivers	Poor performance, must go through two layers of I/O stack
the split-driver model	Better performance, optimized sharing	Requires writing dedicated drivers for front-end and back-end

Disadvantages of software-based sharing

Whether it isdevice emulationorsplit drivers (i.e., paravirtualized drivers), they usually only provide a subset of physical hardware functions, so they maybe unable to utilize advanced features provided by physical devices。

Additionally, when the virtual machine monitor (VMM) implements a virtual software switch (used to forward packets between multiple virtual machines), it may consumea large amount of CPU resources. This CPU overhead can (and often does)reduce the maximum throughput of I/O devices。

For example, if only device emulation is used, a 10Gbps Ethernet controllercan achieve a maximum throughput of only 4.5 to 6.5 Gbps(this range depends on the architecture of the specific test server).
One of the main reasons is:**each packet must pass through a software switch,**which requires the CPU to process these packets, creating a performance bottleneck and preventing the achievement of line-rate or near-line-rate transmission capability.

Although software-based I/O virtualization offers good compatibility, it also has some obviousperformance bottlenecks and functional limitations, which are particularly evident in scenarios with high performance requirements (such as high-frequency trading, data centers, network function virtualization, etc.):

Disadvantages	Description
Incomplete functionality	Inability to access all advanced features of the device, such as hardware acceleration, QoS, SR-IOV, etc.
High CPU overhead	Each I/O operation requires VMM intervention, especially in network scenarios where packet processing goes through a software switch, consuming a large amount of CPU.
Throughput is limited	A theoretical 10Gbps network card actually only achieves a throughput of 4.5~6.5Gbps, far below line rate.

Direct Assignment

also known asdevice passthrough

Software-based sharing adds extra overhead to each I/O operation because there is aemulation layerbetween the guest driver and I/O hardware. The presence of this intermediate layer also brings another impact:the inability to use hardware acceleration features provided by the physical device。
To address these issues, physical hardware can bedirectly exposed to the guest operating system (Guest OS), and let it runnative device drivers, thereby reducing the performance loss caused by the intermediate layer.

Hardware vendors (such as Intel) have introduced enhancements for this purpose, used tosupports memory address translationandensures memory protection, enabling devices todirectly perform DMA operations on host memory. These enhancements canbypass the I/O emulation layer of the virtual machine monitor (VMM), therebyimproving the I/O throughput performance of virtual machines。

A feature of Intel® VT-x technology is that, if configured by the VMM, a virtual machine candirectly access physical addresses. In this way, device drivers in the virtual machine candirectly write to the registers of I/O devices(e.g., configuring DMA descriptors, etc.).

Intel® VT-d technology provides similar capabilities, allowing I/O devices toDirectly write to the virtual machine’s memory space, such as performing DMA operations.

The mechanism for implementing direct assignmentvaries among different vendors, but the basic idea is the same: the hypervisor utilizes and configures technologies such as Intel® VT-x and VT-d to perform address translation during data transmission, ensuring that data can be transferred safely and efficiently between the virtual machine and I/O devices.

✅ Advantages:

Bypass the intermediate emulation layer, reducing CPU overhead;
Support native drivers, enabling all device features (including hardware acceleration);
Significantly improve performance, especially noticeable improvements in throughput and latency;
Suitable for high-performance scenarios, such as Network Functions Virtualization (NFV), GPU-accelerated computing, etc.

⚠️ Issues:

A major problem with direct assignment is itslimited scalability: a physical devicecan only be assigned to one virtual machine。

For example, a dual-port network interface card (NIC) can only be provided to two virtual machines simultaneously (each VM occupies one port). However, there is afundamental limitation。

Imagine a relatively powerful server in the near future:
It might have 4 physical CPUs, each with 12 cores, totaling 48 cores. Following the rule of thumb of ‘one core per virtual machine,’ this server could run48 virtual machines. If you wish to assign devices to each virtual machine using Direct Assignment, you wouldneed 48 physical ports。

Single Root I/O Virtualization (SR-IOV)

The root cause of the above architectural issue is that the hardware does not natively support sharing at the underlying level, requiring a new type ofnatively shareable device（Navitely Shared Devices), these devices need to replicate necessary resources for each virtual machine, so thatA virtual machine can directly connect to I/O devices, andwithout VMM involvementcan complete the main data transfer.

Natively shareable devicestypically provide each virtual interface withindependent memory space, work queues, interrupts, and command processing mechanisms, while sharing somecommon resourcesbehind the host interface. These shared resources still require management, and typically a set of management registers is exposed to theTrusted Partitionin the VMM, providing each virtual machine with independent work queues and command processing capabilities. Such devices cansimultaneously receive instructions from multiple sources, and consolidate them before sending to the secondary interconnect (such as Ethernet or SAS links), eliminating the need for virtualization software to serialize multiple I/O requests.

These natively shareable devices can be implemented in various ways, both standardized and proprietary. Since most such devices are accessed via the PCI interface, PCI-SIG (PCI Special Interest Group) decided to develop a standard mechanism to support this functionality.

This standard is:PCI-SIG Single Root I/O Virtualization and Sharing Specification (SR-IOV)

SR-IOV defines a standard mechanism for devices to achieve native sharing.

PCI-SIG SR-IOV Specification

The goal of the PCI-SIG SR-IOV specification can be summarized in one sentence:Standardize a way to bypass the hypervisor (VMM) involvement in data transfer, by providing each virtual machine withindependent memory space, interrupt channels, and DMA channels, to achieve efficient I/O virtualization.

The architectural design of SR-IOV allows a single device to support multipleVirtual Functions (VFs), with a focus on minimizing the hardware cost of each additional function.

SR-IOV introduces two new types of PCIe functions:

Physical Functions (PFs)：
Full PCIe functions, includingSR-IOV Extended Capability. This capability is used to configure and manage SR-IOV related functions.
Virtual Functions (VF)：
Lightweight PCIe functions that contain the resources required for data transfer, but theirconfiguration resources are streamlined, to reduce hardware overhead.

SR-IOVProvides a mechanism that allows a single physical function (e.g., an Ethernet port)to appear as multiple independent “physical devices”。

An SR-IOV-capable device can be configured (typically by the VMM) toPCI configuration spaceappear asmultiple functions, each with its ownconfiguration space, including its ownBase Address Registers (BARs). SR-IOV devices can be configured to supportmultiple independent Virtual Functions (VFs), each with its own independent PCI configuration space. The VMM assigns these VFs to different virtual machines.

The Virtual Machine Monitor (VMM) assigns one or more Virtual Functions (VFs) to a virtual machine, a process that involves mapping the VF’s real configuration spaceto the PCI configuration space visible to that VM. Meanwhile, technologies such as Intel® VT-x and VT-dmemory address translation technologiesprovidehardware acceleration mechanisms, enabling DMA data to be transferred directly between the VM and the device, therebybypassing the software switching overhead in the VMM。

Ecosystem dependency

BIOS

The BIOS is responsible for allocatingMemory-Mapped I/O (MMIO) spaceandPCI Express Bus Numbersto host bridges.
In many systems, PCI resource allocation lacks standardization, and software relies on the BIOS to configure devices, ensuring sufficient memory space and bus range to support all I/O devices under the host bridge.
The BIOS enumeration code needs to be enhanced to recognizeSR-IOV devices, and allocate sufficient MMIO space to all Virtual Functions (VFs).
For details on how to parse the PCI configuration space and calculate the maximum MMIO space required for VFs, refer to the PCI-SIG SR-IOV specification.

Virtual Machine Monitor (VMM)

SR-IOV defines two function types:
- PF: A full PCIe function with SR-IOV extended capabilities, used to manage and configure virtual functions.
- VFs: A lightweight PCIe function containing minimal configuration resources solely for data transfer.
SR-IOV introduces a new software entity:Single Root PCI Configuration Manager (SR-PCIM or PCIM)。
- The PCIM is responsible for managing and configuring VFs, handling all access requests to the configuration space.
- It presents a complete PCI configuration model to the guest operating system in the virtual machine based on information from the physical function.
- PCIM is a conceptual model, with specific implementations integrated by various VMM vendors.
The Linux kernel has supported SR-IOV since version 2.6.30 (June 2009), and multiple distributions have integrated it.

Virtual Function (VF) Creation

After the device is powered on, VFs do not exist by default, and their configuration space cannot be accessed.
VFs must be configured and enabled through the SR-IOV capability on the physical function before they can be accessed.
The PCI configuration space of the physical function contains an SR-IOV capability structure, which includes aSystem Page Size field, which is set by the VMM to the size supported by the platform.
The memory space of all VFs is contiguously mapped within the memory range specified by the VF Base Address Register.
To ensure memory space isolation, the memory resources of VFs must be aligned to the page protection boundaries provided by the system.
The ARI (Alternative Routing-ID Interpretation) capability flag affects the distribution of the maximum number of VFs. The VMM should enable ARI on root ports and switches and set the ARI Capability Hierarchy bit in the SR-IOV capability. The BIOS may have pre-enabled ARI.

VF Discovery

After setting the VF Enable field, VFs are created and respond to configuration transactions.
Legacy enumeration software will not automatically discover these VFs.
SR-IOV introduces a new mechanism: through theFirst VF OffsetandVF Stridefields forming a linked list, software can locate all VFs associated with this physical function.
SR-IOV devices may require software to allocate multiple PCI bus numbers to support more than 256 functions.

VF driver communication support with the physical function driver

The VMM can support creating shared memory pages to facilitate communication between the VF driver in the virtual machine and the master driver (details in Section 6).

VF assigned to virtual machine

After VFs are created and configured, they can be assigned to virtual machines, enabling direct I/O interaction between the virtual machine and the hardware device.
SR-IOV design assumes that all VFs on the device are identical, presenting the same functionality in PCI configuration.
However, through master driver intervention in allocation, the hardware can provide different functional features or performance levels based on system administrator requirements (e.g., offering a 2Gbps Ethernet performance guarantee for certain VFs).

Summary

The BIOS needs to support the identification and resource allocation of SR-IOV devices, especially MMIO and bus number management.
The VMM introduces the PCIM model to manage VFs, responsible for transparent configuration access and virtualized resource management.
VFs need to be explicitly created and enabled, supporting large-scale virtualization scenarios (hundreds or thousands of VFs).
SR-IOV supports high-performance, low-latency I/O passthrough while offering flexible resource allocation and management capabilities.

Master Driver (MD)

The Master Driver (MD), also known as the Physical Function Driver (PFD or PF Driver), is a specialized driver that manages the global functions of an SR-IOV device and is responsible for configuring shared resources. The MD is specific to the Virtual Machine Monitor (VMM) and runs in a higher-privileged environment than typical virtual machine drivers. It includes all the functionality of a traditional driver to allow the VMM to access I/O resources, and it can also perform operations that affect the entire device.

The MD must run in a persistent environment, be loaded before any virtual machine driver, and be unloaded only after all virtual machine drivers have been unloaded. All operations that affect the entire device can only be received and processed by the MD.

To achieve this control capability, the VF driver in the virtual machine needs to communicate with the MD. For example, changes in the link state of an Ethernet device or changes in the Maximum Transmission Unit (MTU) require this communication mechanism. When the VF driver queries the link state through the communication channel, the MD can return any state. When the MTU of the physical device changes, the MD can notify all VF drivers so that the network stack can make corresponding adjustments.

VF Drivers

A VF is a “lightweight” PCIe function that contains only the resources necessary for data transfer. It is not a complete PCIe device; it only provides a mechanism for data ingress and egress.

The VF driver runs in the virtual machine and should be a paravirtualized driver (i.e., aware that it is in a virtualized environment), capable of only performing its allowed operations. Typically, a VF has the ability to send/receive data and perform its own reset. This reset operation only affects that VF and does not impact the entire physical device. Operations beyond its authority must be achieved through communication with the MD.

The VF driver is a specialized driver that “knows” it can only perform specific functions, such as configuring DMA descriptors, setting MAC addresses, VLAN tags, etc. Each VF has independent resources allocated in the I/O device. An Ethernet VF typically has independent transmit/receive queues, bound to the corresponding Base Address Register (BAR) and descriptors.

Communication between the VF Driver and the Master Driver

The key to device sharing is that the VF driver can communicate with the MD to request operations that affect the global state. This communication channel needs to pass messages and be able to generate interrupts.

SR-IOV does not define this communication mechanism, so it must be built jointly by the master driver, physical function driver, and VMM designers. The simplest approach is to use a VF-specific in-device mailbox and doorbell mechanism (supported by Intel SR-IOV network controllers). Currently, various VMM vendors are implementing their own communication mechanisms, and no unified standard has been established.

Workflow Example

The following is a typical workflow for receiving an Ethernet packet and forwarding it to a virtual machine via a VF:

The packet arrives at the Intel® network card.
The packet is sent to a Layer 2 sorter configured by the master driver.
After classification, the packet enters the receive queue of the target VF.
Initiate a DMA operation, whose target address is determined by the descriptor configured by the VF driver.
The DMA reaches the chipset. The Intel® VT-d configured by the VMM maps the virtual address to the host physical address, completing the DMA.
The NIC issues an interrupt, which is captured by the VMM.
The VMM sends a virtual interrupt to notify the VM of data arrival.

Summary

The platform must support address translation (e.g., Intel® VT-d) to allow devices to DMA data directly into the VM’s memory.
The BIOS and VMM must be able to parse the PCI configuration space, especially the VF location mechanism of SR-IOV.
The VMM, PF driver, and VF driver must work together to provide the VM with a complete PCI space mapping and VF operation capability.

PCIe

PCI（Peripheral Component Interconnect）

PCI was once the most widely used interface in personal computers, with almost all motherboard products featuring this slot. This bus has now been gradually replaced by the PCI Express bus. The PCI bus evolved from the ISA (Industry Standard Architecture) bus and is a synchronous, processor-independent 32-bit or 64-bit local bus. Structurally, PCI is a bus level inserted between the CPU vendor and the original system bus, managed by a bridge circuit that implements the interface between the upper and lower levels to coordinate data transfer.

**Note:**The ISA parallel bus has 8-bit and 16-bit modes, with a clock frequency of 8MHz and an operating frequency of 33MHz/66MHz.

The PCI bus is a tree structure and is independent of the CPU bus, allowing parallel operation with the CPU bus. PCI devices and PCI bridges can be attached to the PCI bus. Only one PCI master device is allowed on the PCI bus at any given time; all others are PCI slave devices. Read and write operations can only occur between master and slave devices, and data exchange between slave devices must be relayed through the master device.

**Note:**This does not mean that all read and write operations must be relayed through the northbridge, because the master and slave device attributes on the PCI bus can change. For example, when Ethernet and SCSI need to transfer data, it can be done through a method called Peer-to-Peer, where Ethernet or SCSI acts as the master, and other devices act as slaves.

A typical 33MHz PCI bus system is shown in the figure above. The processor connects to the northbridge via theFSB（Front Side Busfront-side bus (FSB), which in early computer architectures is the main bus forcommunication between the processor (CPU) and the northbridge chip, used to connect the CPU to core system components such as the memory controller, graphics card, and PCI bus). The northbridge hosts the graphics accelerator (GPU), SDRAM (memory), and the PCI bus. The PCI bus hosts the southbridge, Ethernet, SCSI bus (an older minicomputer bus), and several PCI slots. CD and hard drives connect to the southbridge via IDE, while audio devices, printers, mice, and keyboards also connect to the southbridge, which additionally provides several USB ports.

The PCI bus is a shared bus, so it requires a specificarbiter（Arbiter) to determine control of the bus at any given time. Typically, this arbiter is located in the northbridge, and the arbiter (host) connects to each slave via a pair of pins,REQ#（request) andGNT# （grant) to connect to each slave, as shown in the figure below:

Note that not all devices are capable of acting as an arbiter or initiator.