Timeline
Timeline
2026-07-18
init
This article describes how to compile NVIDIA's open-gpu-kernel-modules, focusing on the steps and key parameters for cross-compiling using the ARM GNU toolchain, including the distinction between ARCH=arm64 and TARGET_ARCH=aarch64, toolchain path settings, and cleanup methods. It also summarizes the repository's two-layer build model: the OS-independent core (src) and the Linux kernel adaptation layer (kernel-open) are assembled together to ultimately generate 5 kernel modules, and it outlines the functions of each module, code size distribution, and the central role of GSP firmware in architectural evolution.
Source code
1 | git clone https://github.com/NVIDIA/open-gpu-kernel-modules.git |
Cross-compilation toolchain
This article uses Arm GNU Toolchain 11.2-2022.02（aarch64-none-linux-gnu-triplet).
ARM has released 8 architectures in total: ARMv1, ARMv2, ARMv3, ARMv4, ARMv5, ARMv6, ARMv7, ARMv8.
For processors supporting the ARMv8 instruction set, you can use-march=armv8-aCompile code with the appropriate parameters; the ARM GNU compiler can be downloaded from the link below.
It is recommended to use the compiler from ARM’s official website. It is portable and can be used after extraction.
This article uses the 11.2 version of the cross-compiler.
You can add it to the environment variables.
123 | # ~/.bashrc or ~/.bash_profileemacs ~/.bashrcexport PATH="$HOME/tools/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH" |
Using version 11.2.1 ofaarch64-none-linux-gnu-gcc
If the toolchain triplet is
aarch64-linux-gnu-(e.g., Ubuntu’sgcc-aarch64-linux-gnupackage), replace all the followingaarch64-none-linux-gnu-withaarch64-linux-gnu-That’s all.
Compilation
123456789101112131415 | make -j$(nproc) modules \ ARCH=arm64 \ CROSS_COMPILE=aarch64-none-linux-gnu- \ TARGET_ARCH=aarch64 \ TARGET_OS=Linux \ CC=aarch64-none-linux-gnu-gcc \ CXX=aarch64-none-linux-gnu-g++ \ LD=aarch64-none-linux-gnu-ld \ AR=aarch64-none-linux-gnu-ar \ STRIP=aarch64-none-linux-gnu-strip \ OBJCOPY=aarch64-none-linux-gnu-objcopy \ HOSTCC=gcc \ SYSSRC="$HOME/repository/linux/linux-5.10.238" \ SYSOUT="$HOME/repository/linux/linux-5.10.238" |
Key points:
- ARCH=arm64 is for Kbuild, TARGET_ARCH=aarch64 is for NVIDIA’s utils.mk (the two cannot be mixed).
- The toolchain is specified via PATH plus explicit CC/CXX/LD/AR as a double guarantee, because utils.mk does not read CROSS_COMPILE.
- SYSSRC/SYSOUT point to linux-5.10.238 already configured with the same toolchain.
To clean:
1234 | make clean \ ARCH=arm64 \ SYSSRC="$HOME/repository/linux/linux-5.10.238" \ SYSOUT="$HOME/repository/linux/linux-5.10.238" |
12345678 | make -j$(nproc) modules \ ARCH=x86_64 \ TARGET_ARCH=x86_64 \ TARGET_OS=Linux \ CC=gcc CXX=g++ LD=ld AR=ar STRIP=strip OBJCOPY=objcopy \ HOSTCC=gcc \ SYSSRC="$HOME/repository/linux/linux-5.10.238-x86_64" \ SYSOUT="$HOME/repository/linux/linux-5.10.238-x86_64" |
Description
- Native compilation does not require CROSS_COMPILE; just use the local gcc. Still need to explicitly pass CC/CXX/LD/AR, because utils.mk does not read CROSS_COMPILE。
- The difference from arm64 is only in ARCH/TARGET_ARCH: for arm64 it is ARCH=arm64 + TARGET_ARCH=aarch64；x86_when 64, both are x86_64. Note that utils.mk has no TARGET_ARCH called arm64, only aarch64.
Cleanup
1234 | make clean \ ARCH=x86_64 \ SYSSRC="$HOME/repository/linux/linux-5.10.238-x86_64" \ SYSOUT="$HOME/repository/linux/linux-5.10.238-x86_64" |
Directory structure
1. Top-level directory structure
12345678910 | open-gpu-kernel-modules/├── Makefile # Top-level entry: make modules → first compile src/ then compile kernel-open/├── utils.mk # NVIDIA's own build framework (CC/CXX/ARCH/object rules, etc.)├── version.mk # Version definitions (e.g., NVIDIA_VERSION = 610.43.03)├── nv-compiler.sh # Script to detect compiler type (gcc/clang)├── src/ # ① OS-independent core driver source (freestanding, compiled to .o)├── kernel-open/ # ② Linux kernel adaptation layer (Kbuild, compiled to .ko)├── nouveau/ # Script to extract GPU firmware from nouveau (not the driver itself)├── README.md / COPYING / SECURITY.md / CODE_OF_CONDUCT.md / CONTRIBUTING.md└── .github/ # CI automation configuration |
Core mechanism: two-layer build model
The key to understanding this repository lies in its two-layer build model. This repository is not a single Linux kernel module, but rather a combination of the NVIDIA OS-independent core and the Linux kernel adaptation layer, ultimately producing 5 .ko modules.
First-layer artifacts: produced in
src/nvidia/_out/Linux_<ARCH>/nv-kernel.o. The code uses-ffreestandingand-fno-stack-protectorcompiled, allowing the core logic to be reused across different operating systems (such as Linux, FreeBSD, etc.).Second layer: The .o files generated in the first layer are linked as “black-box binaries” into the final .ko modules, wrapped with Linux-specific hook code.
2. src: OS-independent core
(approximately 2.11 million lines)
123456789101112131415161718192021222324252627 | src/├── nvidia/ # ★ RM (Resource Manager) core, 1.5 million lines, main body of the repository│ ├── src/kernel/ # Kernel-mode driver implementation│ │ ├── gpu/ # GPU IP submodules (details below)│ │ ├── core/ # Core framework (NVOC object system, threads, scheduling)│ │ ├── rmapi/ # RM API (user-mode ioctl entry)│ │ ├── mem_mgr/ # Memory management│ │ └── os/ # OS abstraction layer interface (implemented by kernel-open)│ ├── inc/ # Header files (mirroring the src/ structure, plus libraries/ containers/ mmu/ nvlog/ ...)│ ├── interface/ # External interface (rmapi, deprecated)│ ├── arch/nvalloc/ # CPU architecture-related code (common / unix / x86emu)│ ├── generated/ # Auto-generated code (NVOC metadata, etc.)│ └── _out/ # Compilation output directory (Linux_aarch64, Linux_x86_64）│├── nvidia-modeset/ # Display modeset logic, 100,000 lines│ └── src/ interface/ os-interface/ kapi/ lib/ include/│└── common/ # Cross-module shared code, 490,000 lines ├── sdk/nvidia/inc/ # Public SDK header files ├── nvswitch/ # NVSwitch chip driver (lr10/ls10/soe/flcn/...) ├── nvlink/ # NVLink high-speed interconnect ├── displayport/ # DisplayPort protocol implementation ├── modeset/ # timing / hdmipacket ├── softfloat/ # Soft float implementation (Berkeley softfloat, 8086-SSE backend) ├── unix/ # nvidia-push / nvidia-3d / xzminidec / headsurface ├── uproc/os/ # Microprocessor OS abstraction (libos-v3.1.0) └── shared/ inc/ src/ # Common utilities and status codes |
3.src/nvidia/src/kernel/gpu/Subdirectories (GPU IP engines)
This is the core hardware control part of the driver, divided by hardware engines:
- gr/: Graphics (rasterization/shading/rendering core)
- fifo/: Command submission queue management
- ce/: Copy Engine (high-performance memory copy engine)
- intr/: Interrupt handling
- sec2/: Security engine
- falcon/: Falcon microcontroller core framework
- gsp/: ★ GSP (GPU System Processor) core
- fsp/：Firmware Security Processor
- gsplite/: Lightweight GSP logic
- pmu/: Performance monitoring unit
- nvdec/ nvenc/ nvjpg/ ofa/: Video encoding/decoding and optical flow acceleration engines
- bif/ bus/: PCIe bus interface and bus control
- mem_mgr/ mem_sys/ mmu/ mig_mgr/: Video memory management, MMU page tables, MIG instance partitioning
- disp/: Display output control
- perf/ hwpm/ rc/ timer/ …: Performance, hardware monitoring, reset recovery, and timers
Architecture evolution background: The RM (Resource Manager) core of modern NVIDIA GPUs mainly runs on the GSP (a RISC-V microprocessor) inside the GPU. A large part of the code in src/nvidia is actually the source code of the GSP firmware itself—this is also the key underlying evolution of NVIDIA gradually moving toward open source after 2020.
4. kernel-open/ Linux adaptation layer
(about 220,000 lines)
1234567891011121314 | kernel-open/├── Kbuild # Module assembly entry: traverses and imports *.Kbuild of all submodules├── Makefile # Called by Kbuild, responsible for locating the Linux kernel source tree (SYSSRC/SYSOUT)├── conftest.sh # ★ Core mechanism: dynamically probe kernel APIs at compile time├── conftest/ # conftest test case list│ ├── compile-tests/ # C source code for "whether the kernel has symbol X / struct member Y"│ └── header_presence/# Detection of "whether a certain header file exists"├── common/inc/ # Adaptation layer public header files│├── nvidia/ # → nvidia.ko Main driver Linux hooks (nv.o, nv-pci.o, nv-acpi.o...)├── nvidia-modeset/ # → nvidia-modeset.ko Display modeset hooks├── nvidia-drm/ # → nvidia-drm.ko DRM/KMS standard graphics framework adaptation├── nvidia-uvm/ # → nvidia-uvm.ko ★ Unified virtual memory, 150,000 lines (including hardware references by generation: ampere/turing/hopper/blackwell/rubin)└── nvidia-peermem/ # → nvidia-peermem.ko InfiniBand/RDMA peer-to-peer memory support |
conftest.shMechanism: This is the core secret of how the NVIDIA driver can be “one source code compatible with countless kernel versions”. Before compilation, it performs a large number of snippet compilation tests against the target kernel (e.g., detecting whether kmem_cache_create_usercopy、pgprot_t, etc. The test results are written into conftest/headers.h, and then in the source code through#if NV_IS_ENABLED(...)macros to conditionally compile the corresponding compatibility code.
Final generated kernel modules
| Module name. | Main purpose | Main source code structure |
|---|---|---|
nvidia.ko | Main driver core: create/dev/nvidia*Device node, providing the core ioctl service entry. | kernel-open/nvidia/+src/nvidia/Compilednv-kernel.o |
nvidia-modeset.ko | Display mode settings: Controls resolution, refresh rate, multi-screen display tiling, etc. | kernel-open/nvidia-modeset/+src/nvidia-modeset/Compiled core.o |
nvidia-drm.ko | DRM/KMS integration: Direct Rendering Manager adaptation required by modern Linux graphics stacks (such as Wayland). | kernel-open/nvidia-drm/ |
nvidia-uvm.ko | Unified virtual memory management: Implements unified CPU/GPU memory addressing. Core of HPC and AI computing, with a huge code size. | kernel-open/nvidia-uvm/(approximately 150,000 lines of independent implementation) |
nvidia-peermem.ko | RDMA P2P support: Allows third-party PCIe devices (e.g., Mellanox NICs) to directly read and write VRAM via GPUDirect. | kernel-open/nvidia-peermem/ |
Code size distribution
1234567 | src/nvidia 150 万行 ████████████████████ ← RM/GSP 核心，绝对主体src/common 49 万行 ██████ ← 共享协议与芯片支持kernel-open/nvidia-uvm 15 万行 ██ ← UVM 独立计算大头src/nvidia-modeset 10 万行 █ ← 显示逻辑kernel-open/nvidia 5 万行 █ ← Linux 主驱动钩子kernel-open/nvidia-drm 1.7 万行 ← DRM 适配层其他 <1 万行 ← 构建脚本与配置 |