ARM Atomic Operation

Timeline

Timeline

2025-10-30

init

This article introduces the atomic operation mechanisms under the ARM architecture, focusing on the two atomic operation implementations provided by ARMv8: the traditional Load-exclusive/Store-exclusive (LDXR/STXR) instructions and the Large System Extensions (LSE) atomic instructions introduced in ARMv8.1. The article first explains the necessity of atomic operations and how the atomic_t type in the Linux kernel relies on the architecture to implement the atomicity of 'read-modify-write'. It then elaborates on the working principles of the LDXR/STXR instructions, including the open/exclusive state transitions of the Exclusive Monitor, the composition of multi-level exclusive monitors, the exclusive reservation granularity (ERG), and usage restrictions. Through a timing case of multi-core lock contention, it analyzes the behavior of LDXR/STXR in concurrent scenarios, and introduces the application of the WFE instruction in low-power waiting for spinlocks. In addition, the article compares the difference between unordered atomic increment and ordered lock implementation (LDAXR/STLXR), and summarizes the CAS and atomic memory newly added in ARMv8.1 LSE.

Reference documents:

Arm Architecture Reference Manual Armv8

https://developer.arm.com/documentation/ddi0487/fc/

Content related to the ARMv8.6 chip manual and exclusive memory access

• Chapter B2.9 Synchronization and semaphores
• Section D1.16 Mechanisms for entering a low-power state
• Chapter C3.2.13 Compare and Swap
• Chapter C3.2.13 Atomic memory operations
• Chapter C3.2.14 Swap

Why are atomic operations needed?

thread_A_func and thread_B_func all attempt to perform the i++ operation

Why are atomic operations needed?

Basic atomic operation functions in the Linux kernel

• The Linux kernel provides atomic variables of type atomic_t, whose implementation depends on different architectures.
• The atomic operation functions of the atomic_t type can guarantee the atomicity and integrity of operations.
• “Read-Modify-Write” mechanism
  • First read the value of the atomic variable into a general-purpose register.
  • Modify the value of an atomic variable in a general-purpose register.
  • Write the new value back to memory

atomic_t

atomic.h

atomic_read and atomic_set

If the CPU merely loads a variable’s value from memory, or merely stores a variable’s value to memory, both operations are uninterruptible.

The following operation function APIs use the “read-modify-write” mechanism.

read-modify-write

Atomic Operation Functions

Atomic Operation Functions

ARMv8 support for atomic operations

• ARMv8 provides two forms of atomic operations.
  • Traditional Load-exclusive and Store-exclusive methods.
    • Both are supported on ARMv8.
    • LL/SC(Load-link/store-conditional)
  • LSE (Large System Extensions) supports atomic operation instructions.
    • Supported starting from ARMv8.1, ARMv8.1-LSE.
    • Adds Compare and Swap instructions.
    • Adds Atomic memory operation instructions.
    • Adds Swap instruction.

LSE

Load-exclusive and Store-exclusive instructions.

• ldxrInstruction: Load-exclusive instruction. Loads the value at a memory address into a general-purpose register in an exclusive manner.

ldxr <xt> , [xn|sp]

• stxrInstruction: Store-exclusive instruction. Stores new data to memory in an exclusive manner.

stxr <ws> , <xt> , [xn|sp]

• Load/Store Exclusive Pair instructions.

ldxp <Xt1>, <Xt2>, [Xn|SP]
stxp <Ws>, <Xt1>, <Xt2>, [<Xn|SP>]

• Load/Store Exclusive instructions with acquire and release primitives.

Example

• An exclusive monitor is used to monitor access to this memory. The exclusive monitor marks the memory address as being in exclusive access mode, ensuring that the memory address is accessed exclusively and is not affected by other factors.

Example of exclusive access.

Exclusive Monitor.

• The exclusive monitor has two states:
  • Open Access state.
  • Exclusive Access state.
• When the LDXR instruction loads data from memory, the CPU marks the memory address as being in exclusive access state.
• When the CPU executes the STXR instruction, it decides based on the state of the exclusive monitor.
  • If the exclusive monitor is in exclusive access state, the STXR instruction stores successfully, returns 0, and the exclusive monitor changes to open access state.
  • If the exclusive monitor is in open access state, the STXR store fails and STXR returns 1.

Exclusive Monitor.

Notes.

• The exclusive monitor itself is not used to prevent CPU cores from accessing the marked memory, and it does not lock the bus.
• The exclusive monitor only serves a monitoring role, monitoring changes in state.
• The exclusive monitor cannot be regarded as a hardware lock.

Exclusive monitor state machine.

Architecture of the exclusive monitor.

• A system typically consists of multiple levels of exclusive monitors (defined at chip design time).
  • Local exclusive monitor (Local monitor),Applicable to non-shareable memory.
  • Cache-coherent global exclusive monitor (Internal coherent global monitor),Applicable to normal memory.
  • External global exclusive monitor (External global monitor),Applicable to device memory.
• Some SoCs do not support an external global exclusive monitor. For example, the BCM2711 used on the Raspberry Pi 4B.
• When the MMU is not enabled, accessing physical memory becomes accessing device-type memory. In this case, using the ldxr and stxr instructions can produce unpredictable errors.

Architecture of the exclusive monitor.

• The use of the ldxr instruction has many restrictions; it requires the memory to be normal memory and shareable.
• If accessing device memory, for example when the MMU is not enabled, then the CPU IP core must support exclusive access to device memory. This requires consulting the specific CPU IP manual.

Granularity of the exclusive monitor (Granularity of Exclusive Monitor)

• The ERG (Exclusives Reservation Granule) in the CTR_EL0 register defines the minimum unit of the exclusive monitor.
• The ERG can define a range from 4 words to 512 words, but it is usually the size of a cache line.
• Example:
  • Assume ERG is 2^4, i.e., 16 bytes. When using the ldxrb instruction to perform an exclusive read operation on address 0x341B4, then addresses from 0x341b0 to 0x341bf will be marked as exclusive access.

Case 1: Implementation of the atomic_add() function.

Case 1: atomic_add

Case 2: Implementation of a simple lock (spinlock).

Case 2 Simple Lock Implementation

cbnz w2, retry

Analysis of ldxr and stxr in the multi-core case

CPU0 and CPU1 simultaneously execute the get_lock() operation

Analysis of ldxr and stxr in the multi-core case

Time T0 Initial state

Time T0: Initial state

Time T1 and T2 CPU0 executes the ldxr instruction

Time T1 and T2: CPU0 executes the ldxr instruction

Time T3 CPU1 executes the ldxr instruction

Time T3: CPU1 executes the ldxr instruction

Time T4 CPU0 acquires the lock using the stxr instruction

Time T4: CPU0 acquires the lock using the stxr instruction

Time T5 CPU1 attempts to acquire the lock using the stxr instruction

Time T5: CPU1 attempts to acquire the lock using the stxr instruction

Application of the WFE instruction in lock implementation

• If CPU0 acquires the lock, having CPUn enter low-power mode while waiting for the lock can save power and improve performance.
• Example code for acquiring the lock

Example code for acquiring the lock

• Example code for releasing the lock

Example code for releasing the lock

Implementing atomic increment using LDXR/STXR (unordered)

loop:
    ldxr    w0, [addr]     // Load value (without memory barrier)
    add     w0, w0, #1
    stxr    w1, w0, [addr] // Attempt to write back
    cbnz    w1, loop       // Retry if failed

This approachdoes not guarantee memory orderingonly guarantees atomicity, and is suitable for scenarios where data is atomically updated but thread synchronization is not involved.

---

Example 2: Implementing a lock using LDAXR/STLXR (ordered)

// try_lock
loop:
    ldaxr   w0, [lock]     // Load lock value (acquire)
    cbnz    w0, loop       // If the lock is already held, retry
    mov     w0, #1
    stlxr   w1, w0, [lock] // Attempt to set the lock (release)
    cbnz    w1, loop       // Retry if failed

This approach isordered, guaranteeing:

• Operations before acquiring the lock will not cross the lock;
• Operations after releasing the lock will not be executed early.

WFE wake-up

• A CPU sleeping via WFE can be woken up in the following ways
  • unmasked interrupt
  • Event (wake-up event)
• Ways to trigger a wake-up event:
  • Executing the SEV instruction
  • The local CPU executing the SEVL instruction
  • Clear the exclusive monitor, changing from exclusive state to open state
• When the CPU holding the lock releases the lock by writing to the lock region with the STLR instruction, a wake-up event is triggered, and the CPU sleeping and waiting on the spinlock is woken up.

WFE wake-up event

Atomic memory access operation

• The following three atomic memory access operations are supported on ARMv8.1 (Large System Extensions)
  • Compare and Swap instructions, CAS and CASP
  • Atomic memory operation instructions
  • Swap instruction
• By ID_AA64ISAR0_The atomic field in the EL1 register is used to determine whether LSE is supported.

ID_AA64ISAR0_EL1

Compare and Swap instruction

• Compare and swap instruction: Check whether the value pointed to by ptr is equal to expected. If equal, assign the value of new to ptr; otherwise do nothing. Regardless of equality, the old value of ptr is returned in the end.

Compare and swap instruction

• On ARMv8.1CAS instruction

CAS <Xs>, <Xt>, [Xn|SP]

If the value at memory address [Xn] == Xs, then store the value of Xt to [Xn], and return the old value of [Xn] to Xs.

CAS instruction

Use of CAS instruction in the Linux kernel

• cmpxchg function prototype

cmpxchg atomically compares the value at ptr with the value of old; if they are equal, sets the value of new at the address of ptr, and returns the old value.

cmpxchg

mov x30, %x[old]

• will Expected valueoldCopy to registerx30
• x30As the comparison register for the CASAL instruction

casal x30, %x[new], %[v]

• Execute CASAL instruction

• Parameters:

  • x30: stores the old value, used for comparison
  • %x[new](x2): new value
  • %[v](*ptr): memory address

• Function:

  1. Compare the value in memory withx30（old）
  2. If equal, write%x[new]to memory address v
  3. If not equal,x30is updated to the current memory value

• Atomicity + Acquire-Release memory ordering semantics, forming abidirectional memory barrier (Full fence)：

  [前面的写]  ----必须在----> CASAL ----必须在----> [后面的读写]

CASAL writes the new value on successful comparison[x0](i.e., memory address *ptr）, on comparison failure returns the old value from memory to the registerx30

mov %x[ret], x30

• writes the value after the operation back to the return value register[ret](bound to x0)
• The returned value tells the caller whether the CAS succeeded or failed

Atomic memory operation instructions

• Atomic load instructions (Atomic loads)

LD<OP> <Xs>, <Xt>,[<Xn|SP>]

equivalent to

tmp = *Xn;
*Xn = *Xn <OP> Xs;
Xt = tmp;

Atomic store instructions

ST<OP> <Xs>,[<Xn|SP>]

equivalent to

*Xn = *Xn <OP> Xs;

• OP

OP operation	Description
ADD	Atomic addition
CLR	Atomic bit clear
SET	Atomic bit set
EOR	Atomic XOR operation
SMAX	Atomic signed maximum
SMIN	Atomic signed minimum operation
UMAX	Maximum value of atomic unsigned integer
UMIN	Minimum value operation for atomic unsigned integer

Example: Using the ldumax instruction to implement a simple spinlock

ldumax

Atomic exchange instruction

swp <Xs>, <Xt>, [<Xn|SP>]

equivalent to

tmp = *Xn;
*Xn = Xs;
Xt = tmp;