ARM Atomic Operation

even629

2025-10-30 (Updated: 2025-10-30)

Timeline

2025-10-30

init

This article introduces the atomic operation mechanism under the ARM architecture, discussing in detail the working principle of Load-exclusive/Store-exclusive (LL/SC) instructions based on the exclusive monitor in ARMv8, multi-core contention analysis, and low-power lock applications combined with the WFE instruction. Additionally, it summarizes the atomic instructions introduced by ARMv8.1's LSE (Large System Extensions), such as the implementation principles of atomic memory access operations like Compare and Swap.

Reference documents:

Arm Architecture Reference Manual Armv8

https://developer.arm.com/documentation/ddi0487/fc/

Content related to exclusive memory access in the ARMv8.6 chip manual

Chapter B2.9 Synchronization and semaphores
Chapter D1.16 Mechanisms for entering a low-power state
Chapter C3.2.13 Compare and Swap
Chapter C3.2.13 Atomic memory operations
Chapter C3.2.14 Swap

Why atomic operations are needed

thread_A_func and thread_B_Both func attempt to perform the i++ operation

Basic atomic operation functions in the Linux kernel

The Linux kernel provides atomic variables of type atomic_t, whose implementation depends on different architectures.
Atomic operation functions of type atomic_t can guarantee the atomicity and integrity of an operation.
“Read-Modify-Write” mechanism
- Reading the value of the atomic variable into a general-purpose register
- Modifying the value of the atomic variable in the general-purpose register
- Writing the new value back to memory

If the CPU only reads (loads) a variable’s value from memory or only writes (stores) a variable’s value to memory, these operations are indivisible.

The following operation function APIs use the “Read-Modify-Write” mechanism

ARMv8’s support for atomic operations

ARMv8 provides two ways of atomic operations
- Traditional Load-exclusive and Store-exclusive methods
  - Supported on ARMv8
  - LL/SC(Load-link/store-conditional)
- LSE (Large System Extensions) supports atomic operation instructions
  - Supported starting from ARMv8.1, ARMv8.1-LSE
  - Added Compare and Swap instructions
  - Added Atomic memory operation instructions
  - Added Swap instruction

Load-exclusive and store-exclusive instructions

ldxrInstruction: Load-exclusive. Loads the value from a memory address into a general-purpose register in an exclusive manner

1	ldxr <xt> , [xn\|sp]

stxrInstruction: Store-exclusive. Stores new data to memory in an exclusive manner

1	stxr <ws> , <xt> , [xn\|sp]

Load/Store Exclusive Pair instructions

1 2	ldxp <Xt1>, <Xt2>, [Xn\|SP] stxp <Ws>, <Xt1>, <Xt2>, [<Xn\|SP>]

Load/Store Exclusive instructions with acquire and release semantics

Example

The exclusive monitor monitors memory access, marking the memory address in exclusive access mode to ensure exclusive access to that memory address without interference from other factors

Exclusive Monitor

The exclusive monitor has two states:
- Open Access state
- Exclusive Access state
When the ldxr instruction loads data from memory, the CPU marks this memory address as being in the Exclusive Access state.
When the CPU executes the stxr instruction, it must make a decision based on the state of the exclusive monitor.
- If the exclusive monitor is in the Exclusive Access state, the stxr instruction stores successfully, returns 0, and the exclusive monitor transitions to the Open Access state.
- If the exclusive monitor is in the Open Access state, the stxr store fails and returns 1.

Notes

The exclusive monitor itself is not used to prevent CPU cores from accessing the marked memory; it does not lock the bus.
The exclusive monitor merely serves a monitoring role, tracking changes in state.
The exclusive monitor should not be regarded as a hardware lock.

Architecture of the Exclusive Monitor

Typically, a system consists of multiple levels of exclusive monitors (defined during chip design).
- Local monitor,Applicable to non-shareable memory
- Internal coherent global monitor,Applicable to normal memory
- External global monitor,Applicable to device memory
Some SoCs do not support external global monitors. For example, the BMC2711 used on the Raspberry Pi 4B
When the MMU is not enabled, accessing physical memory becomes accessing device memory. Using ldxr and stxr instructions at this time will cause unpredictable errors

The use of the ldxr instruction has many restrictions, requiring the memory to be normal memory and shareable
If accessing device memory, such as when the MMU is not enabled, the CPU IP core must support exclusive access to device memory. This requires consulting the specific CPU ID manual

Granularity of Exclusive Monitor

The ERG (Exclusives Reservation Granule) in the CTR_EL1 register defines the minimum unit of the exclusive monitor
The ERG can be defined in the range of 4 words to 512 words, but it is usually the size of a cache line
Example:
- Assume ERG is 2^4, i.e., 16 bytes. When using the ldrxb instruction to perform an exclusive read operation on address 0x341B4, addresses from 0x341b0 to 0x341bf will all be marked as exclusive access.

Case 1: Implementation of the atomic_add() function

Case 2: Implementation of a simple lock (spinlock)

cbnz w2, retry

Analysis of ldxr and stxr in multi-core scenarios

CPU0 and CPU1 simultaneously execute the get_lock() operation

Time T0Initialization state

Times T1 and T2CPU0 executes the ldxr instruction

Time T3CPU1 executes the ldxr instruction

At time T4CPU0 acquires the lock via the stxr instruction

At time T5CPU1 attempts to acquire the lock via the stxr instruction

Application of the WFE instruction in lock implementation

If CPU0 acquires the lock, while CPUn is waiting for the lock, putting the CPU into low-power mode can save power and improve performance
Example code for acquiring a lock

Example code for releasing a lock

Implementing atomic increment (unordered) using LDXR/STXR

loop:
    ldxr    w0, [addr]     // Load value (without memory barrier)
    add     w0, w0, #1
    stxr    w1, w0, [addr] // Attempt to write back
    cbnz    w1, loop       // Retry if failed

This approach isunordered, suitable for atomic data updates without thread synchronization.

Example 2: Implementing a lock with LDAXR/STLXR (ordered)

// try_lock
loop:
    ldaxr   w0, [lock]     // Load lock value (acquire)
    cbnz    w0, loop       // Retry if lock is already held
    mov     w0, #1
    stlxr   w1, w0, [lock] // Attempt to set lock (release)
    cbnz    w1, loop       // Retry if failed

This approach isordered, guaranteeing:
Operations before acquiring the lock do not cross the lock;
Operations after releasing the lock will not be executed early.

WFE Wake-up

A CPU that sleeps via WFE can be woken up in the following ways:
- unmasked interrupt
- Event (Wake-up Event)
Ways to trigger a wake-up event:
- Executing the SEV instruction
- The local CPU executes the SEVL instruction
- Clearing the exclusive monitor, changing from exclusive state to open state
When the CPU holding the lock writes to the lock region via the STLR instruction to release the lock, it triggers a wake-up event, waking up the CPU that is sleeping and waiting for the spinlock.

Atomic Memory Access

ARMv8.1 supports the following three atomic memory access operations (Large System Extensions):
- Compare and Swap instructions, CAS and CASP
- Atomic memory operation instructions
- Swap instruction
By ID_AA64ISAR0_The atomic field in the EL1 register determines whether LSE is supported.

Compare and Swap instruction

Compare and Swap instruction: Checks whether the value pointed to by ptr equals expected. If equal, assigns the new value to ptr; otherwise, does nothing. Regardless of equality, it ultimately returns the old value of ptr.

On ARMv8.1CAS instruction

1	CAS <Xs>, <Xt>, [Xn\|SP]

If the value of Xn (Xn is an address) == Xs, then store the value of Xt in Xn, and return Xs, which equals the old value of Xn

Usage of CAS instruction in the Linux kernel

cmpxchg function prototype

cmpxchg atomically compares whether the value at ptr equals old; if equal, sets the new value to the address ptr and returns the old value

mov x30, %x[old]

Copyexpected valueoldto registerx30
x30as the comparison register for the CASAL instruction

casal x30, %x[new], %[v]

ExecuteCASAL instruction
Parameters:
- x30: Stores old value for comparison
- %x[new](x2): New value
- %[v](*ptr): Memory address
Function:
1. Compare the value in memory withx30（old）
2. If equal, write%x[new]to memory address v
3. If not equal,x30is updated to the current memory value
Atomicity + Acquire-Release memory ordering semantics, forming abidirectional memory barrier (Full fence)：
1
[前面的写] ----必须在----> CASAL ----必须在----> [后面的读写]

CASAL writes the new value to[x0](i.e., memory address *ptr）, on failure returns the old value from memory to registerx30

mov %x[ret], x30

Write the result of the operation back to the return value register[ret](bound to x0)
The returned value can tell the caller whether CAS succeeded or failed

Atomic memory operation instructions

Atomic loads

1	LD<OP> <Xs>, <Xt>,[<Xn\|SP>]

Equivalent to

1
2
3

tmp = *Xn;
*Xn = *Xn <OP> Xs;
Xt = tmp;

Atomic store instructions

1	ST<OP> <Xs>,[<Xn\|SP>]

Equivalent to

1	Xn = Xn <OP> Xs;

OP operation	Description
ADD	Atomic add
CLR	Atomic bit clear
SET	Atomic bit set
EOR	Atomic XOR operation
SMAX	Atomic signed maximum
SMIN	Atomic signed minimum operation
UMAX	Atomic unsigned maximum operation
UMIN	Atomic unsigned minimum operation

Example: Implementing a simple spinlock using the ldumax instruction

Atomic exchange instruction

1	swp <Xs>, <Xt>, [<Xn\|SP>]

Equivalent to

1
2
3

tmp = *Xn;
*Xn = Xs;
Xt = tmp;