Timeline
Timeline
2025-10-31
init
This article introduces the basic principles of ARM ACE (AXI Coherency Extensions) as a hardware cache coherence solution based on the AXI bus, elaborating on its method of achieving system-level cache coherence among multiple masters through the snoop mechanism, additional transmission channels and signal lines, and a five-state cache state machine. Additionally, it discusses the partitioning of shareability domains, specific scenarios of snoop transactions, the key role of RACK/WACK signals in ensuring transaction ordering and coherence, as well as extended mechanisms such as exclusive access and DVM.
Reference documents:
ACE
- ACE (AXI Coherency Extensions) is a hardware cache coherence solution based on the AXI bus
- Focus on system-level cache coherence
Support for the ACE interface first started with the big.LITTLE architecture of Cortex-A15/A7
Cortex-A9 does not support the ACE interface; it connects to L2 and memory via the AXI interface
Cortex-A15 supports the ACE interface; in the big.LITTLE architecture, the big cluster and little cluster connect to the CCI bus via ACE
- Three masters, each with a local cache. ACE allows all three masters to have the same cache copy for the same memory address
- Here, master generally refers to a CPU cluster or a controller with a cache
- ACE ensures that for a given address, all masters can access the correct data
ACE focuses on system cache coherence issues between masters with caches, such as the Cortex-A72 family and Cortex-A53 family.
ACE-Lite is used to connect hardware IO devices without caches, but these devices need to access memory with system cache coherence, such as GPUs, SMMUs, etc. The DVM interface is used for broadcast TLB invalidation.
Basic approach of ACE implementation
- Need toImplement a snoop mechanism to ensure cache coherence among multiple masters.
- Add new signal lines and new transaction types on top of AXI to implement snooping.
- Add new transmission channels on top of AXI.
- Implement a 5-state cache state transition mechanism.(Because some masters use MESI, some use MOESI, etc.)
- ACE supports memory barrier (removed in ACE5).
- ACE supports exclusive access.
- ACE supports DVM.
ACE state machine
valid: cache line is valid
invalid: cache line is invalid
unique: indicates this cache line is exclusive, only the current master has it
shared: multiple masters have copies of this cache line
clean: indicates the cache line content is consistent with memory
dirty: the cache line content is inconsistent with memory and needs to be written back later
similar to MOESI protocol
new channels added by ACE
new signals added by ACE
- AxDOMAIN: used to indicate the shareability domain
- Nono-shareable，Inner Shareable，Outer Shareable，System
- AxSNOOP: used to indicate the snoop transaction type
- AxBAR: indicates initiating a barrier transaction (removed in ACE5)
- AWUNIQUE: used to optimize cache state transitions for write transactions
AxDOMAIN signal line
AxSNOOP signal line
- AxSNOOP: used to indicate the type of snoop transaction
- Non-snooping
- Coherent
- Memory update
- Cache maintenance
- DVM
- barrier
shareability domains
Before initiating a coherency or barrier transaction, the master uses this to determine which masters the transaction should be sent to
- Coherent transaction: determines which masters may have a copy of this data, used to send snoop transactions
- Barrier transaction: determines which masters will establish ordering relationships, and how far the barrier transaction should propagate
Supports 4 shareability domains:
- Non-shareable: contains only one master
- Inner Shareable
- Outer Shareable
- System
4 shareability domains
- Inner share: typically refers to caches integrated within the CPU, closest to the processor core, for example, the Cortex-A72 core can integrate L1 and L2 caches
- Outer share: caches extended via the system bus, such as an extended L3 cache connected to the system bus
Snoop transaction
Example 1: shareable read and miss
Final:
Example 2: shareable read and hit
Example 3: shareable write 1 (full cache line)
Example 4: shareable write 2 (partial cache line)
Example 5: Simultaneous ReadUnique write operations
- At time T0, the cache line states of Master0 and Master1 are both I, and the initial value at address A is 0x11223344
- At time T, Master0 and Master1 simultaneously initiate write operations to address A
- Master0 wants to write to address A: 0x555667788
- Master1 wants to write to address: 0xaabbccdd
The bus will perform arbitration, processing one request before handling the other
RACK and WACK Signal Lines
- In Example 5, the RACK signal line is used to indicate that the ReadUnique transaction has completed
- On the AXI bus, multiple outstanding and out-of-order transfers are supported, but in ACE, cache coherence issues may arise
- The RACK signal line is used to indicate that a read transaction has completed
- The WACK signal line is used to indicate that a write transaction has completed
- The interconnect bus IP uses the RACK/WACK signal lines to ensure thata previous transaction to an address has been processed before sending snooping for other transactions to other masters
- For a read transaction, when the RLAST signal is asserted, it indicates that the last read transfer is complete, and then the master can send a read acknowledge signal
- For a write transaction, after the write response channel sends a response handshake, the master can send the WACK signal
Example 6: Use of the RACK Signal Line - Problem Introduction
At time T0: The cache line states of Master0 and Master1 are both SC
At time T1: Master0 wants to rewrite the data at address A to: 0xAABBCCDD
At time T2: Master1 performs a read operation on address A
Root Cause
When Master1 initiates a ReadShared transaction, Master1 assumes that the previous MakeUnique transaction from Master0 has completed, but in reality, it is blocked in some intermediate state and has not yet finished.
Example 6: Use of RACK Signal Line - Problem Solution
Solution: Master0 uses the RACK signal to inform the bus IP that the MakeUnique transaction has completed, so that the bus can initiate other snoop-related transactions.
exclusive access
Exclusive Access Flow
- Execute exclusive load
- Compute
- Execute exclusive store
- If another master writes to this address, fail
- If no other master writes to this address, success
For exclusive access to Non-shareable and System Shareable memory addresses, the behavior is the same as AXI
For inner share and outer share memory
- The master exclusive monitor ensures that after an exclusive load, no other master writes to this address
- The PoS exclusive monitor of the bus interconnect serializes access
DVM（Distributed Virtual Memory）
- Send broadcast to other masters:
- TLB invalidate
- Branch Predictor Invalidate
- Physical Instruction Cache Invalidate
- Virtual Instruction Cache Invalidate
- Synchronization
DVM transfer transactions - DVM message transactions
DVM message transaction processing flow:
- The master sends a DVM message request through the read address channel
- After the bus receives it, it sends a snoop to other masters
- After other masters receive it, they respond through the snoop response channel
- After the bus collects all responses, it sends a response packet to the initiating master through the read data channel
DVM transfer transactions - DVM Synchronization and DVM Complete transactions
DVM sync and complete transaction processing flow:
- The initiating master sends a DVM synchronization request via the read address channel.
- After receiving it, the bus sends a snoop broadcast to other masters.
- After receiving it, other masters send responses via the snoop response channel.
- After collecting all response packets, the bus sends a response packet to the initiating master via the read data channel.
- Each participating master, after completing TLB invalidation, sends a DVM complete transaction via the read address channel.
- After receiving the DVM complete, the bus immediately responds to the master.
- After the bus collects all DVM complete transactions from participating masters, it sends a DVM complete to the initiating master via the snoop address channel.
- The initiating master responds to the bus via the snoop response channel.
DVM message packetization
Different encodings of AxADDR are used to implement different instructions, such as TLBI.
Example: Modifying PTE
ACE-Lite
- ACE-Lite is applied to hardware devices that do not need to participate in system cache coherence, such as devices without local caches.
- GPU
- SMMU
- Transactions supported by ACE-Lite
- Non-coherent transactions: ReadNoSnoop, WriteNoSnoop
- Partially coherent transactions: ReadOnce, WriteUnique, WriteLineUnique
- Cache maintenance transactions: CleanShared, CleanInvalid, MakeInvalid
ACE-Lite + DVM combination
ACE-Lite+DVM application scenario
Example: ACE_Lite master performs a partial write
- At time T0: The cache line on Master0 is in UD state, assuming the cache line value is: aa bb cc dd
- At time T1: Master1 performs a partial cache line write operation to address A, writing 11 22
