How to configure NUMA nodes with QEMU
QEMU does allow very flexible configuration of NUMA node topology.
When starting QEMU we can select a NUMA topology with the -numa argument.
In its most basic form, we can specify the CPUs assigned to each NUMA node.
-smp cpus=16 -numa node,cpus=0-3,nodeid=0 \
-numa node,cpus=4-7,nodeid=1 \
-numa node,cpus=8-11,nodeid=2 \
-numa node,cpus=12-15,nodeid=3 \
This gives us a system that looks like the following from lscpu.
Note below that we specified 4 NUMA nodes with each NUMA node containing 4 CPUs.
qemu@ubuntu:~$ lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 4
Vendor ID: ARM
Model: 0
Model name: Cortex-A57
Stepping: r1p0
BogoMIPS: 125.00
NUMA node0 CPU(s): 0-3
NUMA node1 CPU(s): 4-7
NUMA node2 CPU(s): 8-11
NUMA node3 CPU(s): 12-15
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
To go one step further we can also specify the NUMA distance between each CPU.
Suppose we have the following NUMA topology.
________ ________
| | | |
| Node 0 | | Node 1 |
| |-15-| |
|________| |________|
| \ / |
20 20 20
____|___ / \ ____|___
| |/ | |
| Node 2 | | Node 3 |
| |-15-| |
|________| |________|
We can use the -numa dist option to add on the specific NUMA distances.
-smp cpus=16 -numa node,cpus=0-3,nodeid=0 \
-numa node,cpus=4-7,nodeid=1 \
-numa node,cpus=8-11,nodeid=2 \
-numa node,cpus=12-15,nodeid=3 \
-numa dist,src=0,dst=1,val=15 \
-numa dist,src=2,dst=3,val=15 \
-numa dist,src=0,dst=2,val=20 \
-numa dist,src=0,dst=3,val=20 \
-numa dist,src=1,dst=2,val=20 \
-numa dist,src=1,dst=3,val=20
The numactl command will confirm the distances that we configured in QEMU.
qemu@ubuntu:~$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3
node 0 size: 983 MB
node 0 free: 852 MB
node 1 cpus: 4 5 6 7
node 1 size: 1007 MB
node 1 free: 923 MB
node 2 cpus: 8 9 10 11
node 2 size: 943 MB
node 2 free: 812 MB
node 3 cpus: 12 13 14 15
node 3 size: 1007 MB
node 3 free: 916 MB
node distances:
node 0 1 2 3
0: 10 15 20 20
1: 15 10 20 20
2: 20 20 10 15
3: 20 20 15 10