Tags
Note: This post provides technical details for our paper “The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms” published in ICPP ’20 conference. To know more details about the paper and our analysis on the performance of CPU pinning for virtual machines and containers, please see here. If you are using the materials of this blog post in your work, we request you to cite the paper.
What is CPU Pinning?
CPU pinning, or what is so called “processor affinity”, helps to bind a process to a certain set of CPU cores. In this way, the pinned process is executed only on the specified (pinned) CPU core(s). CPU Pinning is proven to improve the performance of certain application types (for details, please refer to our paper titled: “The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms“). Although CPU pinning can drastically reduce the overhead of virtualized platforms, it is noteworthy that extensive use of pinning incurs a higher cost and makes the host management more challenging. Therefore, it should be done carefully and only under certain circumstances (see our paper to learn about the circumstances that can benefit pinning).
In this blog post, we are going to teach how to pin virtualized platforms such as Virtual Machines (VMs) and Containers on a host. First, let’s see why someone would need to enable CPU pinning? To answer this question, we need to introduce some basic concepts.
Operating System (OS) Scheduler
Usually, there are numerous processes being executed inside an OS and a queuing system is used to order and manage them. Because each processing core can execute just one process at a time, OS time shares the processes across the cores. Each process is executed for a limited time, known as “quantum”, in a round-robin manner. As shown in the below figure, once a process utilizes its quantum, the OS scheduler picks another process from the list of processes waiting for execution (aka “ready queue”) and dispatches it to the processing core. CPU scheduling module is a critical part of any OS.
NUMA (non-uniform memory access)
Modern computers often have multiple (from two to even tens of) processing cores that are all positioned on one CPU socket. Server-class computers are designed to support as many as CPU cores possible, hence, they generally have several CPU sockets. For instance, the following picture shows a server-class motherboard with two CPU sockets.
Each CPU is assigned a set of memory (RAM) slots, however, the CPUs are able to share their memories via some interconnects (aka Bus), as illustrated in the following figure.
Considering the complexity of having multiple of cores across different CPU sockets, the question is: what happens if a process is allocated on a different CPU cores in different quantums? The answer is simple: the process should reload its state and cache on the new CPU core. It is also possible that the process needs to access its memory via the interconnect. In fact, the normal behavior of the OS scheduler is to disperse the processes as much as it can across all available CPU cores. Although the overhead of using interconnects and reloading the state is worthwhile for many processes to improve the overall CPU utilization, for time-sensitive processes it is better to override the OS scheduler and pinpoint a set of CPU cores to the process. Now, the definition of CPU pinning should makes sense, right?
Provided the above description, in a nutshell, we can consider each CPU socket representing one NUMA. Therefore, in a host with two CPU sockets, we can say there are two NUMA nodes. In Linux shell, you can use the numactl command (with –hardware option) to see the NUMA distribution of your system. Let’s see how numactl describes the CPU cores of a computer with one CPU socket and 4 cores:
Below, you can see the output of numactl for a server machine:
You can see that there are 4 NUMA nodes (sockets) in this system, each featured with 28 CPU cores (totally 112 CPU cores). The exact CPU core ID working in each CPU socket is also shown. In addition, for each NUMA node, the total size of its memory and the amount of its free space is reported. So, if you are to configure CPU pinning and want to find the best set of cores, you would better select them all inside one CPU socket to avoid the interconnect overhead. The last piece of information is a matrix showing the latency of using different NUMA nodes. The latency of each node to itself is the minimum and considered 10. If you require more CPU cores than what is offered in one single socket, you would better to choose the socket that offers the least distance. For example, in the previous figure, NUMA node1 has the lowest distance to node0 and node2 and the highest distance to node3. A graphical NUMA nodes representation of this host is depicted in following figure:
Now that we know the basic concept of OS scheduler and NUMA, we are good to go with the use of CPU pinning technique in virtualization platforms. As you may know, Virtual Machines (VMs) or containers are considered as processes of the host OS. Hence, pinning could be used in order to tune the performance of a specific VM or container in a host. The procedure to enable CPU pinning for each virtualization technology is illustrated in the following. We note that, we chose KVM as the hypervisor for Virtualization, and Docker for the container technology.
How to Perform CPU Pinning
Pinning Virtual Machines in KVM
Let’s consider a VM, called “test”, that is already created with default CPU configurations. Note that pinning must be done before launching a VM. In our scenario, we would like to pin 8 CPU cores to this VM. As a prerequisite, we need to first know the exact CPU core IDs that we are going to assign to the “test” VM. Recall that numactl can help us to know the NUMA nodes and the cores IDs of each one. As an example, let’s pin the test VM to the following core IDs of our server machine: 0,4,8,12,16,20,24,28
These cores reside on NUMA0 which could be a perfect fit for allocate the test VM. All you need to do is to follow these instructions:
- Modify the VM configuration file located in /etc/libvirt/qemu/test.xml
vi /etc/libvirt/qemu/test.xml
2. Find the line starting with “vcpu”:
<vcpu placement='static'>8</vcpu>
3. Change it to look like the following line:
<vcpu placement='static' cpuset='0,4,8,12,16,20,24,28'>8</vcpu>
4. Now just restart libvirtd
systemctl restart libvirtd
This was all! Now you can power on your VM. You can monitor the utilized CPU cores using htop command.
Pinning Docker Containers
Pinning a Docker container is done just at the instantiation time. To do so, just start the container using –cpuset-cpus switch as follows:
docker run --cpuset-cpus=0,4,8,12,16,20,24,28 --name test
You can monitor CPU utilization of the container using htop in the host.