23. The Linux Microcode Loader

Authors:

The kernel has a x86 microcode loading facility which is supposed to provide microcode loading methods in the OS. Potential use cases are updating the microcode on platforms beyond the OEM End-Of-Life support, and updating the microcode on long-running systems without rebooting.

The loader supports three loading methods:

23.1. Early load microcode

The kernel can update microcode very early during boot. Loading microcode early can fix CPU issues before they are observed during kernel boot time.

The microcode is stored in an initrd file. During boot, it is read from it and loaded into the CPU cores.

The format of the combined initrd image is microcode in (uncompressed) cpio format followed by the (possibly compressed) initrd image. The loader parses the combined initrd image during boot.

The microcode files in cpio name space are:

on Intel:

kernel/x86/microcode/GenuineIntel.bin

on AMD :

kernel/x86/microcode/AuthenticAMD.bin

During BSP (BootStrapping Processor) boot (pre-SMP), the kernel scans the microcode file in the initrd. If microcode matching the CPU is found, it will be applied in the BSP and later on in all APs (Application Processors).

The loader also saves the matching microcode for the CPU in memory. Thus, the cached microcode patch is applied when CPUs resume from a sleep state.

Here’s a crude example how to prepare an initrd with microcode (this is normally done automatically by the distribution, when recreating the initrd, so you don’t really have to do it yourself. It is documented here for future reference only).

#!/bin/bash

if [ -z "$1" ]; then
    echo "You need to supply an initrd file"
    exit 1
fi

INITRD="$1"

DSTDIR=kernel/x86/microcode
TMPDIR=/tmp/initrd

rm -rf $TMPDIR

mkdir $TMPDIR
cd $TMPDIR
mkdir -p $DSTDIR

if [ -d /lib/firmware/amd-ucode ]; then
        cat /lib/firmware/amd-ucode/microcode_amd*.bin > $DSTDIR/AuthenticAMD.bin
fi

if [ -d /lib/firmware/intel-ucode ]; then
        cat /lib/firmware/intel-ucode/* > $DSTDIR/GenuineIntel.bin
fi

find . | cpio -o -H newc >../ucode.cpio
cd ..
mv $INITRD $INITRD.orig
cat ucode.cpio $INITRD.orig > $INITRD

rm -rf $TMPDIR

The system needs to have the microcode packages installed into /lib/firmware or you need to fixup the paths above if yours are somewhere else and/or you’ve downloaded them directly from the processor vendor’s site.

23.2. Late loading

You simply install the microcode packages your distro supplies and run:

# echo 1 > /sys/devices/system/cpu/microcode/reload

as root.

The loading mechanism looks for microcode blobs in /lib/firmware/{intel-ucode,amd-ucode}. The default distro installation packages already put them there.

Since kernel 5.19, late loading is not enabled by default.

The /dev/cpu/microcode method has been removed in 5.19.

23.3. Why is late loading dangerous?

23.3.1. Synchronizing all CPUs

The microcode engine which receives the microcode update is shared between the two logical threads in a SMT system. Therefore, when the update is executed on one SMT thread of the core, the sibling “automatically” gets the update.

Since the microcode can “simulate” MSRs too, while the microcode update is in progress, those simulated MSRs transiently cease to exist. This can result in unpredictable results if the SMT sibling thread happens to be in the middle of an access to such an MSR. The usual observation is that such MSR accesses cause #GPs to be raised to signal that former are not present.

The disappearing MSRs are just one common issue which is being observed. Any other instruction that’s being patched and gets concurrently executed by the other SMT sibling, can also result in similar, unpredictable behavior.

To eliminate this case, a stop_machine()-based CPU synchronization was introduced as a way to guarantee that all logical CPUs will not execute any code but just wait in a spin loop, polling an atomic variable.

While this took care of device or external interrupts, IPIs including LVT ones, such as CMCI etc, it cannot address other special interrupts that can’t be shut off. Those are Machine Check (#MC), System Management (#SMI) and Non-Maskable interrupts (#NMI).

23.3.2. Machine Checks

Machine Checks (#MC) are non-maskable. There are two kinds of MCEs. Fatal un-recoverable MCEs and recoverable MCEs. While un-recoverable errors are fatal, recoverable errors can also happen in kernel context are also treated as fatal by the kernel.

On certain Intel machines, MCEs are also broadcast to all threads in a system. If one thread is in the middle of executing WRMSR, a MCE will be taken at the end of the flow. Either way, they will wait for the thread performing the wrmsr(0x79) to rendezvous in the MCE handler and shutdown eventually if any of the threads in the system fail to check in to the MCE rendezvous.

To be paranoid and get predictable behavior, the OS can choose to set MCG_STATUS.MCIP. Since MCEs can be at most one in a system, if an MCE was signaled, the above condition will promote to a system reset automatically. OS can turn off MCIP at the end of the update for that core.

23.3.3. System Management Interrupt

SMIs are also broadcast to all CPUs in the platform. Microcode update requests exclusive access to the core before writing to MSR 0x79. So if it does happen such that, one thread is in WRMSR flow, and the 2nd got an SMI, that thread will be stopped in the first instruction in the SMI handler.

Since the secondary thread is stopped in the first instruction in SMI, there is very little chance that it would be in the middle of executing an instruction being patched. Plus OS has no way to stop SMIs from happening.

23.3.4. Non-Maskable Interrupts

When thread0 of a core is doing the microcode update, if thread1 is pulled into NMI, that can cause unpredictable behavior due to the reasons above.

OS can choose a variety of methods to avoid running into this situation.

23.3.5. Is the microcode suitable for late loading?

Late loading is done when the system is fully operational and running real workloads. Late loading behavior depends on what the base patch on the CPU is before upgrading to the new patch.

This is true for Intel CPUs.

Consider, for example, a CPU has patch level 1 and the update is to patch level 3.

Between patch1 and patch3, patch2 might have deprecated a software-visible feature.

This is unacceptable if software is even potentially using that feature. For instance, say MSR_X is no longer available after an update, accessing that MSR will cause a #GP fault.

Basically there is no way to declare a new microcode update suitable for late-loading. This is another one of the problems that caused late loading to be not enabled by default.

23.4. Builtin microcode

The loader supports also loading of a builtin microcode supplied through the regular builtin firmware method CONFIG_EXTRA_FIRMWARE. Only 64-bit is currently supported.

Here’s an example:

CONFIG_EXTRA_FIRMWARE="intel-ucode/06-3a-09 amd-ucode/microcode_amd_fam15h.bin"
CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"

This basically means, you have the following tree structure locally:

/lib/firmware/
|-- amd-ucode
...
|   |-- microcode_amd_fam15h.bin
...
|-- intel-ucode
...
|   |-- 06-3a-09
...

so that the build system can find those files and integrate them into the final kernel image. The early loader finds them and applies them.

Needless to say, this method is not the most flexible one because it requires rebuilding the kernel each time updated microcode from the CPU vendor is available.