How to verify bugs and bisect regressions¶

This document describes how to check if some Linux kernel problem occurs in code currently supported by developers -- to then explain how to locate the change causing the issue, if it is a regression (e.g. did not happen with earlier versions).

The text aims at people running kernels from mainstream Linux distributions on commodity hardware who want to report a kernel bug to the upstream Linux developers. Despite this intent, the instructions work just as well for users who are already familiar with building their own kernels: they help avoid mistakes occasionally made even by experienced developers.

The essence of the process (aka ‘TL;DR’)¶

[If you are new to building or bisecting Linux, ignore this section and head over to the ‘step-by-step guide’ below. It utilizes the same commands as this section while describing them in brief fashion. The steps are nevertheless easy to follow and together with accompanying entries in a reference section mention many alternatives, pitfalls, and additional aspects, all of which might be essential in your present case.]

In case you want to check if a bug is present in code currently supported by developers, execute just the preparations and segment 1; while doing so, consider the newest Linux kernel you regularly use to be the ‘working’ kernel. In the following example that’s assumed to be 6.0, which is why its sources will be used to prepare the .config file.

In case you face a regression, follow the steps at least till the end of segment 2. Then you can submit a preliminary report -- or continue with segment 3, which describes how to perform a bisection needed for a full-fledged regression report. In the following example 6.0.13 is assumed to be the ‘working’ kernel and 6.1.5 to be the first ‘broken’, which is why 6.0 will be considered the ‘good’ release and used to prepare the .config file.

Preparations: set up everything to build your own kernels:

# * Remove any software that depends on externally maintained kernel modules
#   or builds any automatically during bootup.
# * Ensure Secure Boot permits booting self-compiled Linux kernels.
# * If you are not already running the 'working' kernel, reboot into it.
# * Install compilers and everything else needed for building Linux.
# * Ensure to have 15 Gigabyte free space in your home directory.
git clone -o mainline --no-checkout \
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
git switch --detach v6.0
# * Hint: if you used an existing clone, ensure no stale .config is around.
make olddefconfig
# * Ensure the former command picked the .config of the 'working' kernel.
# * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
#   VPNs, mount network shares, and briefly try the feature that is broken.
yes '' | make localmodconfig
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
./scripts/config -e CONFIG_LOCALVERSION_AUTO
# * Note, when short on storage space, check the guide for an alternative:
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
  -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
# * Hint: at this point you might want to adjust the build configuration;
#   you'll have to, if you are running Debian.
make olddefconfig
cp .config ~/kernel-config-working

Segment 1: build a kernel from the latest mainline codebase.

This among others checks if the problem was fixed already and which developers later need to be told about the problem; in case of a regression, this rules out a .config change as root of the problem.

Checking out latest mainline code:

cd ~/linux/
git switch --discard-changes --detach mainline/master

Build, install, and boot a kernel:

cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Make sure there is enough disk space to hold another kernel:
df -h /boot/ /lib/modules/
# * Note: on Arch Linux, its derivatives and a few other distributions
#   the following commands will do nothing at all or only part of the
#   job. See the step-by-step guide for further details.
sudo make modules_install
command -v installkernel && sudo make install
# * Check how much space your self-built kernel actually needs, which
#   enables you to make better estimates later:
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
du -sh /lib/modules/$(make -s kernelrelease)/
# * Hint: the output of the following command will help you pick the
#   right kernel from the boot menu:
make -s kernelrelease | tee -a ~/kernels-built
reboot
# * Once booted, ensure you are running the kernel you just built by
#   checking if the output of the next two commands matches:
tail -n 1 ~/kernels-built
uname -r
cat /proc/sys/kernel/tainted

Check if the problem occurs with this kernel as well.

Segment 2: ensure the ‘good’ kernel is also a ‘working’ kernel.

This among others verifies the trimmed .config file actually works well, as bisecting with it otherwise would be a waste of time:
1. Start by checking out the sources of the ‘good’ version:
```
cd ~/linux/
git switch --discard-changes --detach v6.0
```
2. Build, install, and boot a kernel as described earlier in segment 1, section b -- just feel free to skip the ‘du’ commands, as you have a rough estimate already.
3. Ensure the feature that regressed with the ‘broken’ kernel actually works with this one.
Segment 3: perform and validate the bisection.
1. Retrieve the sources for your ‘bad’ version:
```
git remote set-branches --add stable linux-6.1.y
git fetch stable
```
2. Initialize the bisection:
```
cd ~/linux/
git bisect start
git bisect good v6.0
git bisect bad v6.1.5
```
3. Build, install, and boot a kernel as described earlier in segment 1, section b.
  
  In case building or booting the kernel fails for unrelated reasons, run git bisect skip. In all other outcomes, check if the regressed feature works with the newly built kernel. If it does, tell Git by executing git bisect good; if it does not, run git bisect bad instead.
  
  All three commands will make Git check out another commit; then re-execute this step (e.g. build, install, boot, and test a kernel to then tell Git the outcome). Do so again and again until Git shows which commit broke things. If you run short of disk space during this process, check the section ‘Complementary tasks: cleanup during and after the process’ below.
4. Once your finished the bisection, put a few things away:
```
cd ~/linux/
git bisect log > ~/bisect-log
cp .config ~/bisection-config-culprit
git bisect reset
```
5. Try to verify the bisection result:
```
git switch --discard-changes --detach mainline/master
git revert --no-edit cafec0cacaca0
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
```
This is optional, as some commits are impossible to revert. But if the second command worked flawlessly, build, install, and boot one more kernel kernel; just this time skip the first command copying the base .config file over, as that already has been taken care off.
Complementary tasks: cleanup during and after the process.
1. To avoid running out of disk space during a bisection, you might need to remove some kernels you built earlier. You most likely want to keep those you built during segment 1 and 2 around for a while, but you will most likely no longer need kernels tested during the actual bisection (Segment 3 c). You can list them in build order using:
```
ls -ltr /lib/modules/*-local*
```
To then for example erase a kernel that identifies itself as ‘6.0-rc1-local-gcafec0cacaca0’, use this:
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0 # * Note, on some distributions kernel-install is missing # or does only part of the job.
1. If you performed a bisection and successfully validated the result, feel free to remove all kernels built during the actual bisection (Segment 3 c); the kernels you built earlier and later you might want to keep around for a week or two.
Optional task: test a debug patch or a proposed fix later:
```
git fetch mainline
git switch --discard-changes --detach mainline/master
git apply /tmp/foobars-proposed-fix-v1.patch
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
```
Build, install, and boot a kernel as described in segment 1, section b -- but this time omit the first command copying the build configuration over, as that has been taken care of already.

Step-by-step guide on how to verify bugs and bisect regressions¶

This guide describes how to set up your own Linux kernels for investigating bugs or regressions you intend to report. How far you want to follow the instructions depends on your issue:

Execute all steps till the end of segment 1 to verify if your kernel problem is present in code supported by Linux kernel developers. If it is, you are all set to report the bug -- unless it did not happen with earlier kernel versions, as then your want to at least continue with segment 2 to check if the issue qualifies as regression which receive priority treatment. Depending on the outcome you then are ready to report a bug or submit a preliminary regression report; instead of the latter your could also head straight on and follow segment 3 to perform a bisection for a full-fledged regression report developers are obliged to act upon.

Preparations: set up everything to build your own kernels.

Segment 1: try to reproduce the problem with the latest codebase.

Segment 2: check if the kernels you build work fine.

Segment 3: perform a bisection and validate the result.

Complementary tasks: cleanup during and after following this guide.

Optional tasks: test reverts, patches, or later versions.

The steps in each segment illustrate the important aspects of the process, while a comprehensive reference section holds additional details for almost all of the steps. The reference section sometimes also outlines alternative approaches, pitfalls, as well as problems that might occur at the particular step -- and how to get things rolling again.

For further details on how to report Linux kernel issues or regressions check out Reporting issues, which works in conjunction with this document. It among others explains why you need to verify bugs with the latest ‘mainline’ kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6), even if you face a problem with a kernel from a ‘stable/longterm’ series (say 6.0.13).

For users facing a regression that document also explains why sending a preliminary report after segment 2 might be wise, as the regression and its culprit might be known already. For further details on what actually qualifies as a regression check out Reporting regressions.

If you run into any problems while following this guide or have ideas how to improve it, please let the kernel developers know.

Preparations: set up everything to build your own kernels¶

The following steps lay the groundwork for all further tasks.

Note: the instructions assume you are building and testing on the same machine; if you want to compile the kernel on another system, check Build kernels on a different machine below.

Create a fresh backup and put system repair and restore tools at hand, just to be prepared for the unlikely case of something going sideways.

[details]

Remove all software that depends on externally developed kernel drivers or builds them automatically. That includes but is not limited to DKMS, openZFS, VirtualBox, and Nvidia’s graphics drivers (including the GPLed kernel module).

[details]

On platforms with ‘Secure Boot’ or similar solutions, prepare everything to ensure the system will permit your self-compiled kernel to boot. The quickest and easiest way to achieve this on commodity x86 systems is to disable such techniques in the BIOS setup utility; alternatively, remove their restrictions through a process initiated by mokutil --disable-validation.

[details]

Determine the kernel versions considered ‘good’ and ‘bad’ throughout this guide:
- Do you follow this guide to verify if a bug is present in the code the primary developers care for? Then consider the version of the newest kernel you regularly use currently as ‘good’ (e.g. 6.0, 6.0.13, or 6.1-rc2).
- Do you face a regression, e.g. something broke or works worse after switching to a newer kernel version? In that case it depends on the version range during which the problem appeared:
  - Something regressed when updating from a stable/longterm release (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a stable/longterm version based on one (say 6.1.5)? Then consider the mainline release your working kernel is based on to be the ‘good’ version (e.g. 6.0) and the first version to be broken as the ‘bad’ one (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed that 6.0 is fine; this hypothesis will be checked in segment 2.
  - Something regressed when switching from one mainline version (say 6.0) to a later one (like 6.1-rc1) or a stable/longterm release based on it (say 6.1.5)? Then regard the last working version (e.g. 6.0) as ‘good’ and the first broken (e.g. 6.1-rc1 or 6.1.5) as ‘bad’.
  - Something regressed when updating within a stable/longterm series (say from 6.0.13 to 6.0.15)? Then consider those versions as ‘good’ and ‘bad’ (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series.
Note, do not confuse ‘good’ version with ‘working’ kernel; the latter term throughout this guide will refer to the last kernel that has been working fine.

[details]

Boot into the ‘working’ kernel and briefly use the apparently broken feature.

[details]

Ensure to have enough free space for building Linux. 15 Gigabyte in your home directory should typically suffice. If you have less available, be sure to pay attention to later steps about retrieving the Linux sources and handling of debug symbols: both explain approaches reducing the amount of space, which should allow you to master these tasks with about 4 Gigabytes free space.

[details]

Install all software required to build a Linux kernel. Often you will need: ‘bc’, ‘binutils’ (‘ld’ et al.), ‘bison’, ‘flex’, ‘gcc’, ‘git’, ‘openssl’, ‘pahole’, ‘perl’, and the development headers for ‘libelf’ and ‘openssl’. The reference section shows how to quickly install those on various popular Linux distributions.

[details]

Retrieve the mainline Linux sources; then change into the directory holding them, as all further commands in this guide are meant to be executed from there.

Note, the following describe how to retrieve the sources using a full mainline clone, which downloads about 2,75 GByte as of early 2024. The reference section describes two alternatives : one downloads less than 500 MByte, the other works better with unreliable internet connections.

Execute the following command to retrieve a fresh mainline codebase while preparing things to add branches for stable/longterm series later:
```
git clone -o mainline --no-checkout \
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
```
[details]

Is one of the versions you earlier established as ‘good’ or ‘bad’ a stable or longterm release (say 6.1.5)? Then download the code for the series it belongs to (‘linux-6.1.y’ in this example):
```
git remote set-branches --add stable linux-6.1.y
git fetch stable
```

Start preparing a kernel build configuration (the ‘.config’ file).

Before doing so, ensure you are still running the ‘working’ kernel an earlier step told you to boot; if you are unsure, check the current kernelrelease identifier using uname -r.

Afterwards check out the source code for the version earlier established as ‘good’. In the following example command this is assumed to be 6.0; note that the version number in this and all later Git commands needs to be prefixed with a ‘v’:
```
git switch --discard-changes --detach v6.0
```
Now create a build configuration file:
```
make olddefconfig
```
The kernel build scripts then will try to locate the build configuration file for the running kernel and then adjust it for the needs of the kernel sources you checked out. While doing so, it will print a few lines you need to check.

Look out for a line starting with ‘# using defaults found in’. It should be followed by a path to a file in ‘/boot/’ that contains the release identifier of your currently working kernel. If the line instead continues with something like ‘arch/x86/configs/x86_64_defconfig’, then the build infra failed to find the .config file for your running kernel -- in which case you have to put one there manually, as explained in the reference section.

In case you can not find such a line, look for one containing ‘# configuration written to .config’. If that’s the case you have a stale build configuration lying around. Unless you intend to use it, delete it; afterwards run ‘make olddefconfig’ again and check if it now picked up the right config file as base.

[details]

Disable any kernel modules apparently superfluous for your setup. This is optional, but especially wise for bisections, as it speeds up the build process enormously -- at least unless the .config file picked up in the previous step was already tailored to your and your hardware needs, in which case you should skip this step.

To prepare the trimming, connect external hardware you occasionally use (USB keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted since you started that guide, ensure that you tried using the feature causing trouble since you started the system. Only then trim your .config:
```
yes '' | make localmodconfig
```
There is a catch to this, as the ‘apparently’ in initial sentence of this step and the preparation instructions already hinted at:

The ‘localmodconfig’ target easily disables kernel modules for features only used occasionally -- like modules for external peripherals not yet connected since booting, virtualization software not yet utilized, VPN tunnels, and a few other things. That’s because some tasks rely on kernel modules Linux only loads when you execute tasks like the aforementioned ones for the first time.

This drawback of localmodconfig is nothing you should lose sleep over, but something to keep in mind: if something is misbehaving with the kernels built during this guide, this is most likely the reason. You can reduce or nearly eliminate the risk with tricks outlined in the reference section; but when building a kernel just for quick testing purposes this is usually not worth spending much effort on, as long as it boots and allows to properly test the feature that causes trouble.

[details]

Ensure all the kernels you will build are clearly identifiable using a special tag and a unique version number:
```
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
./scripts/config -e CONFIG_LOCALVERSION_AUTO
```
[details]

Decide how to handle debug symbols.

In the context of this document it is often wise to enable them, as there is a decent chance you will need to decode a stack trace from a ‘panic’, ‘Oops’, ‘warning’, or ‘BUG’:
```
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
  -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
```
But if you are extremely short on storage space, you might want to disable debug symbols instead:
```
./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
  -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
```
[details]

Check if you may want or need to adjust some other kernel configuration options:
- Are you running Debian? Then you want to avoid known problems by performing additional adjustments explained in the reference section.
  
  [details].
- If you want to influence other aspects of the configuration, do so now using your preferred tool. Note, to use make targets like ‘menuconfig’ or ‘nconfig’, you will need to install the development files of ncurses; for ‘xconfig’ you likewise need the Qt5 or Qt6 headers.
  
  [details].

Reprocess the .config after the latest adjustments and store it in a safe place:
```
make olddefconfig
cp .config ~/kernel-config-working
```
[details]

Segment 1: try to reproduce the problem with the latest codebase¶

The following steps verify if the problem occurs with the code currently supported by developers. In case you face a regression, it also checks that the problem is not caused by some .config change, as reporting the issue then would be a waste of time. [details]

Check out the latest Linux codebase.
- Are your ‘good’ and ‘bad’ versions from the same stable or longterm series? Then check the front page of kernel.org: if it lists a release from that series without an ‘[EOL]’ tag, checkout the series latest version (‘linux-6.1.y’ in the following example):
```
cd ~/linux/
git switch --discard-changes --detach stable/linux-6.1.y
```
  Your series is unsupported, if is not listed or carrying a ‘end of life’ tag. In that case you might want to check if a successor series (say linux-6.2.y) or mainline (see next point) fix the bug.
- In all other cases, run:
```
cd ~/linux/
git switch --discard-changes --detach mainline/master
```
[details]

Build the image and the modules of your first kernel using the config file you prepared:
```
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
```
If you want your kernel packaged up as deb, rpm, or tar file, see the reference section for alternatives, which obviously will require other steps to install as well.

[details]

Install your newly built kernel.

Before doing so, consider checking if there is still enough space for it:
```
df -h /boot/ /lib/modules/
```
For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how much your kernels actually require will be determined later during this guide.

Now install the kernel’s modules and its image, which will be stored in parallel to the your Linux distribution’s kernels:
```
sudo make modules_install
command -v installkernel && sudo make install
```
The second command ideally will take care of three steps required at this point: copying the kernel’s image to /boot/, generating an initramfs, and adding an entry for both to the boot loader’s configuration.

Sadly some distributions (among them Arch Linux, its derivatives, and many immutable Linux distributions) will perform none or only some of those tasks. You therefore want to check if all of them were taken care of and manually perform those that were not. The reference section provides further details on that; your distribution’s documentation might help, too.

Once you figured out the steps needed at this point, consider writing them down: if you will build more kernels as described in segment 2 and 3, you will have to perform those again after executing command -v installkernel [...].

[details]

In case you plan to follow this guide further, check how much storage space the kernel, its modules, and other related files like the initramfs consume:
```
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
du -sh /lib/modules/$(make -s kernelrelease)/
```
Write down or remember those two values for later: they enable you to prevent running out of disk space accidentally during a bisection.

[details]

Show and store the kernelrelease identifier of the kernel you just built:
```
make -s kernelrelease | tee -a ~/kernels-built
```
Remember the identifier momentarily, as it will help you pick the right kernel from the boot menu upon restarting.
Reboot into your newly built kernel. To ensure your actually started the one you just built, you might want to verify if the output of these commands matches:
```
tail -n 1 ~/kernels-built
uname -r
```

Check if the kernel marked itself as ‘tainted’:
```
cat /proc/sys/kernel/tainted
```
If that command does not return ‘0’, check the reference section, as the cause for this might interfere with your testing.

[details]

Verify if your bug occurs with the newly built kernel. If it does not, check out the instructions in the reference section to ensure nothing went sideways during your tests.

[details]

Did you just built a stable or longterm kernel? And were you able to reproduce the regression with it? Then you should test the latest mainline codebase as well, because the result determines which developers the bug must be submitted to.

To prepare that test, check out current mainline:
```
cd ~/linux/
git switch --discard-changes --detach mainline/master
```
Now use the checked out code to build and install another kernel using the commands the earlier steps already described in more detail:
```
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot
```
Confirm you booted the kernel you intended to start and check its tainted status:
```
tail -n 1 ~/kernels-built
uname -r
cat /proc/sys/kernel/tainted
```
Now verify if this kernel is showing the problem. If it does, then you need to report the bug to the primary developers; if it does not, report it to the stable team. See Reporting issues for details.

[details]

Do you follow this guide to verify if a problem is present in the code currently supported by Linux kernel developers? Then you are done at this point. If you later want to remove the kernel you just built, check out Complementary tasks: cleanup during and after following this guide.

In case you face a regression, move on and execute at least the next segment as well.

Segment 2: check if the kernels you build work fine¶

In case of a regression, you now want to ensure the trimmed configuration file you created earlier works as expected; a bisection with the .config file otherwise would be a waste of time. [details]

Build your own variant of the ‘working’ kernel and check if the feature that regressed works as expected with it.

Start by checking out the sources for the version earlier established as ‘good’ (once again assumed to be 6.0 here):
```
cd ~/linux/
git switch --discard-changes --detach v6.0
```
Now use the checked out code to configure, build, and install another kernel using the commands the previous subsection explained in more detail:
```
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot
```
When the system booted, you may want to verify once again that the kernel you started is the one you just built:
```
tail -n 1 ~/kernels-built
uname -r
```
Now check if this kernel works as expected; if not, consult the reference section for further instructions.

[details]

Segment 3: perform the bisection and validate the result¶

With all the preparations and precaution builds taken care of, you are now ready to begin the bisection. This will make you build quite a few kernels -- usually about 15 in case you encountered a regression when updating to a newer series (say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build configuration created earlier this works a lot faster than many people assume: overall on average it will often just take about 10 to 15 minutes to compile each kernel on commodity x86 machines.

Start the bisection and tell Git about the versions earlier established as ‘good’ (6.0 in the following example command) and ‘bad’ (6.1.5):
```
cd ~/linux/
git bisect start
git bisect good v6.0
git bisect bad v6.1.5
```
[details]

Now use the code Git checked out to build, install, and boot a kernel using the commands introduced earlier:
```
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot
```
If compilation fails for some reason, run git bisect skip and restart executing the stack of commands from the beginning.

In case you skipped the ‘test latest codebase’ step in the guide, check its description as for why the ‘df [...]’ and ‘make -s kernelrelease [...]’ commands are here.

Important note: the latter command from this point on will print release identifiers that might look odd or wrong to you -- which they are not, as it’s totally normal to see release identifiers like ‘6.0-rc1-local-gcafec0cacaca0’ if you bisect between versions 6.1 and 6.2 for example.

[details]

Now check if the feature that regressed works in the kernel you just built.

You again might want to start by making sure the kernel you booted is the one you just built:
```
cd ~/linux/
tail -n 1 ~/kernels-built
uname -r
```
Now verify if the feature that regressed works at this kernel bisection point. If it does, run this:
```
git bisect good
```
If it does not, run this:
```
git bisect bad
```
Be sure about what you tell Git, as getting this wrong just once will send the rest of the bisection totally off course.

While the bisection is ongoing, Git will use the information you provided to find and check out another bisection point for you to test. While doing so, it will print something like ‘Bisecting: 675 revisions left to test after this (roughly 10 steps)’ to indicate how many further changes it expects to be tested. Now build and install another kernel using the instructions from the previous step; afterwards follow the instructions in this step again.

Repeat this again and again until you finish the bisection -- that’s the case when Git after tagging a change as ‘good’ or ‘bad’ prints something like ‘cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit’; right afterwards it will show some details about the culprit including the patch description of the change. The latter might fill your terminal screen, so you might need to scroll up to see the message mentioning the culprit; alternatively, run git bisect log > ~/bisection-log.

[details]

Store Git’s bisection log and the current .config file in a safe place before telling Git to reset the sources to the state before the bisection:
```
cd ~/linux/
git bisect log > ~/bisection-log
cp .config ~/bisection-config-culprit
git bisect reset
```
[details]

Try reverting the culprit on top of latest mainline to see if this fixes your regression.

This is optional, as it might be impossible or hard to realize. The former is the case, if the bisection determined a merge commit as the culprit; the latter happens if other changes depend on the culprit. But if the revert succeeds, it is worth building another kernel, as it validates the result of a bisection, which can easily deroute; it furthermore will let kernel developers know, if they can resolve the regression with a quick revert.

Begin by checking out the latest codebase depending on the range you bisected:
- Did you face a regression within a stable/longterm series (say between 6.0.13 and 6.0.15) that does not happen in mainline? Then check out the latest codebase for the affected series like this:
```
git fetch stable
git switch --discard-changes --detach linux-6.0.y
```
- In all other cases check out latest mainline:
```
git fetch mainline
git switch --discard-changes --detach mainline/master
```
  If you bisected a regression within a stable/longterm series that also happens in mainline, there is one more thing to do: look up the mainline commit-id. To do so, use a command like git show abcdcafecabcd to view the patch description of the culprit. There will be a line near the top which looks like ‘commit cafec0cacaca0 upstream.’ or ‘Upstream commit cafec0cacaca0’; use that commit-id in the next command and not the one the bisection blamed.
Now try reverting the culprit by specifying its commit id:
```
git revert --no-edit cafec0cacaca0
```
If that fails, give up trying and move on to the next step; if it works, adjust the tag to facilitate the identification and prevent accidentally overwriting another kernel:
```
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
```
Build a kernel using the familiar command sequence, just without copying the the base .config over:
```
make olddefconfig &&
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot
```
Now check one last time if the feature that made you perform a bisection works with that kernel: if everything went well, it should not show the regression.

[details]

Complementary tasks: cleanup during and after the bisection¶

During and after following this guide you might want or need to remove some of the kernels you installed: the boot menu otherwise will become confusing or space might run out.

To remove one of the kernels you installed, look up its ‘kernelrelease’ identifier. This guide stores them in ‘~/kernels-built’, but the following command will print them as well:
```
ls -ltr /lib/modules/*-local*
```
You in most situations want to remove the oldest kernels built during the actual bisection (e.g. segment 3 of this guide). The two ones you created beforehand (e.g. to test the latest codebase and the version considered ‘good’) might become handy to verify something later -- thus better keep them around, unless you are really short on storage space.

To remove the modules of a kernel with the kernelrelease identifier ‘6.0-rc1-local-gcafec0cacaca0’, start by removing the directory holding its modules:
```
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
```
Afterwards try the following command:
```
sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
```
On quite a few distributions this will delete all other kernel files installed while also removing the kernel’s entry from the boot menu. But on some distributions kernel-install does not exist or leaves boot-loader entries or kernel image and related files behind; in that case remove them as described in the reference section.

[details]

Once you have finished the bisection, do not immediately remove anything you set up, as you might need a few things again. What is safe to remove depends on the outcome of the bisection:
- Could you initially reproduce the regression with the latest codebase and after the bisection were able to fix the problem by reverting the culprit on top of the latest codebase? Then you want to keep those two kernels around for a while, but safely remove all others with a ‘-local’ in the release identifier.
- Did the bisection end on a merge-commit or seems questionable for other reasons? Then you want to keep as many kernels as possible around for a few days: it’s pretty likely that you will be asked to recheck something.
- In other cases it likely is a good idea to keep the following kernels around for some time: the one built from the latest codebase, the one created from the version considered ‘good’, and the last three or four you compiled during the actual bisection process.
[details]

Optional: test reverts, patches, or later versions¶

While or after reporting a bug, you might want or potentially will be asked to test reverts, debug patches, proposed fixes, or other versions. In that case follow these instructions.

Update your Git clone and check out the latest code.
- In case you want to test mainline, fetch its latest changes before checking its code out:
```
git fetch mainline
git switch --discard-changes --detach mainline/master
```
- In case you want to test a stable or longterm kernel, first add the branch holding the series you are interested in (6.2 in the example), unless you already did so earlier:
```
git remote set-branches --add stable linux-6.2.y
```
  Then fetch the latest changes and check out the latest version from the series:
```
git fetch stable
git switch --discard-changes --detach stable/linux-6.2.y
```
Copy your kernel build configuration over:
```
cp ~/kernel-config-working .config
```
Your next step depends on what you want to do:
- In case you just want to test the latest codebase, head to the next step, you are already all set.
- In case you want to test if a revert fixes an issue, revert one or multiple changes by specifying their commit ids:
```
git revert --no-edit cafec0cacaca0
```
  Now give that kernel a special tag to facilitates its identification and prevent accidentally overwriting another kernel:
```
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
```
- In case you want to test a patch, store the patch in a file like ‘/tmp/foobars-proposed-fix-v1.patch’ and apply it like this:
```
git apply /tmp/foobars-proposed-fix-v1.patch
```
  In case of multiple patches, repeat this step with the others.
  
  Now give that kernel a special tag to facilitates its identification and prevent accidentally overwriting another kernel:
```
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
```

Build a kernel using the familiar commands, just without copying the kernel build configuration over, as that has been taken care of already:

make olddefconfig &&
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot

Now verify you booted the newly built kernel and check it.

[details]

Conclusion¶

You have reached the end of the step-by-step guide.

Did you run into trouble following any of the above steps not cleared up by the reference section below? Did you spot errors? Or do you have ideas how to improve the guide?

If any of that applies, please take a moment and let the maintainer of this document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is vital to improve this text further, which is in everybody’s interest, as it will enable more people to master the task described here -- and hopefully also improve similar guides inspired by this one.

Reference section for the step-by-step guide¶

This section holds additional information for almost all the items in the above step-by-step guide.

Preparations for building your own kernels¶

The steps in this section lay the groundwork for all further tests. [...]

The steps in all later sections of this guide depend on those described here.

[back to step-by-step guide].

Prepare for emergencies¶

Create a fresh backup and put system repair and restore tools at hand. [...]

Remember, you are dealing with computers, which sometimes do unexpected things -- especially if you fiddle with crucial parts like the kernel of an operating system. That’s what you are about to do in this process. Hence, better prepare for something going sideways, even if that should not happen.

How to verify bugs and bisect regressions¶

The essence of the process (aka ‘TL;DR’)¶

Step-by-step guide on how to verify bugs and bisect regressions¶

Preparations: set up everything to build your own kernels¶

Segment 1: try to reproduce the problem with the latest codebase¶

Segment 2: check if the kernels you build work fine¶

Segment 3: perform the bisection and validate the result¶

Complementary tasks: cleanup during and after the bisection¶

Optional: test reverts, patches, or later versions¶

Conclusion¶

Reference section for the step-by-step guide¶

Preparations for building your own kernels¶

Prepare for emergencies¶

Remove anything related to externally maintained kernel modules¶

Deal with techniques like Secure Boot¶

Boot the last kernel that was working¶

Space requirements¶

Bisection range¶

Install build requirements¶

Download the sources using Git¶

Downloading Linux mainline sources using a bundle¶

Downloading Linux mainline sources using a shallow clone¶

Start defining the build configuration for your kernel¶

Trim the build configuration for your kernel¶

Tag the kernels about to be build¶

Decide to enable or disable debug symbols¶

Adjust build configuration¶

Distro specific adjustments¶

Individual adjustments¶

Put the .config file aside¶

Try to reproduce the problem with the latest codebase¶

Check out the latest Linux codebase¶

Build your kernel¶

Dealing with build errors¶

Package your kernel up¶

Put the kernel in place¶

Storage requirements per kernel¶

Check if your newly built kernel considers itself ‘tainted’¶

Check the kernel built from a recent mainline codebase¶

Check the kernel built from the latest stable/longterm codebase¶

Ensure the ‘good’ version is really working well¶

Build your own version of the ‘good’ kernel¶

Perform a bisection and validate the result¶

Start the bisection¶

Build a kernel from the bisection point¶

Bisection checkpoint¶

Put the bisection log away¶

Try reverting the culprit¶

Cleanup steps during and after following this guide¶

Cleaning up during the bisection¶

Cleaning up after the bisection¶

Test reverts, patches, or later versions¶

Additional information¶

Build kernels on a different machine¶

Additional reading material¶