Ksplice was originally released in 2008 and remains the most comprehensive in-memory patching solution available. Ksplice can patch both the Linux kernel and critical user space libraries to keep systems operational and increase up-time and security. Ksplice supports between 3,000 and 5,000 kernel versions at any given time. Let's look behind the scenes to learn more about how Ksplice patches are created and the quality assurance that goes into making sure these patches work flawlessly on every customer system.
When a new errata becomes available for Oracle Linux, our automation system creates dedicated build and QA environments that will be used for building and testing the patch required to fix the errata.
First, a highly trained Ksplice kernel engineer needs to identify which patches need to be included. Ksplice patches are primarily done for CVEs and critical bug fixes rather than for feature enhancements. Not all patches for CVEs can be automatically be performed safely in-memory and many require modification before they can be live patched onto a system. Certain rare edge cases exist where hardware microcode is being patched or where the patch may interfere with in-memory data structures, so the code has to be evaluated and tested to ensure that in can be safely applied and removed if necessary. There’s no margin for error and Ksplice patches must work perfectly on all customer systems. Modifications are evaluated and the decision to make the patch available for Ksplice is made.
The patch is then built, packaged and the testing phase begins. Patches are tested against all previous kernels for each support Linux distribution in our automated test farm. Patches are extensively tested to determine if the patch is effective and consistent and does not produce any undesired results.
The patch then moves to code review which is performed by a different developer. If any issues are found, the code is refactored and tested again and another code review is performed. When the patch passes its final code review, it is ported to all supported versions of the kernel to which it applies and additional testing is done.
The patch is then released to the Ksplice servers and becomes immediately available to Ksplice running in online mode. The patch is also handed off to the ULN team for packaging and testing to publish the patch in RPM format for Ksplice running in offline mode.
Being able to patch the kernel and user space in-memory and being able to remove those patches and restore the system to its original state all without rebooting is no trivial matter. There are many factors that can go wrong during the process because of the complexity of the task. This insight to the Ksplice patch build process was provided to highlight that there’s a lot more to patching in-memory than having a patch and a mechanism for performing an in-memory update. There’s a human factor to the process which requires a lot of experience and judgement. In addition to the human factor, Oracle has a very carefully defined process with extensive testing to ensure predictable results every single time. This gives our customers the confidence to be able to patch in their production environments with no human interaction if they choose to.
There are newcomers to the market that offer in-memory patching and we encourage you to evaluate all options. We also recommend asking Oracle and any other vendor some important questions: can they patch both the kernel and user space? Are the patches completely removable? What does their QA process involve? What real-world product stacks are tested with each patch? Make sure the vendor you select has the experience to ensure that your systems' stability is never compromised while maintaining the highest possible level of security.