We know that live patching has real benefits because it significantly reduces the downtime associated with frequent patching. But live patching is relatively difficult to achieve without causing other problems and for that reason live patching is not implemented as frequently as it could be. After all, the last thing sysadmins want is a live patch that crashes a system.

Reproducible builds are one of the tools that can help developers to implement live patching consistently and safely. In this article, I explain why reproducible builds matter for live patching, what exactly reproducible builds are, and how containers are coming to the rescue.

Live patching: a key threat management tool

Patching is a critical part of systems maintenance because patching fixes faulty and buggy code. More importantly, security teams rely on patching to plug security holes, and there is a real urgency to it. Waiting for a convenient maintenance window to patch is risky because it leaves an opportunity for hackers to take advantage of an exploit.

It creates a difficult conundrum: maintain high availability but run a security risk, or patch frequently but end up with frustrated stakeholders. Live patching bridges that gap. With live patching, the offending code is swapped out while a process is actively running, without restarting the application or service that depends on that process.

Implementing live patching isn’t easy

Live patching is not that straightforward to accomplish – the drop-in code must “fit” in a like-for-like manner, or all sorts of unwanted things can happen. Get it wrong, and the application – or entire server – will crash.

The code behind a running process usually comes from a binary executable file – a machine-readable block of code compiled from source code. A kernel, for example, has thousands of source files all compiled into a few binaries.

With live patching, the live patch code must fit in at an exact level. Yes, the binary file containing the patch code will be different from the binary file containing the bad code. Nonetheless, the new code must slot into place precisely and must depend on the same version of imported libraries. The live patch code must also be compiled using the same compiler options and flags. Bit endianness matters too – the binary file must be ordered in exactly the same way.

In principle, all this is achievable – but in practice, it is a challenge. For example, day-to-day system updates often impact libraries. These libraries could be slightly different, in turn producing binaries that are slightly different when compiling code.