GNU Guix: A build daemon in Guile

When using Guix, you might be aware of the
daemon.
It runs in the background but it's a key component in Guix. Whenever
you've been using Guix to operate on the store,
whether that's building something or downloading some substitutes,
it's the daemon managing that operation.

The daemon also is a key part of the history of Guix. The Guix
project started mixing Guile with ideas from the Nix project, and the
guix-daemon is a fork of the nix-daemon with some tweaks made over
the years. Rather than being implemented in Guile though, the daemon
is implemented in C++ with some helpers written in Guile. Given the
focus on Guile in Guix, this is unusual, and I believe it's made
working on the daemon less desirable, especially since rewriting it in
Guile has been discussed for many years now. It has been the topic of
a Google Summer of Code internship by Caleb Ristvedt back in 2017,
which helped clarify implementation details and led to some
preliminary code.

What would a build daemon in Guile bring?

Guix already has code written in Guile for doing some of what the
daemon does internally, so being able to use this Guile code inside
and outside the daemon would simplify Guix and allow removing the C++
code.

There isn't Guile code yet for everything the daemon does though, so
getting to this point will make new exciting features easier to
implement. That could be things like making it easier to use Guix in
environments where running the daemon in the usual way is inconvenient
or infeasible. It may also help with portability, so help with
running Guix on the Hurd and new architectures.

As someone who's more experienced writing Guile than C++, I'm also
hoping it'll generally make hacking on the daemon more accessible.
This in turn might lead to new features. For example, I think having
a build daemon written in Guile will simplify implementing a way to
jump in to a build and inspect the environment.

With that in mind, I'm excited to announce that support from
NLNet, will allow me to
focus for the next year on getting a Guile implementation of the build
daemon written and adopted.

A technical plan

Building on the recent discussion
of this topic on the guix-devel@gnu.org mailing list, here's some
technical thoughts on how I'm approaching this.

While I think there's a substantial amount of work to do, progress
towards a Guile guix-daemon has already been made. Given that things
in Guix have probably changed since this work has happened, I plan to
carefully review that existing work (most of which can be found on the
guile-daemon branch

The priority for the Guile daemon is backwards compatibility, so the
plan is to allow switching between the C++ implementation and Guile
implementation which any issues. This'll require not making changes
to the database schema, and generally doing things in a way which the
current C++ daemon will understand.

Like the Guix Build Coordinator,
I'm planning to make the daemon a single process using
Fibers for concurrency. This is in
contrast to the forking model using by the C++ daemon. Even though
it's not a priority to address feature issues with the current daemon,
this approach might help to reduce database contention issues
experienced with the current daemon, and allow for less locking, like
not having the big GC lock for example.

I'm planning on publishing more blog posts as the project progress, so
keep an eye on the Guix blog for future updates.

Acknowledgments

Thanks to Simon Tournier and Ludovic Courtès for providing feedback on
an earlier draft of this post.

About GNU Guix

GNU Guix is a transactional package manager
and an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
Guile programming interfaces and
extensions to the Scheme language.

Source: Planet GNU