January 4, 2023 | by Arround The Web | No comments

GNU Guix: Dissecting Guix, Part 1: Derivations

To a new user, Guix's functional architecture can seem quite alien, and possibly
offputting. With a combination of extensive #guix-querying, determined
manual-reading, and plenty of source-perusing, they may eventually figure out
how everything fits together by themselves, but this can be frustrating and
often takes a fairly long time.

However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style implemented
by the likes of dpkg and
pacman.
This series of blog posts will cover basic Guix concepts, taking a "ground-up"
approach by dealing with lower-level concepts first, and hopefully make those
months of information-gathering unnecessary.

Before we dig in to Guix-specific concepts, we'll need to learn about one
inherited from Nix, the original functional package manager
and the inspiration for Guix; the idea of a
derivation
and its corresponding store items.

These concepts were originally described by Eelco Dolstra, the original author
of Nix, in their PhD thesis;
see ยง 2.1 The Nix store and ยง 2.4 Store Derivations.

Store Items

As you probably know, everything that Guix builds is stored in the store,
which is almost always the /gnu/store directory. It's the job of the
guix-daemon
to manage the store and build things. If you run
guix build PKG,
PKG will be built or downloaded from a substitute server if available, and a
path to an item in the store will be displayed.

$ guix build irssi
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3

This item contains the final result of building irssi.
Let's peek inside:

$ ls $(guix build irssi)
bin/  etc/  include/  lib/  share/
$ ls $(guix build irssi)/bin
irssi*

irssi is quite a simple package. What about a more complex one, like
glib?

$ guix build glib
/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static

glib produces five /gnu/store items, because it's possible for a package to
produce multiple outputs.
Each output can be referred to separately, by prefixing a package's name with
:OUTPUT where supported. For example, this
guix install
invocation will add glib's bin output to your profile:

$ guix install glib:bin

The default output is out, so when you pass glib by itself to that command,
it will actually install glib:out to the profile.

guix build also provides the --source flag, which produces the store item
corresponding to the given package's downloaded source code.

$ guix build --source irssi
/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
$ guix build --source glib
/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz

But how does Guix know how to build these store outputs in the first place?
That's where derivations come in.

`.drv` Files

You've probably seen these being printed by the Guix program now and again.
Derivations, represented in the daemon's eyes by .drv files, contain
instructions for building store items. We can retrieve the paths of these
.drv files with the guix build --derivations command:

$ guix build --derivations irssi
/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv

guix build can actually also accept derivation paths as an argument, in lieu
of a package, like so:

$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3

Let's look inside this derivation file.

Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])

It's... not exactly human-readable. We could try to format it and break it
down, but it'd still be pretty hard to understand, since .drv files contain no
labels for the fields or any other human-readable indicators. Instead, we're
going to explore derivations in a Guile REPL.

Exploring Guix Interactively

Before we continue, we'll want to start a REPL, so that we can try out the Guix
Guile API interactively. To run a REPL in the terminal, simply
call guix repl.

If you're using Emacs, you can instead install
Geiser, which provides a comfortable Emacs UI for
various Lisp REPLs, invoke guix repl --listen=tcp:37146 &, and type
M-x geiser-connect RET RET RET to connect to the running Guile instance.

Your .guile file may contain code for enabling colours and readline bindings
that Geiser will choke on. The default Guix System .guile contains code to
suppress these features when INSIDE_EMACS is set, so you'll need to run
guix repl like this:

INSIDE_EMACS=1 guix repl --listen=tcp:37146 &

There are a few Guix modules we'll need. Run this Scheme code to import them:

(use-modules (guix)
             (guix derivations)
             (guix gexp)
             (guix packages)
             (guix store)
             (gnu packages glib)
             (gnu packages irc))

We now have access to the store, G-expression, package, and derivation APIs,
along with the irssi and glib <package> objects.

Creating a `<derivation>`

The Guix API for derivations revolves around the <derivation> record, which is
the Scheme representation of that whole block of text surrounded by
Derive(...). If we look in guix/derivations.scm, we can see that it's
defined like this:

(define-immutable-record-type <derivation>
  (make-derivation outputs inputs sources system builder args env-vars
                   file-name)
  derivation?
  (outputs  derivation-outputs)      ; list of name/<derivation-output> pairs
  (inputs   derivation-inputs)       ; list of <derivation-input>
  (sources  derivation-sources)      ; list of store paths
  (system   derivation-system)       ; string
  (builder  derivation-builder)      ; store path
  (args     derivation-builder-arguments)         ; list of strings
  (env-vars derivation-builder-environment-vars)  ; list of name/value pairs
  (file-name derivation-file-name))               ; the .drv file name

With the exception of file-name, each of those fields corresponds to a field
in the Derive(...) form. Before we can examine them, though, we need to
figure out how to lower that irssi <package> object into a derivation.

guix repl provides the ,lower command to create derivations quickly,
as shown in this sample REPL session:

scheme@(guile-user)> ,use (guix)
scheme@(guile-user)> ,use (gnu packages irc)
scheme@(guile-user)> irssi
$1 = #<package irssi@1.4.3 gnu/packages/irc.scm:153 7f3ff98e0c60>
scheme@(guile-user)> ,lower irssi
$2 = #<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3ff7782d70>
;; Below we use the $N variable automatically bound by the REPL.
scheme@(guile-user)> (derivation-system $2)
$3 = "x86_64-linux"

Since ,lower is a REPL command, however, we can't use it in proper Scheme
code. It's quite useful for exploring specific derivations interactively, but
since the purpose of this blog post is to explain how things work inside, we're
going to use the pure-Scheme approach here.

The procedure we need to use to turn a high-level object like <package> into a
derivation is called lower-object; more on that in a future post. However,
this doesn't initially produce a derivation:

(pk (lower-object irssi))
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)

pk is an abbreviation for the procedure peek, which takes the given object,
writes a representation of it to the output, and returns it. It's especially
handy when you want to view an intermediate value in a complex expression.

The returned object is a monadic value (more on those in the next post on
monads) that needs to be evaluated in the context of a store connection. We do
this by first using with-store to connect to the store and bind the connection
to a name, then wrapping the lower-object call with run-with-store:

(define irssi-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object irssi)))))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)

(define glib-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object glib)))))
;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)

And we have liftoff! Now we've got two <derivation> records to play with.

Exploring `<derivation>`

`<derivation-output>`

The first "argument" in the .drv file is outputs, which tells the Guix
daemon about the outputs that this build can produce:

(define irssi-outputs
  (pk (derivation-outputs irssi-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))

(pk (assoc-ref irssi-outputs "out"))

(define glib-outputs
  (pk (derivation-outputs glib-drv)))
;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))

(pk (assoc-ref glib-outputs "bin"))
;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)

It's a simple association list mapping output names to <derivation-output>
records, and it's equivalent to the first "argument" in the .drv file:

[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
]

The hash-algo and hash fields are for storing the content hash and the
algorithm used with that hash for what we term a fixed-output derivation,
which is essentially a derivation where we know what the hash of the content
will be in advance. For instance, origins produce fixed-output derivations:

(define irssi-src-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object (package-source irssi))))))
;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)

(define irssi-src-outputs
  (pk (derivation-outputs irssi-src-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
  
(pk (assoc-ref irssi-src-outputs "out"))
;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)

Note how the hash and hash-algo now have values.

Perceptive readers may note that the <derivation-output> has four fields,
whereas the tuple in the .drv file only has three (minus the label). The
serialisation of recursive? is done by adding the prefix r: to the
hash-algo field, though its actual purpose is difficult to explain, and is out
of scope for this post.

`<derivation-input>`

The next field is inputs, which corresponds to the second field in the .drv
file format:

[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
  ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
  ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
  ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
  ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
  ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
  ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
  ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
]

Here, each tuple specifies a derivation that needs to be built before this
derivation can be built, and the outputs of the derivation that the build
process of this derivation uses. Let's grab us the Scheme equivalent:

(define irssi-inputs
  (pk (derivation-inputs irssi-drv)))
;;; [a fairly large amount of output]

(pk (car irssi-inputs))
;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)

Unlike derivation-outputs, derivation-inputs maps 1:1 to the .drv
form; the drv field is a <derivation> to be built, and the
sub-derivations field is a list of outputs.

Builder Configuration

The other fields are simpler; none of them involve new records. The third is
derivation-sources, which contains a list of all store items used in the build
which aren't themselves built using derivations, whereas derivation-inputs
contains the dependencies which are.

This list usually just contains the path to the Guile build script that
realises the store items when run, which we'll examine in a later post, and
the path to a directory containing extra modules to add to the build script's
%load-path, called /gnu/store/...-module-import.

The next field is derivation-system, which specifies the system type (such as
x86_64-linux) we're building for. Then we have derivation-builder, pointing
to the guile executable that runs the build script; and the second-to-last is
derivation-builder-arguments, which is a list of arguments to pass to
derivation-builder. Note how we use -L and -C to extend the Guile
%load-path and %load-compiled-path to include the module-import and
module-import-compiled directories:

(pk (derivation-system irssi-drv))
;;; ("x86_64-linux")

(pk (derivation-builder irrsi-drv))
;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")

(pk (derivation-builder-arguments irrsi-drv))
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))

The final field contains a list of environment variables to set before we start
the build process:

(pk (derivation-builder-environment-vars irssi-drv))
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))

The last record field, derivation-file-name contains the path to the .drv
file, and so isn't represented in a serialised derivation.

Utilising `<derivation>`

Speaking of serialisation, to convert between the .drv text format and the
Scheme <derivation> record, you can use write-derivation, read-derivation,
and read-derivation-from-file:

(define manual-drv
  (with-store %store
    (derivation %store "manual"
                "/bin/sh" '())))

(write-derivation drv (current-output-port))
;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])

(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)

(call-with-input-file (derivation-file-name irssi-drv)
  read-derivation)
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)

You can realise <derivation>s as store items using the build-derivations
procedure:

(use-modules (ice-9 ftw))

(define irssi-drv-out
  (pk (derivation-output-path
       (assoc-ref (derivation-outputs irssi-drv) "out"))))
;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")

(pk (scandir irssi-drv-out))
;;; (#f)

(pk (with-store %store (build-derivations %store (list irssi-drv))))
;;; (#t)

(pk (scandir irssi-drv-out))
;;; (("." ".." "bin" "etc" "include" "lib" "share"))

Conclusion

Derivations are one of Guix's most important concepts, but are fairly easy to
understand once you get past the obtuse .drv file format. They provide the
Guix daemon with the initial instructions that it uses to build store items
like packages, origins, and other file-likes such as computed-file and
local-file, which will be discussed in a future post!

To recap, a derivation contains the following fields:

derivation-outputs, describing the various output paths that the derivation
builds
derivation-inputs, describing the other derivations that need to be built
before this one is
derivation-sources, listing the non-derivation store items that the
derivation depends on
derivation-system, specifying the system type a derivation will be compiled
for
derivation-builder, the executable to run the build script with
derivation-builder-arguments, arguments to pass to the builder
derivation-builder-environment-vars, variables to set in the builder's
environment

About GNU Guix

GNU Guix is a transactional package manager and
an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.

Source: Planet GNU

GNU Guix: Dissecting Guix, Part 1: Derivations

Store Items

.drv Files

Exploring Guix Interactively

Creating a <derivation>

Exploring <derivation>

<derivation-output>

<derivation-input>

Builder Configuration

Utilising <derivation>

Conclusion

About GNU Guix

Leave a Reply Cancel reply

`.drv` Files

Creating a `<derivation>`

Exploring `<derivation>`

`<derivation-output>`

`<derivation-input>`

Utilising `<derivation>`