GNU Guix: Building packages targeting psABIs
Starting with version 2.33, the GNU C library (glibc) grew the capability to
search for shared libraries using additional paths, based on the hardware
capabilities of the machine running the code. This was a great boon for
x86_64, which was first released in 2003, and has seen many changes in the
capabilities of the hardware since then. While it is extremely common for
Linux distributions to compile for a baseline which encompasses all of an
architecture, there is performance being left on the table by targeting such an
old specification and not one of the newer revisions.
One option used internally in glibc and in some other performance-critical
libraries is indirect functions, or
IFUNCs (see also
here)
The loader, ld.so uses them to pick function implementations optimized for
the available CPU at load time. GCC's (functional multi-versioning
(FMV))[https://gcc.gnu.org/wiki/FunctionMultiVersioning] generates several
optimized versions of functions, using the IFUNC mechanism so the approprate
one is selected at load time. These are strategies which most
performance-sensitive libraries do, but not all of them.
With the --tune using package
transformation
option, Guix implements so-called package
multi-versioning,
which creates package variants using compiler flags set to use optimizations
targeted for a specific CPU.
Finally - and we're getting to the central topic of this post! - glibc since
version 2.33 supports another approach: ld.so would search not just the
/lib folder, but also the glibc-hwcaps folders, which for x86_64 included
/lib/glibc-hwcaps/x86-64-v2, /lib/glibc-hwcaps/x86-64-v3 and
/lib/glibc-hwcaps/x86-64-v4, corresponding to the psABI micro-architectures
of the x86_64 architecture. This means that if a library was compiled against
the baseline of the architecture then it should be installed in /lib, but if
it were compiled a second time, this time using (depending on the build
instructions) -march=x86-64-v2, then the libraries could be installed in
/lib/glibc-hwcaps/x86-64-v2 and then glibc, using ld.so, would choose the
correct library at runtime.
These micro-architectures aren't a perfect match for the different hardware
available, it is often the case that a particular CPU would satisfy the
requirements of one tier and part of the next but would therefore only be able
to use the optimizations provided by the first tier and not by the added
features that the CPU also supports.
This of course shouldn't be a problem in Guix; it's possible, and even
encouraged, to adjust packages to be more useful for one's needs. The problem
comes from the search paths: ld.so will only search for the glibc-hwcaps
directory if it has already found the base library in the preceding /lib
directory. This isn't a problem for distributions following the File System
Hierarchy (FHS), but for Guix we will need to ensure that all the different
versions of the library will be in the same output.
With a little bit of planning this turns out to not be as hard as it sounds.
Lets take for example, the GNU Scientific
Library, gsl, a math library which helps
with all sorts of numerical analysis. First we create a procedure to generate
our 3 additional packages, corresponding to the psABIs that are searched for in
the glibc-hwcaps directory.
(define (gsl-hwabi psabi)
(package/inherit gsl
(name (string-append "gsl-" psabi))
(arguments
(substitute-keyword-arguments (package-arguments gsl)
((#:make-flags flags #~'())
#~(append (list (string-append "CFLAGS=-march=" #$psabi)
(string-append "CXXFLAGS=-march=" #$psabi))
#$flags))
((#:configure-flags flags #~'())
#~(append (list (string-append "--libdir=" #$output
"/lib/glibc-hwcaps/" #$psabi))
#$flags))
;; The building machine can't necessarily run the code produced.
((#:tests? _ #t) #f)
((#:phases phases #~%standard-phases)
#~(modify-phases #$phases
(add-after 'install 'remove-extra-files
(lambda _
(for-each (lambda (dir)
(delete-file-recursively (string-append #$output dir)))
(list (string-append "/lib/glibc-hwcaps/" #$psabi "/pkgconfig")
"/bin" "/include" "/share"))))))))
(supported-systems '("x86_64-linux" "powerpc64le-linux"))
(properties `((hidden? . #t)
(tunable? . #f)))))We remove some directories and any binaries since we only want the libraries
produced from the package; we want to use the headers and any other bits from
the main package. We then combine all of the pieces together to produce a
package which can take advantage of the hardware on which it is run:
(define-public gsl-hwcaps
(package/inherit gsl
(name "gsl-hwcaps")
(arguments
(substitute-keyword-arguments (package-arguments gsl)
((#:phases phases #~%standard-phases)
#~(modify-phases #$phases
(add-after 'install 'install-optimized-libraries
(lambda* (#:key inputs outputs #:allow-other-keys)
(let ((hwcaps "/lib/glibc-hwcaps/"))
(for-each
(lambda (psabi)
(copy-recursively
(string-append (assoc-ref inputs (string-append "gsl-" psabi))
hwcaps psabi)
(string-append #$output hwcaps psabi))
'("x86-64-v2" "x86-64-v3" "x86-64-v4"))))))))
(native-inputs
(modify-inputs (package-native-inputs gsl)
(append (gsl-hwabi "x86-64-v2")
(gsl-hwabi "x86-64-v3")
(gsl-hwabi "x86-64-v4"))))
(supported-systems '("x86_64-linux"))
(properties `((tunable? . #f)))))In this case the size of the final package is increased by about 13 MiB, from
5.5 MiB to 18 MiB. It is up to you if the speed-up from providing an optimized
library is worth the size trade-off.
To use this package as a replacement build input in a package
package-input-rewriting/spec is a handy tool:
(define use-glibc-hwcaps
(package-input-rewriting/spec
;; Replace some packages with ones built targeting custom packages build
;; with glibc-hwcaps support.
`(("gsl" . ,(const gsl-hwcaps)))))
(define-public inkscape-with-hwcaps
(package
(inherit (use-glibc-hwcaps inkscape))
(name "inkscape-with-hwcaps")))Of the Guix supported architectures, x86_64-linux and powerpc64le-linux can
both benefit from this new capability.
Through the magic of newer versions of GCC and LLVM it is safe to use these
libraries in place of the standard libraries while compiling packages; these
compilers know about the glibc-hwcap directories and will purposefully link
against the base library during build time, with glibc's ld.so choosing the
optimized library at runtime.
One possible use case for these libraries is crating guix packs
of packages to run on other systems. By substituting these libraries it
becomes possible to crate a guix pack which will have better performance than
a standard package used in a guix pack. This works even when the included
libraries don't make use of the IFUNCs from glibc or functional
multi-versioning from GCC. Providing optimized yet portable pre-compiled
binaries is a great way to take advantage of this feature.
About GNU Guix
GNU Guix is a transactional package manager
and an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
Guile programming interfaces and
extensions to the Scheme language.
Source: Planet GNU