frankie-tales

Creating your own Guix substitute server

2025-12-21T16:30:00Z

I lately dedicated some time to setting up my own substitute server for Guix on a foreign distribution. This post is about that experience, after verifying that such a process is currently quite underdocumented. A substitute server is clearly a required step in order to cultivate a personal or unofficial/alternative channel for Guix, at least if one has more than one box (and possibly one physical location) to manage.

For people not up to date with the news from these digital bits, in the last year or so, I elected the Guix package manager and system as my second-preferred OS, after Debian GNU/Linux. I published more posts about it, all readable under my dedicated feed.

What is a Substitute Server?

Guix is mainly a source-level roll-on distribution, so at every pull, there is a concrete risk of having to build your own binaries if no binary provider is available to speed up your update process. That is a certainty if you play with your own packages. This is the concrete reason to possibly have a substitute server in the LAN, even though the official ones can be subject to problems and DDoS. Specifically, I set up a substitute server based on Debian GNU/Linux reachable both at home and at work. While the Guix manual is currently quite verbose about how to use a substitute server from a client, it is bit incomplete about the server side.

Installation of Guix on Debian

The very first step is to install the Guix package manager on Debian, which is currently not more possible using a Debian package from the main archive, as in the past (see this LWN post about). Currently, a binary provided by the Guix project has to be used for that. It is quite straightforward if you use the installer script provided with some caution.

For instance, to install the current Guix version, it is possible to run:

$ wget -O /tmp/guix-binary.tar.xz \
    https://ci.guix.gnu.org/search/latest/archive?query=spec:tarball+status:success+system:x86_64-linux+guix-binary.tar.xz
$ wget -O /tmp/guix-install.sh https://guix.gnu.org/install.sh

# do your homework to check and verify binary and script, then...
$ sudo -i

# export GUIX_BINARY_FILE_NAME=/tmp/guix-binary.tar.xz
# chmod a+x /tmp/guix-install.sh
# apt install uidmap gnupg
# /tmp/guix-install.sh
# rm -f /tmp/guix-install.sh /tmp/guix-binary.tar.xz

That will install a daemon for building sources under an unprivileged guix-daemon user, which has full control over the guix data store at /gnu/store.

Daemon configuration and storage

Such a daemon is run under systemd control with a specific unit file. First, the Guix data store needs plenty of disk space and inodes; I suggest moving the /gnu/store tree under an XFS or even a BTRFS filesystem, for that.

Second, the current installer tries to mount /gnu/store in read-only mode, which can prevent running the guix command as an unprivileged user on foreign distros. This can be disabled if problems occur, as could happen under specific conditions. Once the guix-daemon service is stopped, the gnu-store.mount can be disabled by ensuring the /gnu/store directory is correctly assigned to the guix-daemon user:

$ ls -ld /gnu/store/
drwxrwxr-t 584 guix-daemon guix-daemon 12M 18 dic 07.31 /gnu/store/

The systemd unit file for the Guix daemon

This is the proposed unit file for the guix-daemon process:

[Unit]
Description=Build daemon for GNU Guix
# Start before 'gnu-store.mount' to get a writable view of the store.
Before=gnu-store.mount

[Service]
ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon \
  --discover=yes \
  --substitute-urls='https://bordeaux.guix.gnu.org https://ci.guix.gnu.org'
Environment='GUIX_STATE_DIRECTORY=/var/guix' 'GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8
# Run under a dedicated unprivileged user account.
User=guix-daemon
# Bind-mount the store read-write in a private namespace.
PrivateMounts=true
BindPaths=/gnu/store
MountFlags=private
Type=exec
AmbientCapabilities=CAP_CHOWN
StandardOutput=journal
StandardError=journal
OOMPolicy=continue
Restart=always
TasksMax=8192

[Install]
WantedBy=multi-user.target

Note that, when building from sources in specific cases, it could be a good idea to provide an additional TMPDIR environment variable to bypass the limited size of the /tmp directory (specifically for tmpfs implementation as for Debian 13+). For instance, building a full Linux kernel could require more than 32 GB of storage available in /tmp, which corresponds by default to at least 64 GB of RAM on the system. A typical personal box can be quite limited for that. So adding TMPDIR=/var/tmp to the Environment variable could be a sane idea.

Setting up Guix Publish

To set up the substitute server, a guix publish process must be started via systemd. First, create a publishing asymmetric key pair and a destination for the cache directory:

$ sudo -i guix archive --generate-key
$ sudo mkdir -p /var/cache/guix/publish
$ sudo chown guix-daemon:guix-daemon /var/cache/guix/publish

This is the systemd unit file for the guix-publish.service:

[Unit]
Description=Guix substitute server
After=network.target

[Service]
Type=simple
User=root
Environment='GUIX_LOCPATH=/root/.guix-profile/lib/locale'
ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix publish \
  --listen=0.0.0.0 \
  --port=8080 \
  --user=guix-daemon \
  --compression=gzip:9 \
  --compression=lzip:9 \
  --cache=/var/cache/guix/publish \
  --ttl=30d --negative-ttl=1h \
  --workers=4
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This will start the service in unprivileged mode on an unprivileged port, which can be easily routed via an NGINX proxy, for instance (my preferred way). I use this system to route the service on both the home and the at-work LANs.

Once done, of course, the next point is defining a policy to populate such a server. While it could be nice running a proper cuirass build service to run continuously building from a custom repository, it is also perfectly fine simply using a manifest file with common packages or the multiple configurations used by personal boxes. Here is a series of alternatives to build a handful of packages.

guix build gcc
guix system build my-laptop-config.scm
guix build -m manifest.scm
guix build make cmake vim emacs perl

A simple manifest example:

(specifications->manifest
 '("gcc-toolchain"
   "python"
   "emacs"
   "git"
   "make"
   "cmake"
   "perl"
   "grass"
   "gdal"
   "geos"
   "gawk"
   "vim"
   ;; Add more packages...
))

Of course, a very simple strategy could consist of running regular pulls & builds for all packages of primary interest and making frequent enough purges of the data store to avoid excessive storage consumption. That can be avoided if plenty of space is available for builds.

Once setup the server and populating it, it is possibile to add its URL to the configurations of all personal/LAN boxes for foreign and native installations, and live peacefully.

Happy guixing, folks!

About computing environments for reproducible science

2025-12-09T13:00:00Z

A few weeks ago I gave a lecture for the Spatial Ecology course to introduce a handful of junior and not-so-junior researchers from various domains to the not-so-nice world of scientific computing environments.

For people interested, here are my slides about this topic. They are somehow specialized for the Python ecosystem (which has become nowadays the programming language adopted for scientific computing in multiple contexts), where, in the last few years, a lot of evolutions have taken place for the management of dependencies and the management of the computing environment. This problem is amplified in the HPC context (I already wrote a semi-serious post about such an argument).

I also cited guix without more details (it was impossible to deal with all sub-topics in the lecture, and I know that multiple listeners already had problems fully understanding the matter).

Reasoning about that, it is not a silly idea to write some blog notes about the whole topic. First of all, what is the context? Reproducible science is not a novel matter. Any scientific experiment should be reproducible, starting from the same data and giving comparable results: this is the basis of the scientific method (Galileo docet). In the context of scientific computing, that implies that the whole execution environment should be fully reproducible in order to ensure the possibility of replication of the executions, with the same outputs starting from the same inputs. Possibly later on, running on the same platform, or after deployment on a new, completely different system.

The key point is that the long-term reproducibility of such results on current platforms and with current languages is minimal, to be generous. Having the full source code of a Python notebook, a git source repository, or anything comparable is simply only the starting point. The sad reality is that in practice, the source code has, in too many cases, a lifetime of a few months because of the understimation of such a problem by the average scholar. When following a few good practices, such a lifetime can be extended to a few years, maybe.

When I wrote my thesis, too many years ago, I developed the whole C source for execution on a parallel computer of the time. It was a Meiko Computing Surface, a SIMD platform based on INMOS Transputers. The C code used a proprietary message-passing library, CSTools, to enable communication among T-800 processors (unfortunately, there is no relation with the Terminator series, sorry). Now, it is somewhat expected that a code based on a dead proprietary library running on a dead hardware platform could have reproducibility issues today, after more than 30 years.

What is unexpected is that one could have the same reproducibility problems after 30 months, or in some limited cases, after 30 days. I mean both at the binary and source levels, often. Now, part of the problem is due to the FAFO attitude of some development communities. Not all teams are like the GDAL one, which is capable of maintaining the same well-refined APIs for dozens of years. More usually, from time to time, new versions of libraries and tools introduce expected or unexpected breakages against past versions and APIs, which backfire on programs that use them. In other cases, new versions can fix and/or introduce bugs of primary interest for dependent software. Those are the main reasons to meticulously annotate and document every single version of direct and indirect dependencies. This is somewhat solved by dependency resolvers, as explained in my lecture. But that's only part of the whole chain.

Unfortunately, nowadays this chain of dependencies traverses a single language and crashes against system-level dependencies, including the whole operating system, with its system compilers, interpreters, and libraries. This problem is amplified in a fully containerized world, which is nowadays used intensively. Depending on a third-party-provided binary image taken from any hub out there is not a safer approach. Such images can disappear from night to day or have a limited lifetime, so the conscious scholar should also develop his/her own from scratch, which often is particularly out of the skill perimeter of the average scholar.

This is exactly where Guix tries to give an answer. Guix is a source-level package manager with a set of full descriptions written in Guile Scheme for the whole chain of dependencies up to the kernel level. Combining such an analytical description of the system for any built artifact in the timeline from the starting point (derivations), along with the possibility to use build systems to cache binary artifacts (substitutes), and install any software at the user level, does allow the creation of a source-level definition of a full execution environment.

Such an ambitious goal is not without problems, as magistrally summarized by Ludovic Courtès here, but anyway it is at light-years of distance from the possibility of the average deployment system that needs instead continuous babysitting in order to ensure a working environment.

What is probably of interest for general consumption would also be a consistent additional security tagging for derivations in order to fast identify sources with known CVE-tagged versions in the chain of dependencies. That would increase the level of self-awareness when the Guix time machine is used to go back in the past and pick some sources from Pandora's box. It would also be of considerable interest in the SBOM context outside the perimeter of science computing.

So, guix is not perfect, but again a sure advancement towards reproducible computing environments, which currently lack in a way or another in the science domain (and not only that).

References

[1] A guide to reproducible research papers

[2] Guix as a tool for reproducible science

[3] Using Guix for managing reproducible, flexible, and collaborative environments in a PhD thesis

[4] Reproducible genomics analysis pipelines with GNU Guix

[5] Replication crisis

Guix for geeks

2025-09-03T19:40:00Z

In the last few months, I have installed and upgraded my second preferred GNU/Linux system, GNU Guix, on multiple boxes. Regarding that system, I have already written a few introductory posts in the recent past. This is an update about my experiences as a user and developer. I still think Guix is a giant step forward in packaging and management, in comparison with Debian and other distributions, for elegance and inner coherence.

On the negative side, I can confirm that the most important aspects to consider in order to adopt Guix for daily use are as follows.

Guix is essentially a roll-on type of distribution with a limited user base, so a grain of salt is required at upgrade time because instabilities are somehow expected. The integrated time machine can be used to step back in case of problems, anyway.
The development team and other support teams are tiny enough that I would avoid using Guix for publicly exposed services. Also, it can lag on specific applications and issues. Tasks and goals do not specifically structure teams, so there is no proper security team, but I guess all maintainers can participate to cover all required tasks.
Packages are not always up-to-date with the latest upstream versions. Some packages are even older than in Debian stable, so if you are looking for the latest and coolest products, that's not the place.
Some choices for the core distribution are not mainstream, such as using shepherd as its init system instead of systemd. This implies some delays because patches are sometimes due for complex software, such as Gnome, which introduced strict dependencies on systemd in recent years. This is the reason why, as of the current date, Gnome is still at version 46, while Debian stable is at version 48.
The geospatial tools and libraries available are still quite limited, which may require a more in-depth analysis to estimate the level of lag. See the section below.
The readiness for immediate use is not comparable to the main distributions. For instance, if one absolutely needs a non-free Linux kernel due to firmware constraints, the nonguix add-on repository is available; however, one must build and install an ad hoc ISO installer by hand. Nothing transcendental, but complicated enough to discourage most users. Here is my fork of the System Crafters repository to prepare such an ISO image, updated for use of the current main Guix (now hosted on Codeberg) and NonGuix channels. That could be easily prepared on any foreign distribution that includes the guix package management software, such as Debian.

As of the current date, I have installed at least a few laptops and VMs, including a Dell Inspiron 15, an Asus EEE 1215P, and an old Acer Travelmate. I strongly suggest using at least a box with 8GB of RAM and a 512GB SSD, because building from sources can be overwhelming. My dual-core Atom EEE Pc has only 2 GB of RAM, and it is slow enough, but it has the big advantage of being fully supported by the official GNU Linux libre kernel.

Additionally, I believe that having a local build host for substitutions within the home/work LAN is the most efficient solution. That is because the officially provided worldwide substitution hosts could be generally heavily loaded and occasionally fail (when that happens, the time of recovery is on a best effort basis).

Of course, it is also entirely possible to use Guix only as a package management system on a foreign distribution, having most of the advantages of a reproducible environment, as well as within a container or a virtual machine. In that case, one has to consider that the host system is typically based on the systemd init system, which can cause a few headaches when porting existing package descriptions for services taken from the Guix repository. Probably the best compromise is still using a foreign distribution on the physical box and a Guix-based container to prepare an execution environment, something I already experimented as explained in previous posts.

Guix and geospatial software

In 2004, Paolo Cavallini and I started a subproject within the Debian community of developers and users to improve the status of geospatial software in Debian. That was the birth of what is today the DebianGIS pure blend, with a mildly coordinated team of developers and maintainers that work together in Debian (and derivative distributions) on a set of essential libraries and tools for the geospatial community. For people interested in the history of FOSS, I wrote a contribution to the ASITA conference in 2009 about that. That was and still is grunt work, often neglected and at the time perceived as a secondary effort by upstream developers. Indeed, that is an essential task instead, because collecting together properly and ensuring that you have a complete and well-built stack of software for a geospatial application is not trivial at all.

One of such base libraries is GDAL, which can be built with multiple optional dependencies and plugins. Like other platforms, some of those optional dependencies are missing in the Guix package, and that should be taken into consideration. Some tools are instead totally missing, such as MapServer and others. Of course, some packages could be patched here and there in Debian, or can require a Guix-specific patching (because of the peculiarities of such a system, which does not respect the Linux FHS), and that definitely could be a reason to have a package in good shape or not. That's to say that the Guix ecosystem would need a serious help for packaging such niche packages, but where most people can see a lack, I see an opportunity in that regard. After all, when I started my life in the Debian project, it was because I did not trust an operating environment that I could not develop and adjust personally for whatever reason. That's the way and Guix seems a brilliant example of a vibrant community for that.

Playing with Vagrant, Virtualbox and Guix

2024-09-04T18:20:00Z

It is specifically convenient using Guix-the-system within a foreign distribution, such as Debian, for development and tests. The package management system can be used on top of the system, but I find it quite interesting to explore the potential of the Guix distribution in the context of virtualized environments. For personal use, that is also the ideal way to avoid breaking your own daily boxes every couple of days with daredevil approaches to personal computing.

I find it quite handy to use Vagrant to synthetically describe multiple virtual hosts within a virtual network, by means of a single Vagrantfile. Nowadays, there are multiple virtualization systems usable for the same purpose, including the libvirt ecosystem. Under the hood, those systems can generally manage multiple virtual machine engines (or even cloud systems). Vagrant under Debian can use either Libvirt+KVM or Virtualbox (the default one used by the upstream Vagrant) as primary engines.

Users can develop their own Vagrant images (called boxes) or use those made available on the Hashicorp cloud, as developed by third-party teams (which is the common case).

Unfortunately, while there are a handful of Debian boxes developed by the Debian Cloud Team, like many other ones for multiple distributions and releases, this is not the case for Guix. At the time of this post, there is only a single box created some years ago by Tom Parker-Shemilt.

Of course, the Guix team also made a qcow2 image available to run a virtualized desktop machine under a QEMU/KVM system, as explained in the manual. What is described in this post can also be easily applied to create an alternative custom implementation with QEMU. What is required is to create a virtual disk image of some kind and use it to populate a custom distribution configuration via guix. The resulting image will be runnable under a suitable virtualization system in the host foreign distribution.

Let's assume to use any recent version of Vagrant and Virtualbox, both installed on the host system, along with the Guix package manager. The following are the steps required to create a Vagrant box with a minimal configuration of Guix that is compatible (with some limitations) with Vagrant and can be used to run any number of Guix VMs. Each Guix vm can be reconfigured for specific goals using multiple configurations.

In this case, we will not need to use Packer to prepare a box because the vagrant package command directly supports Virtualbox images with the same purpose.

Create an empty virtual disk image and partition it.
Prepare a suitable configuration written in Guile for Guix.
Run an initial setup of the mounted virtual disk using guix on your host.
Create a basic Virtualbox machine and attach the virtual disk to it.
Convert the machine into a Vagrant box and register it.
Prepare a Vagrantfile snippet and run the vm(s).
(Re)configure vm(s) on the basis of your needs.
Done!

Create a virtual disk image and partition it

This is the straightforward part. Here, I'm using the vdi format, the native Virtualbox one, but the VMware vmdk format can be used instead.

 # load the Network Block Device module
 sudo modprobe nbd
 # create a thin device
 qemu-img create -f vdi guix-hd.vdi 100G
 # ... mount it
 sudo qemu-nbd -f vdi --connect=/dev/nbd0 guix-hd.vdi
 # ... partition it 
 sudo parted /dev/nbd0 mklabel msdos
 sudo parted -a cylinder /dev/nbd0 mkpart primary ext4 1 93G
 sudo parted -a cylinder /dev/nbd0 mkpart primary linux-swap 93G 100%
 sudo parted /dev/nbd0 set 1 boot on 
 # ... create a suitable fs
 sudo mkfs.ext4 /dev/nbd0p1
 sudo mkswap /dev/nbd0p2
 # ... have a look to the partions IDs
 sudo blkid /dev/nbd0p1 
 sudo blkid /dev/nbd0p2
 # ... mount the root fs
 sudo mount /dev/nbd0p1 /mnt

Prepare a suitable configuration written in Guile for Guix

Any valid configuration can be prepared, in this case this is a minimal one with a pair of changes for Vagrant. Specifically, a vagrant user in the wheel group needs to be added, and it should be able to run sudo without a password. Even, the unsecure Vagrant key needs to be authorized to access via ssh.

Note that the key is added at the system level, and the root user has no password, which should be appropriately changed after the first run or, even better, at provisioning time. That is a requirement if the box would be exposed on public network, because the default private and public keys of Vagrant are publicly distributed.

That is usually and automagically done by vagrant at first boot, but Guix is a read-only system and - as we will see - the Guix system is still not completely supported by Vagrant.

ROOTFS_UUID=$(sudo blkid -o value /dev/nbd0p1|head -1)
SWAP_UUID=$(sudo blkid -o value /dev/nbd0p2|head -1)
DEVICE=/dev/nbd0

# Download the unsecure ssh key used by Vagrant
wget -q -O vagrant.pub https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub

cat >guix-config.scm <<EOF
;; Indicate which modules to import to access the variables
;; used in this configuration.
(use-modules (gnu))
(use-modules (gnu system))
(use-modules (gnu services avahi))
(use-service-modules desktop networking ssh)

;; Prepare a salt function for a less silly password encryption
(set! *random-state* (random-state-from-platform))
(define str "0123456789abcdefghijklmnopqrstuvwxyz")
(define rnd-chr (lambda () (string-ref str (random (- (string-length str) 1)))))
(define salt (lambda () (string-append (string (rnd-chr)) (string (rnd-chr)) (string (rnd-chr)))))

(operating-system
  (locale "en_US.utf8")
  (timezone "Europe/Rome")
  (keyboard-layout (keyboard-layout "us" "intl"))
  (host-name "guix")

  ;; The list of user accounts ('root' is implicit).
  (users (cons* (user-account
                  (name "vagrant")
                  (comment "Vagrant user")
                  (group "users")
                  (home-directory "/home/vagrant")
                  (password (crypt "vagrant" (string-append "\$6\$" (salt))))
                  (supplementary-groups '("wheel" "netdev" "audio" "video")))
                %base-user-accounts))

  ;; Packages installed system-wide.  Users can also install packages
  ;; under their own account: use 'guix search KEYWORD' to search
  ;; for packages and 'guix install PACKAGE' to install a package.
  (packages (append (list (specification->package "nss-certs")
                          (specification->package "rsync"))
                    %base-packages))

  ;; Below is the list of system services.  To search for available
  ;; services, run 'guix system search KEYWORD' in a terminal.
  (services
   (append (list (service dhcp-client-service-type)
                 (service openssh-service-type
                    ;; here the official unsecure Vagrant ssh key is used...
                    (openssh-configuration
                      (authorized-keys \`(("vagrant" ,(local-file "vagrant.pub")))))))

           ;; This is the default list of services we
           ;; are appending to.
           %base-services))

  ;; Authorize vagrant to run sudo without password.
  (sudoers-file
    (plain-file "sudoers"
                 (string-append (plain-file-content %sudoers-specification)
                                "vagrant ALL=(ALL) NOPASSWD: ALL\\n")))

  (bootloader (bootloader-configuration
                (bootloader grub-bootloader)
                (targets (list "$DEVICE"))
                (keyboard-layout keyboard-layout)))
  (swap-devices (list (swap-space
                        (target (uuid
                                 "$SWAP_UUID")))))

  ;; The list of file systems that get "mounted".  The unique
  ;; file system identifiers there ("UUIDs") can be obtained
  ;; by running 'blkid' in a terminal.
  (file-systems (cons* (file-system
                         (mount-point "/")
                         (device (uuid
                                  "$ROOTFS_UUID"
                                  'ext4))
                         (type "ext4")) %base-file-systems)))
EOF

Run an initial setup of the mounted virtual disk using `guix` on your host

The system init can be run more than once, if required.

sudo guix pull # only if needed to update packages...
sudo guix system init guix-config.scm /mnt
sudo umount /mnt
sudo qemu-nbd --disconnect /dev/nbd0

Create a basic Virtualbox machine and attach the virtual disk to it

Now, the virtual disk should already be registered and visibile under your Virtualbox configuration.

VDI_UUID=`vboxmanage showhdinfo guix-hd.vdi|grep ^UUID|awk '{print $2}'` && echo $VDI_UUID

should show the hexadecimal code associated to the now populated disk. It is now possible to create a simple virtual machine, for instance:

vboxmanage createvm --name=Guix --default --ostype=Linux_64 --register
vboxmanage modifyvm Guix --memory=4096 --cpus=2 --ioapic=on --vram=256 --cpu-profile=host \
                --audio-enabled=off --usb-xhci=off --usb-ehci=off --usb-ohci=off --mouse=ps2

and attach the virtual disk to it, as follows:

vboxmanage storageattach Guix --storagectl=SATA --type=hdd --port=0 --device=0 --medium=$VDI_UUID
vboxmanage showvminfo Guix

Convert the machine into a Vagrant box and register it

The resulting vm can be directly used under Virtualbox, but the final touch is creating a proper Vagrant box to recycle and possibly publish on the cloud.

vagrant package --base Guix --output guix-small.box
vagrant box add guix-small.box --name=guix-small
vagrant box list

Now the new box is ready for use in multiple configurations within a Vagrantfile.

Prepare a Vagrantfile snippet and run the vm(s)

A simple Vagrantfile can be used to create an instance of the box, such as:

Vagrant.configure("2") do |config|
    config.vm.define "guix1" do |vm1|
     vm1.vm.box = "guix-small"
     vm1.vm.network "private_network", ip: "192.168.1.2"
     vm1.vm.provider "virtualbox" do |vb1|
        vb1.memory = "8192"
    	vb1.name = "guix8G"
    	vb1.cpus = 4
    end
    config.vm.provision "shell", inline: <<-SHELL
        guix pull
        guix install htop 
     SHELL
    end
end

A new machine can be created, started up and connected easily, with also an initial provisioning, by issuing:

vagant up guix1
vagrant ssh guix1

(Re)configure vm(s) on the basis of your needs

Both the box and the dependent virtual machines can be reconfigured as usual by guix system reconfigure in the guest or even remounting the original virtual disk (or any of the vmdk copies cloned by Vagrant) as previously done, then re-issuing a system init with a new Guile configuration. Note that the same reconfigure can also be run as described in the manual.

Missing features in Vagrant to support Guix

Vagrant needs to recognize the installed system in order to perform a few operations, but unfortunately that is still not the case for Guix. Specifically, that prevents the capability of halting the system, which is not exactly a nice thing. That must be done manually by running halt within an ssh session. For the same reason, Vagrant is not able to fully configure networking, so the second network interface shown in the previous Vagrantfile needs to be configured manually by adding a suitable static-networking-service-type section to the Guile configuration.

A suitable support would need to be added as a proper vagrant plugin as in the case of other operating systems and distributions. So, good but not good enough, maybe it is matter for another post, even if I'm not exactly a Ruby language fan.

In the meantime, have a good time by upping and destroying Vagrant Guix boxes.

The Guix system, take two

2024-09-02T20:00:00Z

Let's give a second look at Guix-the-system the main GNU Project distribution I dealt with in a previous post. This post is not specifically limited to the distribution, it is also of interest when using Guix in a foreign distribution, even if some configuration details change.

Substitutes and grafts

As said previously, the daily use of a store-based rolling distribution adds some overhead to the system at both upgrade and installation time. This pain (that remembers me the old times of the source-based Gentoo distribution with its emerge tool) is specifically alleviated by the use of the so-called substitutes servers, which provide pre-built binaries for the base system, as well as alternative/unofficial packages, too.

The fall-back alternative is based on regular build-from-sources on the host system, which could imply long times for both installations and distribution upgrades. The official Guix system comes with a couple of official substitutes (i.e., ci.guix.gnu.org and bordeaux.guix.gnu.org) but others can be added, including possibly any suitable user's server in the LAN.

Another trick in Guix for alleviating the need for long local rebuilds is the use of grafts that are sort of in-place replacements for binary dependencies, expressed at the source level. To explain in brief, if a package A@1 has been replaced by A@2 and they both maintain the same ABI, any reverse dependency B can avoid being rebuilt. This is called a graft in Guix, and greatly simplifies the long chains of forced rebuilds in many cases (for instance, in case of security upgrades). Specifically, the @ in a Guix package is the version separator.

A monorepo for the whole distribution

The Guix source archive is strictly based on git and distributed development by means of a monorepo, which, along with the need to represent the tree of dependencies for any package and its updates at run-time require some operations that are specific of the Guix package management approach:

pulling
garbage collection
branching
using channels

The pull phase specifically could require ages (hours on my old low-end Asus EEEpc), so it is an operation that should be applied typically in batch mode and quite often in order to minimize execution times. It is not a potentially destructive operation - similar to an apt update with steroids - so it is better to perform it quite often, every few days. At the end of the day, it does not alter the status of the system, so it is safe enough for background execution.

The gc operation works directly on the store to purge obsolete (out of the current status tree of dependencies) entries. If you jump to a past status, the purge would impact bandwidth and CPU loads, of course.

The branch specification has the same role as any sane distributed environment organization of code. It is usable to pull from separate branches of archives, instead of following the default one, latest. This feature can be paired with the Guix time-machine to jump to any past tree of packages in the chronology of the archive sources.

Finally, a channel is simply an alternative archive of sources that is prepared by third-party teams to integrate the official Guix one. For instance, a few independent packages are offered by INRIA and other institutions for the HPC community, as well as an handful of non-free packages hosted on the nonguix repository to solve the well-known users-hostages' dilemma about using closed-source firmwares and a few other proprietary stuff.

One Scheme to Rule Them All

An exciting feature of Guix-the-System is the use of Guile to describe the whole system, including the core, all services, and even users. In perspective, all installed packages could be also fully configured by using functional Guile snippets of code. This is something currently done in Debian only in a limited way, through debconf templates and dpkg selections. In practice, today, one has to use ansible or similar tools to declare configurations in some ad hoc DSL, using tons of plugins and templates, case by case. In a word, it is a mess.

This is, at least for me, the most intriguing feature and open exciting possibilities.

Currently, any developer with a reasonably decent computer can easily use Guix to rebuild and customize Guix itself by starting from a monorepo fork, changing its main configuration and adding/modifying packages in a totally independent and self-consistent way. Something that in traditional distributions is done by a plethora of tools and interfaces, written in multiple generale-purpose or specific languages, and often not wholly documented and held together with the glue.

Once added a layer of general-purpose configurators for common packages the whole distribution generation could become fully autoconsistent and complete for any host or set of boxes. Isn't that challenging?

Guix in foreign distributions

Using Guix as an additional package manager in a foreign distribution has more limitations, of course. First of all, one must deal with systemd as the typical init system. Therefore, the general configuration of the host cannot be expressed in a synthetic way as a Scheme script. That said, it is perfectly possible to use Guix as a development environment for multiple languages ecosystems, thanks to various Guile build modules. Even, it is possible to run Guix-the-system in a container (or a virtual machine), to use the host system just as a basic platform and create Guix-based services and applications on top.

But those will be the topics for other posts...

References

An initial dive into Guix

2024-08-18T19:00:00Z

In the last few days, I got familiar with Guix, which is both a modern package management system and the main GNU Project distribution for Linux and Hurd (the Guix system). As a package management system, it can be installed on most foreign distributions, including Debian and any other, as an alternative/additional packaging system.

I both installed the Guix system natively on a small ancient laptop of mine (an ASUS EEEpc 1215N), and the Guix package manager on one of my Debian stable boxes. An interesting variant could be installing the whole system under a container in a non-interactive mode, but that may be a task for the future. Indeed, the last one could be the most exciting application of Guix for reproducible deployment.

Guix, the package manager

Guix (the package management system) is the most interesting part. It is a modern system with multiple advanced features, inspired by Nix, a pre-existing system based on a JSON-based DSL. Nix and Guix are both declarative and functional package managers with similar goals for software maintenance.

Both of them declare to have the largest collection of FOSS packages in the world, but ok: both currently have hubs with tens of thousands of binary packages. Maybe not the largest, but respectful. Of course, Guix is an FSF project and therefore highly choosy about the software that can be distributed within the Guix archives. That's not specifically different from the Debian approach, but for the derogation that historically the Debian Project considered for limited proprietary-but-distributable stuff (the non-free+contrib section).

One interesting aspect of Guix (at least for me) is that it is specifically based on Guile, an extendable Scheme dialect which is the main DSL used in the GNU ecosystem. All packages are expressed as small snippets of Guile functions that declare pre-dependencies, as well as the build, installation and test phases of each software.

Anyone who has worked on software packaging in Debian from the beginning knows that the mythical debian/rules is essentially a Makefile with steroids, accompanied by a handful of declarative files, lately simplified by the use of some frameworks, such as Joey Hess's debhelper or others commonly used in the past. Maybe not the most elegant approach to packaging and configuring, let me say. Probably at the time - 30 years ago or so - the most pragmatic one, for sure. And it worked even for a lot of years.

In respect with traditional system-wide packaging, the Nix/Guix approach has several interesting features:

transitional, per-user, multi-profile capabilities
a rolling-back capability at the system and user level
an all-in-one system of packing with dependencies
a single expressive way of defining software configurations at both system-wide and per-user levels

Guix per se adds the use of Guile to all the rest so that all configurations are synthetically expressed in S-exp, without the need to learn yet another DSL to describe the software chains of dependencies.

Guix, the system

The Guix free-software-only system has some interesting characteristics, including the use of shepherd as an alternative Guile-based init system and the rolling-on distribution style. The non-FHS basic organization of the filesystem could also pose some problems to install software that are strictly dependent on that, that's for sure a good reason to use Guix-on-Debian instead of Guix-the-system only. That issue is also partially mitigated by a combination of a container technology support, and an FHS emulation layer.

In my opinion, the whole thing is quite interesting for building development environments and exploring reproducible deployment systems.

The GNU touch

But for the FSF strictly free approach to collecting software (including the missing firmware blobs for the Linux-libre software, as for Debian until version 12), the Guix system has some typical geek-only pillars for its ecosystem and community:

strictly read the reference manuals and info files (i.e., do your homework)
use mailing lists
use IRC dedicated channels
use your brain and experience to solve issues

Nothing different from the traditional Debian project community and approach to personal computing. Therefore, nowadays something for geeky folk mainly, I guess.

Issues and challenges

For sure, the user workflow to install and run software changes radically and in a very different manner. One has the need to get familiar the Guix CLI interface and mode of operations. The Guix approach to deployment and maintenance adds an evident overhead to the system (for both storage and CPUs), partially mitigated by the use of substitutes/hubs to reduce building from source requirements. Not the best thing for old boxes, I guess.

Anyway, it is possible to use local substitutes to reduce the load for average systems. As a rolling distribution and/or software hub, I found it reasonably updated, but that largely depends on applications and domains. Nobody makes miracles in those regards, DebianGis packages in testing/unstable are more up-to-date for geospatial apps, and probably even more flexible. Sorry, no silver bullet, guys. Even Guix does not have (still?) a dedicated security team, so I would recommend it currently only for personal/development use, not for servers.

An important feature of Guix is its support for reproducible software deployment, as an alternative/integration to the ubiquitous containers, an aspect to deeply explore.

References

The main resources about Guix are the Reference Manual and The Cookbook, of course. I found some interesting non-trivial articles about Guix and its internals on some indie sites here listed:

frankie-tales

Creating your own Guix substitute server

What is a Substitute Server?

Installation of Guix on Debian

Daemon configuration and storage

The systemd unit file for the Guix daemon

Setting up Guix Publish

About computing environments for reproducible science

References

Guix for geeks

Guix and geospatial software

Playing with Vagrant, Virtualbox and Guix

Create a virtual disk image and partition it

Prepare a suitable configuration written in Guile for Guix

Run an initial setup of the mounted virtual disk using guix on your host

Create a basic Virtualbox machine and attach the virtual disk to it

Convert the machine into a Vagrant box and register it

Prepare a Vagrantfile snippet and run the vm(s)

(Re)configure vm(s) on the basis of your needs

Missing features in Vagrant to support Guix

The Guix system, take two

Substitutes and grafts

A monorepo for the whole distribution

One Scheme to Rule Them All

Guix in foreign distributions

References

An initial dive into Guix

Guix, the package manager

Guix, the system

The GNU touch

Issues and challenges

References

Run an initial setup of the mounted virtual disk using `guix` on your host