#+TITLE: Guixy Guile & the Derivation Factory: a tour of the Guix source tree
#+AUTHOR: Josselin Poiret

* A general reading guide
2594 files with =git ls-files= when I ran it last week.

Most of it is packages + package patches, but still 600+ Guile files!

Each Guile file has:
- a copyright notice at the top;
- a define-module declaration;
- a general file commentary! (not guaranteed, but very often).

=grep= and derivatives are your friends.

* Foreword: The gnu and guix distinction
- =guix/= contains internal Guix mechanisms and machinery.  See it as a library.
- =gnu/= contains things built with the help of Guix.  See it as the application
  built on top of the libraries.

The distinction gets murky on some areas though, but these are the general
guidelines.

* Scripts
=scripts/guix.in=

Guix's entrypoint, not much going on, delegates to guix/scripts.scm.

Every guix command is defined in guix/scripts/

* UI/UX modules
Provides generic procedures to interact with the user, either to communicate
events in a uniform manner, or translate things.

guix/diagnostics.scm: errors, warnings, notes
guix/i18n.scm: internationalization
guix/status.scm: status of ongoing builds
guix/ui.scm: general interface with the user

* Packages
The package definitions.

gnu/artwork.scm: artwork repository definition
gnu/packages.scm: package cache, =specification->package= and friends
gnu/packages/*
gnu/packages/aux-files/*
gnu/packages/patches/*

** important examples
gnu/packages/aux-files/linux-libre/*: Linux-libre configurations
gnu/packages/commencement.scm: basic toolchains from bootstrap blobs
gnu/packages/base.scm: basic programs like =grep=, =hello=, =coreutils=
gnu/packages/bootstrap.scm: building the bootstrap blobs from even less

* Higher-level interface
What is used to write package definitions, but also most things that result in
store items elsewhere in Guix.

guix.scm: convenient re-export
guix/gexp.scm: the lair of the beast
guix/grafts.scm
guix/licenses.scm
guix/packages.scm
guix/profiles.scm: profiles and manifests (external and internal)
guix/search-paths.scm
guix/transformations.scm: customize packages to your needs
guix/cpu.scm: detect the cpu's microarch

* Interlude: to build or not to build
There are a lot of different build/ folders throughout Guix source.  They
contain code that should be directly usable in a G-Exp, most of the time inside
a build environment but sometimes not.  There almost always exists a distinction
between code that defines features and the one that does the corresponding
building.

For example, =guix/elf.scm= defines Guile structures pertaining to ELF files such
as the file format itself, while =guix/build/gremlin.scm= defines runpath
validation and stripping.

Examples:
gnu/build/chromium-extension.scm
gnu/build/cross-toolchain.scm: Building a cross-compiler.
gnu/build/locale.scm
gnu/build/svg.scm: Render svg to png using rsvg and cairo bindings
guix/build/profiles.scm
guix/build/syscalls.scm: Linux syscalls, used throughout Guix

* Build systems
The usual: =guix/build-system/*.scm= define the build-systems record objects to
be used in package declarations, while =guix/build/*.scm= define the
corresponding procedures used in the build phases.  The former uses the latter,
but not the other way around.

guix/build-system.scm
guix/build-system/*
guix/build/*

** Chosen bits
guix/build-system/gnu.scm: the almighty GNU Autotools build system
guix/build-system/trivial.scm: most flexible since it does nothing by itself
guix/build/gnu-build-system.scm: the Autotools-calling code
guix/build/utils.scm

* Downloaders
When building a package, there's not only the question of how to build the
sources, but also of how to fetch the sources themselves!  We've got multiple
downloaders to cover our bases.

guix/*-download.scm
guix/download.scm
guix/git.scm
guix/git-download.scm
guix/swh.scm: fetching from Software Heritage!

* Low-level derivations, store and daemon interaction
In the end, all of this is just an abstraction layer above raw daemon
interaction.  We need to convert all of this into nix daemon language, hence
these files:

guix/derivations.scm
guix/nar.scm: the Nix ARchive format
guix/narinfo.scm
guix/store.scm
guix/store/*.scm
guix/monads.scm: Haskell in Guix
guix/monad-repl.scm
guix/platform.scm: classifying build triplets
guix/platforms/*
guix/serialization.scm: serialization code for nars
guix/ssh.scm: communicating with stores over SSH

* Substitutes

* System
Definitions of all the operating system-related structures, along with some
examples.

gnu.scm: convenient re-export
gnu/system.scm: base definitions, =operating-system='s home
gnu/system/accounts.scm
gnu/system/examples/*
gnu/system/file-systems.scm
gnu/system/mapped-devices.scm
gnu/system/hurd.scm
gnu/image.scm - Basic image definitions
gnu/system/image.scm
gnu/system/images/*
gnu/system/install.scm: definition of the installation system
gnu/system/keyboard.scm
gnu/system/linux-container.scm
gnu/system/linux-initrd.scm
gnu/system/*.scm
gnu/bootloader.scm
gnu/bootloader/*.scm

* Services
Definition of all user-facing Guix System services.

gnu/services.scm: some basic services are defined here!
gnu/services/*
gnu/services/base.scm: others here!
gnu/services/desktop.scm
gnu/services/xorg.scm
guix/least-authority.scm: launching programs with no rights

* Building a system
Most of the system building code lives in gnu/build/.

gnu/build/accounts.scm
gnu/build/activation.scm
gnu/build/bootloader.scm
gnu/build/dbus-service.scm
gnu/build/file-systems.scm
gnu/build/hurd-boot.scm
gnu/build/image.scm
gnu/build/install.scm
gnu/build/jami-service.scm: most tightly integrated Guix service
gnu/build/linux-boot.scm
gnu/build/linux-container.scm
gnu/build/linux-initrd.scm
gnu/build/linux-modules.scm
gnu/build/secret-service.scm
gnu/build/shepherd.scm
guix/cpio.scm: CPIO archive format for initrds

* Installer
The installer separates front-end and back-end.  Things in =gnu/installer/= are
not compiled by =make=.

gnu/installer.scm: Main G-Exp for the program
gnu/installer/aux-files/logo.txt
gnu/installer/*.scm
gnu/installer/newt.scm
gnu/installer/newt/*

* Guix home
Very similar structure to Guix system!  But less files and specificities to wade
through (for now :) ).

gnu/home.scm
gnu/home/services.scm
gnu/home/services/*.scm

* Working automatically with package definitions
Linting with
guix/cve.scm
guix/lint.scm

Importing
guix/ftp-client.scm
guix/hash.scm
guix/http-client.scm
guix/import/*
guix/scripts/import/*

Updating with
guix/gnu-maintenance.scm
guix/upstream.scm

* Graph
Surprise!  JavaScript files in Guix!

They are used for the web interface of =guix graph=.

guix/d3.v3.js
guix/graph.js
guix/graph.scm

* Guix deploy
Script for provisioning distant machines!  See [[https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/][Managing Servers with GNU Guix: A Tutorial]] for the announcement and =(guix) Invoking guix deploy=.

gnu/machine.scm
gnu/machine/digital-ocean.scm
gnu/machine/ssh.scm
guix/remote.scm: Evaluating G-Exps on a remote machine, after copying the needed
dependencies.

* building with guix channel
How people build Guix when using Guix as a channel.

build-aux/build-self.scm: build script when building with =guix pull=
guix/self.scm: how Guix builds itself
guix/channels.scm: basic definitions of a channel
guix/config.scm.in: template file for Guix build time info
guix/describe.scm: looking up the channel versions
guix/git-authenticate.scm
guix/inferior.scm
guix/quirks.scm: fixing the past
.guix-authorizations
.guix-channel

* Misc guile commands
guix/avahi.scm: =guix discover=
guix/ci.scm: interaction with Cuirass, =guix weather=
guix/docker.scm: =guix container= and =guix pack=
guix/ipfs.scm
guix/substitutes.scm: =guix substitute=

* Misc Guile tools
A lot of Guix-unrelated code that's useful to Guix!  Home of some of the
Guix-sms.

guix/base16.scm
guix/base32.scm
guix/base64.scm
guix/cache.scm
guix/colors.scm
guix/combinators.scm
guix/deprecation.scm
guix/discovery.scm
guix/glob.scm
guix/gnupg.scm
guix/man-db.scm: our own =man-db= implementation, why not?
guix/memoization.scm
guix/modules.scm: module dependency scanning
guix/openpgp.scm
guix/pki.scm
guix/profiling.scm
guix/progress.scm
guix/read-print.scm: pretty printer with comments
guix/records.scm: The =define-record-type*= definition
guix/repl.scm
guix/sets.scm
guix/utils.scm
guix/workers.scm

* building with autotools
How people build when developing locally.

Makefile.am
bootstrap
config-daemon.ac
configure.ac
gnu/local.mk: Don't forget to put your added files/patches in there!
m4/guix.m4

* nix
The infamous gnix daemon, in C++.  =boost/= contains some vendored code from the
boost library, while =libutil/= is a miscellaneous library containing various
utilities.  The meat of the daemon resides in =libstore/=, with =build.cc= being
the “most” interesting.

nix/.gitignore
nix/AUTHORS
nix/COPYING
nix/boost/*
nix/libstore/build.cc
nix/libstore/*
nix/libutil/*
nix/local.mk
nix/nix-daemon/guix-daemon.cc
nix/nix-daemon/nix-daemon.cc
nix/nix-daemon/shared.hh

* doc
doc/build.scm
doc/contributing.texi: the `contributing' chapter of the manual
doc/environment-gdb.scm
doc/fdl-1.3.texi
doc/guix-cookbook.texi: the cookbook
doc/guix.texi: the manual
doc/he-config-bare-bones.scm
doc/htmlxref.cnf
doc/images/*
doc/local.mk
doc/package-hello.json
doc/package-hello.scm

* testing
Tests are either .scm or .sh files, the first using SRFI 64, the second being
simple shell scripts with =set -e=, ie. that exit on *any* error.

gnu/build/marionette.scm: Framework to test Guix system/installer with QEMU VMs
gnu/ci.scm: Definition of CI jobs for =make cuirass-jobs=.
gnu/tests.scm
gnu/tests/*
guix/tests.scm
guix/tests/git.scm
guix/tests/gnupg.scm
guix/tests/http.scm
tests/*

* build-aux
Various scripts used by maintainers and contributers.

build-aux/check-channel-news.scm
build-aux/check-final-inputs-self-contained.scm
build-aux/compile-all.scm
build-aux/compile-as-derivation.scm
build-aux/convert-xref.scm
build-aux/cuirass/evaluate.scm
build-aux/cuirass/hurd-manifest.scm
build-aux/generate-authors.scm
build-aux/git-version-gen
build-aux/gitlog-to-changelog
build-aux/pre-inst-env.in
build-aux/test-driver.scm
build-aux/test-env.in
build-aux/update-NEWS.scm
build-aux/update-guix-package.scm

* etc
Potpourri of shell completions, daemon service files, snippets for Emacs, etc...

etc/committer.scm.in
etc/completion/bash/guix
etc/completion/bash/guix-daemon
etc/completion/fish/guix.fish
etc/completion/zsh/_guix
etc/copyright.el
etc/disarchive-manifest.scm
etc/git/gitconfig
etc/git/pre-push
etc/gnu-store.mount.in
etc/guix-daemon.cil.in
etc/guix-daemon.conf.in
etc/guix-daemon.service.in
etc/guix-gc.service.in
etc/guix-gc.timer
etc/guix-install.sh
etc/guix-publish.conf.in
etc/guix-publish.service.in
etc/historical-authorizations
etc/init.d/guix-daemon.in
etc/news.scm
etc/openrc/guix-daemon.in
etc/release-manifest.scm
etc/snippets/tempel/*
etc/snippets/yas/*
etc/source-manifest.scm
etc/substitutes/berlin.guix.gnu.org.pub
etc/substitutes/bordeaux.guix.gnu.org.pub
etc/substitutes/ci.guix.gnu.org.pub
etc/substitutes/ci.guix.info.pub
etc/system-tests.scm
etc/teams.scm.in
etc/time-travel-manifest.scm

* po files
The translation apparatus and files!  These should not be edited manually.

po/doc/guix-cookbook.*.po
po/doc/guix-manual.*.po
po/doc/local.mk
po/guix/LINGUAS
po/guix/Makevars
po/guix/POTFILES.in
po/guix/*
po/packages/LINGUAS
po/packages/Makevars
po/packages/POTFILES.in
po/packages/*.po

* Extra
.dir-locals.el
.gitattributes
.gitignore
.mailmap
AUTHORS
CODE-OF-CONDUCT
COPYING
ChangeLog
HACKING
NEWS
README
ROADMAP
THANKS
TODO

* Happy hacking!
I hope you've discovered some hidden sides of Guix, and now have a better idea
of who's where!

Any questions?