From 68537c60f606366f05419ff9827205f61d6082bf Mon Sep 17 00:00:00 2001 From: Philippe PITTOLI Date: Fri, 24 Apr 2020 04:49:03 +0200 Subject: [PATCH] Adding sections about spec files and `packaging`. --- content/baguette/index.md | 221 +++++++++++++++++++++++++++++++------- 1 file changed, 181 insertions(+), 40 deletions(-) diff --git a/content/baguette/index.md b/content/baguette/index.md index 9560bae..6c8aad9 100644 --- a/content/baguette/index.md +++ b/content/baguette/index.md @@ -272,15 +272,13 @@ There is a strict separation between core system and third party software.
All our tools are designed to be simple to use, to understand, to read. Feel free to provide a feedback. -TODO: spec files, the file format used in `package`, `packaging` and `service`. - --- **Spec files.** *our declarative format*
Before presenting our tools, here a file format named `spec` that we tend to use when relevant. It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`). -This way, we can provide an easy to understand format which looks like YAML a lot, but allows us to do much more. +The `spec format` is only composed of variables, lists, code block and named sections. Here a quick example. ```yaml @@ -299,14 +297,14 @@ list: - c ``` -And now, we can add `block sections` to it: sometime we do want to tell instructions. +This looks a lot like YAML, but now we add `code block` to it: because sometimes we do want to tell instructions. ```yaml # We have the URL of the tarballs for a software software-url: 'https://example.com/my-software/' # ... and we want to get its last version number -# "@watch" is a section, the name "watch" has a meaning in the application reading the file +# "@watch" is a block, the name "watch" has a meaning in the application reading the file @watch # the following is simple shell scripting curl %{software-url} -o- 2>/dev/null | \ @@ -314,23 +312,27 @@ software-url: 'https://example.com/my-software/' tail -1 ``` -Sometime, we want to refer to a file (or directory), and add metadata to it. +Sometimes, we want to refer to a file (or directory) and add metadata to it: here `named sections`. + ```yaml # as for "@watch", "%configuration" has a meaning in the application reading the file -# this time, the block is named +# this time, the block has an arbitrary name %configuration postgresql.conf - # within a named block, we find simple declarations as outside the block + # within a named section, we find simple declarations as outside the block name: database configuration creation-command: my-script.sh -a -b -c ``` +Next, the usage in practice: [packaging](#packaging), [service](#service). + --- TODO: explains why it's different / better than other package managers. + **[Package][package]: our package manager.**
-Package covers the basics: install, remove, search and provide informations about a package, and it creates rootfs. -Package knows the minimal set of binaries and configuration required to build the target, so it only installs the minimal environment to perform compilation. +`Package` covers the basics: install, remove, search and provide informations about a package. +`Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example. Package provides slotting by default: no need for custom environments for each software. @@ -339,37 +341,37 @@ The database format contains `world`, `installed`, `[package-name]/[slot]/manife Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on). +Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code. + --- -**[Packaging][packaging]: to create packages.**
-Packaging uses simple, declarative recipe files, here some examples: [hello][recipe-hello], [dhcpcd][recipe-dhcpcd], [alsa-utils][recipe-alsautils]. -The only required parameters are `name` and `sources`. +**[Packaging][packaging]: creates packages.**
+Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries. +As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries. +`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format). +`Packaging` has a few advantages compared to most used packaging tools: +- declarative recipes abstact OS specifics
+ The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies. + This only requires to provide a backend for the target package manager. +- recipe readability is great
+ A few variable declarations are better than a dozen lines of code. +- trivial shell script patterns become automated
+ `Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project. +- tooling may evolve, very few recipes will be require to change
+ Everybody wants to change its build system? + *(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`. +- it creates hash'ed and signed packages by default
+ You need your own set of cryptographic keys, which is created at first use. +- it creates repositories (automatic at first compilation), helping people maintaining their own set of tools
+ Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository. + That is that simple. -```yaml -# GNU Hello example program -name: hello -version: 2.10 -release: 2 -sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz - -dependencies: - - gettext - -build-dependencies: - - make - -options: - - configure: --disable-nls - -@watch - curl 'https://ftp.gnu.org/gnu/hello/' -o- 2>/dev/null | - sed -n "/hello-.*\.tar\.gz/{s/\.tar\.gz.*//;s/.*hello-//;p}" | - tail -1 -``` +shell script is lava -Packaging creates build environments to test packages before validation. +**`Packaging` build environments**
+`Packages` creates build environments to test packages before validation. It works as follow: 1. creation of a `/tmp/packaging/build-UUID/` directory 2. sources are downloaded, extracted then compiled
@@ -379,14 +381,152 @@ It works as follow: `Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD. +In this case, `package` only installs the minimal set of binaries required by the package to build. +Besides the target application, a building environment size is only a few kilobytes. -TODO. -Packaging configuration is simple. +**`Packaging` configuration**. *common configuration for your packages*
+`Packaging` may be configured globally for your system with the file `/etc/packaging.conf` which contains the following: -TODO. - -Packaging recipes. +```YAML +# Configuration file for `packaging` + +# where to send built packages +packages-directory: /usr/local/pkg/ +# where to download sources +sources-directory: /usr/local/src/ + +# prefixes for `packaging` running environment and child processes +prefixes: + # the first prefix is the default slot used for building application + - /usr/baguette/ + - / + +# list of environment variables we want to have when building +environment: + # you may choose another compiler, provide some CFLAGS, etc. + - CC: clang + - CFLAGS: -Os -Wall + + # next three have special meaning + # to provide parameters to the `./configure` script when building + - configure: --disable-nls --without-gettext + + # to provide parameters to the `make` command + - make: + + # to provide parameters to the final `make install` command + - make install: + +# wanna build for another system? not a problem, just add the backend (we currently have `apk` and `package`) +package-manager: package +``` +That's it. You know all about `packaging` configuration. +These parameters may be override by recipes. + + +**`Packaging` recipes.** *we need to create packages*
+A recipe is the way to reproduce something; here we want to create a package, the recipe should provide all data necessary to be able to reproduce the package. +This means at least having a name for the software and a version (they appear in the package name) and sources (software code). +Let's take an example. + + +```yaml +# GNU Hello program +name: hello # software name +version: 2.10 # software version +release: 2 # recipe release: incremented when the recipe change for the current version of the software + +# the description will appear in the package information we can retrieve with `package` +description: "This is the GNU Hello program." + +# sources may be multiple: you may want to add arbitrary files along the tarball +sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz + +# we provide running dependencies: the program needs these to `run` +dependencies: + - gettext + +# we provide build dependencies: the program needs these to `be compiled` for the package +# no need to require these dependencies for a simple user installing the program +build-dependencies: + - make + +# if we want to add or override compilation options +options: + - configure: --disable-nls + +# feels like déjà vu, right? +# "watch" code block helps to check if the recipe covers the last software version +@watch + curl 'https://ftp.gnu.org/gnu/hello/' -o- 2>/dev/null | + sed -n "/hello-.*\.tar\.gz/{s/\.tar\.gz.*//;s/.*hello-//;p}" | + tail -1 +``` + +This was [a real example][recipe-hello], and not the simplest one. +Most of our recipes currently look like this: + +```YAML +name: dhcpcd +version: 8.0.3 +sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz +``` +That's it. +Yes, we can add a few meta-data, but this is a working recipe. +Configuration, compilation and packaging are done without providing anything else. +The only required parameters are `name`, `sources` and `version`. + + + +Sometimes, developers are assholes and force you to fix their build system. +When manual operations really are required, you can use `@configure`, `@build` and `@install` code blocks. +Let's see an example with a special snowflake… like `perl`. +```YAML +# in our example, we want to configure the build +@configure + # we set some script execution parameters: exit at any error, print everything you do + set -e -x + + # we currently are in `/tmp/packaging/build-UUID/` + # now we enter the directory created by the tarball extraction, where the build occurs + cd perl-%{version} # we can interpolate variables with this: %{} + + # Perl needs a few environment variables for configuration + BUILD_ZLIB=0 + BUILD_BZIP2=0 + BZIP2_LIB=/usr/lib + BZIP2_INCLUDE=/usr/include + export BUILD_ZLIB BUILD_BZIP2 BZIP2_LIB BZIP2_INCLUDE + + # now we can launch the `configure` script… + # … but wait kids, that's not `configure`, that is `Configure` (see the capitalized C?) + # this is a totally different and very special script, but you should love it anyway + ./Configure -des -Dcccdlflags='-fPIC' \ + -Dcccdlflags='-fPIC' \ + -Dccdlflags='-rdynamic' \ + # ... some long and uninteresting list of very specific parameters because we are Perl, we are historic and stuff +``` + +Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems. +Non standard build operations happen from time to time, and code blocks help you overcome this. +When a lot of packages have the same workarounds, we think about detection and integration into `packaging`, so we can keep only specifics into recipes. + +If you want to investigate a bit more, you can check [our recipe repository][recipes]. +Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome. + + +TODO: talk about creating a repository. + + +**Future of `packaging`.** *let's be even more declarative*
+As we saw, `packaging` allows maintainers to create very simple and **readable** recipes. +Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit. +In the future, manual operations should be reduced even more: +- by adding a few other parameters to the environment
+ This imply to check for patterns in the recipes and to provide workarounds. +- by adding new code blocks *before* the `configuration`, `build` and `install` steps
+ This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) and still keeping automatic operations: the best of the two worlds. --- @@ -561,6 +701,7 @@ We currently aim at providing a rootfs with our tools, when we will have enough [baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc [baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile +[recipes]: https://git.baguette.netlib.re/Baguette/recipes/ [recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec [recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec [recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec