Baguette
/
website
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Adding sections about spec files and `packaging`.

master
Philippe PITTOLI 2020-04-24 04:49:03 +02:00
parent acb59721d3
commit 68537c60f6
1 changed files with 181 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

221
content/baguette/index.md
View File

@ -272,15 +272,13 @@ There is a strict separation between core system and third party software.<br />
All our tools are designed to be simple to use, to understand, to read. All our tools are designed to be simple to use, to understand, to read.
<u>Feel free to provide a feedback.</u> <u>Feel free to provide a feedback.</u>
<red>TODO: spec files, the file format used in `package`, `packaging` and `service`.</red>
--- ---
<a name="spec-format"></a> <a name="spec-format"></a>
**Spec files.** *our declarative format*<br /> **Spec files.** *our declarative format*<br />
Before presenting our tools, here a file format named `spec` that we tend to use when relevant. Before presenting our tools, here a file format named `spec` that we tend to use when relevant.
It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`). It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`).
This way, we can provide an easy to understand format which looks like YAML a lot, but allows us to do much more. The `spec format` is only composed of variables, lists, code block and named sections.
Here a quick example. Here a quick example.
```yaml ```yaml
@ -299,14 +297,14 @@ list:
- c - c
``` ```
And now, we can add `block sections` to it: sometime we <u>do</u> want to tell instructions. This looks a lot like YAML, but now we add `code block` to it: because sometimes we <u>do</u> want to tell instructions.
```yaml ```yaml
# We have the URL of the tarballs for a software # We have the URL of the tarballs for a software
software-url: 'https://example.com/my-software/' software-url: 'https://example.com/my-software/'
# ... and we want to get its last version number # ... and we want to get its last version number
# "@watch" is a section, the name "watch" has a meaning in the application reading the file # "@watch" is a block, the name "watch" has a meaning in the application reading the file
@watch @watch
# the following is simple shell scripting # the following is simple shell scripting
curl %{software-url} -o- 2>/dev/null | \ curl %{software-url} -o- 2>/dev/null | \
@ -314,23 +312,27 @@ software-url: 'https://example.com/my-software/'
tail -1 tail -1
``` ```
Sometime, we want to refer to a file (or directory), and add metadata to it. Sometimes, we want to refer to a file (or directory) and add metadata to it: here `named sections`.
```yaml ```yaml
# as for "@watch", "%configuration" has a meaning in the application reading the file # as for "@watch", "%configuration" has a meaning in the application reading the file
# this time, the block is named # this time, the block has an arbitrary name
%configuration postgresql.conf %configuration postgresql.conf
# within a named block, we find simple declarations as outside the block # within a named section, we find simple declarations as outside the block
name: database configuration name: database configuration
creation-command: my-script.sh -a -b -c creation-command: my-script.sh -a -b -c
``` ```
Next, the usage in practice: [packaging](#packaging), [service](#service).
--- ---
<red>TODO: explains why it's different / better than other package managers.</red> <red>TODO: explains why it's different / better than other package managers.</red>
<a name="package"></a>
**[Package][package]: our package manager.**<br /> **[Package][package]: our package manager.**<br />
Package covers the basics: install, remove, search and provide informations about a package, and it creates rootfs. `Package` covers the basics: install, remove, search and provide informations about a package.
Package knows the minimal set of binaries and configuration required to build the target, so it only installs the minimal environment to perform compilation. `Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example.
Package provides slotting by default: no need for custom environments for each software. Package provides slotting by default: no need for custom environments for each software.
@ -339,37 +341,37 @@ The database format contains `world`, `installed`, `[package-name]/[slot]/manife
Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on). Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on).
Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code.
--- ---
<a name="packaging"></a> <a name="packaging"></a>
**[Packaging][packaging]: to create packages.**<br /> **[Packaging][packaging]: creates packages.**<br />
Packaging uses simple, declarative recipe files, here some examples: [hello][recipe-hello], [dhcpcd][recipe-dhcpcd], [alsa-utils][recipe-alsautils]. Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries.
The only required parameters are `name` and `sources`. As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries.
`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
`Packaging` has a few advantages compared to most used packaging tools:
- declarative recipes abstact OS specifics<br />
The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies.
This only requires to provide a backend for the target package manager.
- recipe readability is great<br />
A few variable declarations are better than a dozen lines of code.
- trivial shell script patterns become automated<br />
`Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project.
- tooling may evolve, very few recipes will be require to change<br />
Everybody wants to change its build system?
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`.
- it creates hash'ed and signed packages by default<br />
You need your own set of cryptographic keys, which is created at first use.
- it creates repositories (automatic at first compilation), helping people maintaining their own set of tools<br />
Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
That is that simple.
```yaml <img src="/shell-scripting-is-lava.png" alt="shell script is lava" />
# GNU Hello example program
name: hello
version: 2.10
release: 2
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
dependencies:
- gettext
build-dependencies:
- make
options:
- configure: --disable-nls
@watch
curl 'https://ftp.gnu.org/gnu/hello/' -o- 2>/dev/null |
sed -n "/hello-.*\.tar\.gz/{s/\.tar\.gz.*//;s/.*hello-//;p}" |
tail -1
```
<a name="packaging-build-env"></a> <a name="packaging-build-env"></a>
Packaging creates build environments to test packages before validation. **`Packaging` build environments**<br />
`Packages` creates build environments to test packages before validation.
It works as follow: It works as follow:
1. creation of a `/tmp/packaging/build-UUID/` directory 1. creation of a `/tmp/packaging/build-UUID/` directory
2. sources are downloaded, extracted then compiled<br /> 2. sources are downloaded, extracted then compiled<br />
@ -379,14 +381,152 @@ It works as follow:
`Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD. `Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
In this case, `package` only installs the minimal set of binaries required by the package to build.
Besides the target application, a building environment size is <u>only a few kilobytes</u>.
<red>TODO.</red>
<a name="packaging-host-config"></a> <a name="packaging-host-config"></a>
Packaging configuration is simple. **`Packaging` configuration**. *common configuration for your packages*<br />
`Packaging` may be configured globally for your system with the file `/etc/packaging.conf` which contains the following:
<red>TODO.</red> ```YAML
<a name="packaging-host-config"></a> # Configuration file for `packaging`
Packaging recipes.
# where to send built packages
packages-directory: /usr/local/pkg/
# where to download sources
sources-directory: /usr/local/src/
# prefixes for `packaging` running environment and child processes
prefixes:
# the first prefix is the default slot used for building application
- /usr/baguette/
- /
# list of environment variables we want to have when building
environment:
# you may choose another compiler, provide some CFLAGS, etc.
- CC: clang
- CFLAGS: -Os -Wall
# next three have special meaning
# to provide parameters to the `./configure` script when building
- configure: --disable-nls --without-gettext
# to provide parameters to the `make` command
- make:
# to provide parameters to the final `make install` command
- make install:
# wanna build for another system? not a problem, just add the backend (we currently have `apk` and `package`)
package-manager: package
```
That's it. You know all about `packaging` configuration.
These parameters may be override by recipes.
<a name="recipes"></a>
**`Packaging` recipes.** *we need to create packages*<br />
A recipe is the way to reproduce something; here we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
This means at least having a name for the software and a version (they appear in the package name) and sources (software code).
Let's take an example.
```yaml
# GNU Hello program
name: hello # software name
version: 2.10 # software version
release: 2 # recipe release: incremented when the recipe change for the current version of the software
# the description will appear in the package information we can retrieve with `package`
description: "This is the GNU Hello program."
# sources may be multiple: you may want to add arbitrary files along the tarball
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
# we provide running dependencies: the program needs these to `run`
dependencies:
- gettext
# we provide build dependencies: the program needs these to `be compiled` for the package
# no need to require these dependencies for a simple user installing the program
build-dependencies:
- make
# if we want to add or override compilation options
options:
- configure: --disable-nls
# feels like déjà vu, right?
# "watch" code block helps to check if the recipe covers the last software version
@watch
curl 'https://ftp.gnu.org/gnu/hello/' -o- 2>/dev/null |
sed -n "/hello-.*\.tar\.gz/{s/\.tar\.gz.*//;s/.*hello-//;p}" |
tail -1
```
This was [a real example][recipe-hello], and not the simplest one.
Most of our recipes currently look like this:
```YAML
name: dhcpcd
version: 8.0.3
sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz
```
That's it.
Yes, we can add a few meta-data, but this is a working recipe.
Configuration, compilation and packaging are done without providing anything else.
The only required parameters are `name`, `sources` and `version`.
<a name="packaging-manual-operations"></a>
Sometimes, developers are assholes and force you to fix their build system.
When manual operations <u>really are required</u>, you can use `@configure`, `@build` and `@install` code blocks.
Let's see an example with a special snowflake… like `perl`.
```YAML
# in our example, we want to configure the build
@configure
# we set some script execution parameters: exit at any error, print everything you do
set -e -x
# we currently are in `/tmp/packaging/build-UUID/`
# now we enter the directory created by the tarball extraction, where the build occurs
cd perl-%{version} # we can interpolate variables with this: %{}
# Perl needs a few environment variables for configuration
BUILD_ZLIB=0
BUILD_BZIP2=0
BZIP2_LIB=/usr/lib
BZIP2_INCLUDE=/usr/include
export BUILD_ZLIB BUILD_BZIP2 BZIP2_LIB BZIP2_INCLUDE
# now we can launch the `configure` script…
# … but wait kids, that's not `configure`, that is `Configure` (see the capitalized C?)
# this is a totally different and very special script, but you should love it anyway
./Configure -des -Dcccdlflags='-fPIC' \
-Dcccdlflags='-fPIC' \
-Dccdlflags='-rdynamic' \
# ... some long and uninteresting list of very specific parameters because we are Perl, we are historic and stuff
```
Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems.
Non standard build operations happen from time to time, and code blocks help you overcome this.
When a lot of packages have the same workarounds, we think about detection and integration into `packaging`, so we can keep only specifics into recipes.
If you want to investigate a bit more, you can check [our recipe repository][recipes].
Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
<a name="packaging-own-repository"></a>
<red>TODO: talk about creating a repository.</red>
<a name="packaging-future"></a>
**Future of `packaging`.** *let's be even more declarative*<br />
As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit.
In the future, manual operations should be reduced even more:
- by adding a few other parameters to the environment<br />
This imply to check for patterns in the recipes and to provide workarounds.
- by adding new code blocks *before* the `configuration`, `build` and `install` steps<br />
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) and still keeping automatic operations: the best of the two worlds.
--- ---
@ -561,6 +701,7 @@ We currently aim at providing a rootfs with our tools, when we will have enough
[baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc [baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc
[baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile [baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile
[recipes]: https://git.baguette.netlib.re/Baguette/recipes/
[recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec [recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec
[recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec [recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec
[recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec [recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec