Baguette
/
website
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Slotting++, `service` review.

master
Philippe PITTOLI 2020-04-26 23:25:57 +02:00
parent 3cda8dcbe3
commit b469fec7d4
1 changed files with 79 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

121
content/baguette/index.md
View File

@ -504,20 +504,19 @@ Let's take an example.
# GNU Hello program
name: hello # software name
version: 2.10 # software version
release: 2 # recipe release: incremented when the recipe change for the current version of the software
release: 2 # recipe release: incremented when the recipe changes for the current version of the software
# the description will appear in the package information we can retrieve with `package`
description: "This is the GNU Hello program."
# sources may be multiple: you may want to add arbitrary files along the tarball
# sources may be multiple: you may want to add arbitrary files along the tarball (patches for example)
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
# we provide running dependencies: the program needs these to `run`
dependencies:
- gettext
# we provide build dependencies: the program needs these to `be compiled` for the package
# no need to require these dependencies for a simple user installing the program
# we provide build dependencies: these programs are needed to compile our recipe, not to run the application
build-dependencies:
- make
@ -540,17 +539,18 @@ Most of our recipes currently look like this:
name: dhcpcd
version: 8.0.3
sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz
description: "dhcp server"
```
That's it.
Yes, we can add a few meta-data, but this is a working recipe.
Configuration, compilation and packaging are done without providing anything else.
The only required parameters are `name`, `sources` and `version`.
Configuration, compilation and packaging are done without needing anything else.
The only required parameters are `name`, `version` and `sources`.
<a name="packaging-manual-operations"></a>
Sometimes, developers are assholes and force you to fix their build system.
When manual operations <u>really are required</u>, you can use `@configure`, `@build` and `@install` code blocks.
Let's see an example with a special snowflake… like `perl`.
Let's see an example with a special snowflake… like `perl` which has a non-standard build system.
```YAML
# in our example, we want to configure the build
@configure
@ -579,7 +579,7 @@ Let's see an example with a special snowflake… like `perl`.
Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems.
Non standard build operations happen from time to time, and code blocks help you overcome this.
When a lot of packages have the same workarounds, we think about detection and integration into `packaging`, so we can keep only specifics into recipes.
If a lot of packages have the same workarounds, we might add detection and integration into `packaging`, so that only specifics are kept into recipes.
If you want to investigate a bit more, you can check [our recipe repository][recipes].
Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
@ -588,24 +588,26 @@ Feel free to improve these recipes with meta-data, `@watch` code blocks… contr
**Future of `packaging`.** *let's be even more declarative*<br />
As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit.
In the future, manual operations should be reduced even more:
- by adding a few other parameters to the environment<br />
This imply to check for patterns in the recipes and to provide workarounds.
- by adding new code blocks *before* the `configuration`, `build` and `install` steps<br />
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) and still keeping automatic operations: the best of the two worlds.
In the future, manual operations should be reduced even more by adding:
- a few other parameters to the environment<br />
This implies to check for patterns in the recipes and to provide workarounds.
- new code blocks *before* the `configuration`, `build` and `install` steps<br />
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) while still keeping automatic operations.
Another possible addition to `packaging` could be to take cross-OS recipes as first-class citizen.
Another possible addition to `packaging` could be to take cross-OS recipe as first-class citizen.
It was part of the design process of spec files: `named sections` can be used to discriminate information based on the target OS.
An example:
```YAML
%OS Ubuntu-20.04
dependencies: libxxx
```
```YAML
%OS Debian-12
dependencies: lib-with-specific-naming-standard-for-reasons
```
This way, bootstrapping an application and providing it to any system <u>with their own tools</u> could be easy.
This way, bootstrapping an application and providing it to any system <u>with its own tools</u> could be easy.
It's actually what we did during the Baguette's bootstrap.
Providing universal recipes could even become a game for patient system administrators.
@ -619,16 +621,16 @@ Service management often comes with:
- default configuration files, users should learn how to configure them and do it manually
- default user and group, so two instances may have security-involved issues
- a single possible instance, otherwise the configuration has to be heavily changed
- root-only management, simple users rarely run their own services (except on systemd, kuddos for once)
- root-only management, simple users rarely run their own services (except on systemd, kudos for once)
- no domain management
These shortcomings imply manual configuration, scripting to manage databases and users, specific tooling for each database and service: this is heavy machinery.
To overcome drawbacks of having simplistic tools, sys-admins developed all kind of monstrous architectures.
- **LXC** *chroot + network + software limits*
- **Qemu + KVM, Xen** *let's add software mimicking hardware's complexity to the mix, telling everyone it's for security and simplicity*
- **docker** *I don't know how to do simple applications nor packages, so I ship my whole dev environment*<br />
Note: we have to admit, packaging on most OS is painful for absolute no good reason.
- **Chef and Puppet** *the 500 MB running Ruby's code on our virtual machines just to check for new configuration is okay, memory is cheap right?*<br />
- **docker** *I don't know how to do simple applications nor packages, so I give to you my whole dev environment*<br />
Note: we have to admit, packaging on most OS is painful for absolutely no good reason.
- **Chef and Puppet** *the 500 MB running Ruby code on our virtual machines just to check for configuration updates is okay 'cause memory is cheap, right?*<br />
We talk about the importance of security from time to time, but running a software designed by people telling [it's okay not to free memory][ruby-memory-bp] is far from being wise.
- **Ansible** *templating your applications… from another machine*<br />
As Chef and Puppet, ansible provides templating for applications, this time configuration propagation is **way** simpler since it uses well-known, trusted and loved ssh.
@ -637,8 +639,8 @@ To overcome drawbacks of having simplistic tools, sys-admins developed all kind
As for many tools in IT: this is not for simple users.
Simple users:
1. should be only required to provide absolute necessary information for their services
2. should be able to run as many services they want
1. should only have to provide absolutely necessary information for their services
2. should be able to run as many services as they want
3. shouldn't have to learn configuration syntax for their services
4. shouldn't be afraid of updates
5. ... and should have BACKUPS! Where are they? We should have that **by default** on our systems over 20 years ago.
@ -649,7 +651,7 @@ And advanced users should have an uncomplicated CLI tool to do that.
**Let's take an example with `service`** *better than a thousand words*
```sh
# We want a wordpress service, proxied by a nginx and using postgresql as DBMS
# We want a wordpress service, proxied by an nginx and using postgresql as DBMS
# 1. we add an nginx
$ service add nginx
@ -682,34 +684,41 @@ A bit of explanation:
The configuration of a service is made when `service` starts it.
In this case:
1. `nginx` is started as a proxy for `example.com`
2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
2. `postgresql` is started, its internal directories and files are created, then a user, a database and a password for `wordpress` are created
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
Stopping a service also stops its dependencies, if they aren't required elsewhere (or that we explicitely said to keep them running).
Stopping a service also stops its dependencies, unless specified otherwise.
Of course, a service is not stopped if it is required elsewhere.
Here are a few functionalities `service` brings.
1. **uncomplicated domain-wise service configuration**
1. **uncomplicated service configuration with shared information**<br />
Services can share:
- passwords (which are auto-generated)
- user names (which follow a naming convention)
- port numbers (auto-attributed unless specified)
- ... etc.
2. **templates** *configuration files are generated by templates and user data*<br />
2. **templates** *configuration files are generated by templates and a few parameters*<br />
When we want a software, for instance a blog, we want to provide the minimum informations it requires to work and that's it.
When `service` starts a service, it verifies if its configuration file is installed and up-to-date, or create it.
When `service` starts a service, it verifies if its configuration file is installed and up-to-date, and creates it otherwise.
Users shouldn't need to manually change the configuration.
<u>Syntax may change at any time without breaking a single service</u>, since the configuration will smoothly be regenerated with useful information at start-up.
3. **environments**<br />
Each service can be installed in a specific environment (read: a custom rootfs, a virtual machine, etc.).
Each service can be installed in a specific environment, such as a custom rootfs, a virtual machine, etc.
4. **automatic user and group creation**<br />
Each service needs to be separated from other users on the system, for security reasons.
Let's make it the default.
Each service needs to run with a unique user, dedicated to it, different from other users on the system for security reasons.
Let's make it the default behavior.
5. **tokens** *service a needs b, b needs c, when a starts, b and c start and are configured*<br />
`service` knows the relation between services, and use it to configure them through the templates.
5. **tokens** *service a needs b, b needs c, when a starts, b and c are configured then started*<br />
`service` knows the relations between services, and uses them to configure services through the templates.
*everything is smoother now*
6. **automatic backup solution**<br />
Since we know each running database, service configuration and data directories, we can backup everything once a backup server has been configured.
Since we know each running database, service configuration and data directories, we can backup everything.
We just need a backup server to be configured.
Of course, we can add a lot of parameters to have a fine-grained backup solution.
```sh
@ -719,8 +728,7 @@ $ backup add ssh:user@example.com:/srv/backup
7. **unified way to configure the system** *best user experience possible*<br />
It alleviates the need for manual configuration.
For example, adding a Wordpress service will automatically change the `nginx` configuration, create a new database and a new user in `mariadb` for this specific service.
If several `nginx` are required, ports will be registered and automatically managed for each instance, no need for user input.<br />
The CLI tool is the same for every provided service.<br />
Behind the scene, it's a simple token system with configuration templating!<br />
<u>No heavy machinery here, and we'll keep it that way.</u>
@ -762,7 +770,7 @@ ports: http
%configuration gitea.cfg
```
*We **currently** uses a bit more configuration for the database, but we will get there at some point.*
*We **currently** use a bit more configuration for the database, but we will get there at some point.*
Now, a quick look at its configuration template. *just a sample, we'll skip most, don't worry*<br />
Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
@ -960,7 +968,8 @@ So, we need a language for both administration dashboard and online services, he
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
**Problem:**
### Current set of problems
- what happens when two programs need a different version of a library?<br />
The installation of both may no be possible.
See python from version 2 to 3 as an example: developers knew it will break OS systems.
@ -971,18 +980,46 @@ In the long run, software and libraries change, which is no big deal since maint
*see companies running decade-old OSs and databases*
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
### What is slotting?
Slotting is a lot like repositories, except that repositories provide packages in the same prefixes than your base system.
**Without slotting**<br />
Let's take an example with simple repositories.
You add a non-official repository for <blue>my-overly-awesome-game</blue> to your Debian system.
This newly installed program will be in `/usr/bin`, as every other program.
1. what if the game requires libraries?<br />
These libraries are installed in `/usr/lib`.
2. what if the game requires libraries that are not in the official repository?<br />
Either the repository for <blue>my-overly-awesome-game</blue> provides them directly, or you can find another repository providing them.
In <u>both cases</u> these libraries will end-up in `/usr/lib`.
**With slotting**<br />
With slotting, the program will be in `/usr/`<blue>my-overly-awesome-game</blue>`/bin`.
1. What if requires libraries?
These libraries will be installed in your base system so any of your non-official slot can use them.
2. What if the required libraries aren't available in the official `baguette` slot?
Either the game slot provides them, or they are in another slot.
In <u>both cases</u> the base system won't change a bit.
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
Here, the slot is `baguette`.
Any package outside the official ones are in another named slot.
Wanna support Python 2.7 **for life**? Just maintain a `python-2.7` slot.
It will be available for everyone.
Wanna support Python 2.7 **for life**?
Just maintain a `python-2.7` slot and tell the world!
If BaguetteOS do not provide required libraries for your slot, just add them in your slot.
**This is nothing new, however not often used, and still maybe the best way to handle the problem.**
**This is nothing new, however not used directly in OSs, and still maybe the best way to handle the problem.**
Others are doing it: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br />
### Why not use X?
Others are doing slotting too: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br />
They all use slotting... *but*.
Since they are is *yet another package manager for your system*, you need to install all your snap software dependencies from their own dependency tree.
Since they are *yet another package manager for your system*, you need to install all your software dependencies from their own dependency tree.
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
**Having an alternate package manager for your system is a fancy way of saying:**<br />