Slotting++, `service` review.
parent
3cda8dcbe3
commit
b469fec7d4
|
@ -504,20 +504,19 @@ Let's take an example.
|
||||||
# GNU Hello program
|
# GNU Hello program
|
||||||
name: hello # software name
|
name: hello # software name
|
||||||
version: 2.10 # software version
|
version: 2.10 # software version
|
||||||
release: 2 # recipe release: incremented when the recipe change for the current version of the software
|
release: 2 # recipe release: incremented when the recipe changes for the current version of the software
|
||||||
|
|
||||||
# the description will appear in the package information we can retrieve with `package`
|
# the description will appear in the package information we can retrieve with `package`
|
||||||
description: "This is the GNU Hello program."
|
description: "This is the GNU Hello program."
|
||||||
|
|
||||||
# sources may be multiple: you may want to add arbitrary files along the tarball
|
# sources may be multiple: you may want to add arbitrary files along the tarball (patches for example)
|
||||||
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
|
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
|
||||||
|
|
||||||
# we provide running dependencies: the program needs these to `run`
|
# we provide running dependencies: the program needs these to `run`
|
||||||
dependencies:
|
dependencies:
|
||||||
- gettext
|
- gettext
|
||||||
|
|
||||||
# we provide build dependencies: the program needs these to `be compiled` for the package
|
# we provide build dependencies: these programs are needed to compile our recipe, not to run the application
|
||||||
# no need to require these dependencies for a simple user installing the program
|
|
||||||
build-dependencies:
|
build-dependencies:
|
||||||
- make
|
- make
|
||||||
|
|
||||||
|
@ -540,17 +539,18 @@ Most of our recipes currently look like this:
|
||||||
name: dhcpcd
|
name: dhcpcd
|
||||||
version: 8.0.3
|
version: 8.0.3
|
||||||
sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz
|
sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz
|
||||||
|
description: "dhcp server"
|
||||||
```
|
```
|
||||||
That's it.
|
That's it.
|
||||||
Yes, we can add a few meta-data, but this is a working recipe.
|
Yes, we can add a few meta-data, but this is a working recipe.
|
||||||
Configuration, compilation and packaging are done without providing anything else.
|
Configuration, compilation and packaging are done without needing anything else.
|
||||||
The only required parameters are `name`, `sources` and `version`.
|
The only required parameters are `name`, `version` and `sources`.
|
||||||
|
|
||||||
|
|
||||||
<a name="packaging-manual-operations"></a>
|
<a name="packaging-manual-operations"></a>
|
||||||
Sometimes, developers are assholes and force you to fix their build system.
|
Sometimes, developers are assholes and force you to fix their build system.
|
||||||
When manual operations <u>really are required</u>, you can use `@configure`, `@build` and `@install` code blocks.
|
When manual operations <u>really are required</u>, you can use `@configure`, `@build` and `@install` code blocks.
|
||||||
Let's see an example with a special snowflake… like `perl`.
|
Let's see an example with a special snowflake… like `perl` which has a non-standard build system.
|
||||||
```YAML
|
```YAML
|
||||||
# in our example, we want to configure the build
|
# in our example, we want to configure the build
|
||||||
@configure
|
@configure
|
||||||
|
@ -579,7 +579,7 @@ Let's see an example with a special snowflake… like `perl`.
|
||||||
|
|
||||||
Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems.
|
Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems.
|
||||||
Non standard build operations happen from time to time, and code blocks help you overcome this.
|
Non standard build operations happen from time to time, and code blocks help you overcome this.
|
||||||
When a lot of packages have the same workarounds, we think about detection and integration into `packaging`, so we can keep only specifics into recipes.
|
If a lot of packages have the same workarounds, we might add detection and integration into `packaging`, so that only specifics are kept into recipes.
|
||||||
|
|
||||||
If you want to investigate a bit more, you can check [our recipe repository][recipes].
|
If you want to investigate a bit more, you can check [our recipe repository][recipes].
|
||||||
Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
|
Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
|
||||||
|
@ -588,24 +588,26 @@ Feel free to improve these recipes with meta-data, `@watch` code blocks… contr
|
||||||
**Future of `packaging`.** *let's be even more declarative*<br />
|
**Future of `packaging`.** *let's be even more declarative*<br />
|
||||||
As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
|
As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
|
||||||
Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit.
|
Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit.
|
||||||
In the future, manual operations should be reduced even more:
|
In the future, manual operations should be reduced even more by adding:
|
||||||
- by adding a few other parameters to the environment<br />
|
- a few other parameters to the environment<br />
|
||||||
This imply to check for patterns in the recipes and to provide workarounds.
|
This implies to check for patterns in the recipes and to provide workarounds.
|
||||||
- by adding new code blocks *before* the `configuration`, `build` and `install` steps<br />
|
- new code blocks *before* the `configuration`, `build` and `install` steps<br />
|
||||||
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) and still keeping automatic operations: the best of the two worlds.
|
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) while still keeping automatic operations.
|
||||||
|
|
||||||
Another possible addition to `packaging` could be to take cross-OS recipes as first-class citizen.
|
Another possible addition to `packaging` could be to take cross-OS recipe as first-class citizen.
|
||||||
It was part of the design process of spec files: `named sections` can be used to discriminate information based on the target OS.
|
It was part of the design process of spec files: `named sections` can be used to discriminate information based on the target OS.
|
||||||
An example:
|
An example:
|
||||||
|
|
||||||
```YAML
|
```YAML
|
||||||
%OS Ubuntu-20.04
|
%OS Ubuntu-20.04
|
||||||
dependencies: libxxx
|
dependencies: libxxx
|
||||||
|
```
|
||||||
|
```YAML
|
||||||
%OS Debian-12
|
%OS Debian-12
|
||||||
dependencies: lib-with-specific-naming-standard-for-reasons
|
dependencies: lib-with-specific-naming-standard-for-reasons
|
||||||
```
|
```
|
||||||
|
|
||||||
This way, bootstrapping an application and providing it to any system <u>with their own tools</u> could be easy.
|
This way, bootstrapping an application and providing it to any system <u>with its own tools</u> could be easy.
|
||||||
It's actually what we did during the Baguette's bootstrap.
|
It's actually what we did during the Baguette's bootstrap.
|
||||||
Providing universal recipes could even become a game for patient system administrators.
|
Providing universal recipes could even become a game for patient system administrators.
|
||||||
|
|
||||||
|
@ -619,16 +621,16 @@ Service management often comes with:
|
||||||
- default configuration files, users should learn how to configure them and do it manually
|
- default configuration files, users should learn how to configure them and do it manually
|
||||||
- default user and group, so two instances may have security-involved issues
|
- default user and group, so two instances may have security-involved issues
|
||||||
- a single possible instance, otherwise the configuration has to be heavily changed
|
- a single possible instance, otherwise the configuration has to be heavily changed
|
||||||
- root-only management, simple users rarely run their own services (except on systemd, kuddos for once)
|
- root-only management, simple users rarely run their own services (except on systemd, kudos for once)
|
||||||
- no domain management
|
- no domain management
|
||||||
|
|
||||||
These shortcomings imply manual configuration, scripting to manage databases and users, specific tooling for each database and service: this is heavy machinery.
|
These shortcomings imply manual configuration, scripting to manage databases and users, specific tooling for each database and service: this is heavy machinery.
|
||||||
To overcome drawbacks of having simplistic tools, sys-admins developed all kind of monstrous architectures.
|
To overcome drawbacks of having simplistic tools, sys-admins developed all kind of monstrous architectures.
|
||||||
- **LXC** *chroot + network + software limits*
|
- **LXC** *chroot + network + software limits*
|
||||||
- **Qemu + KVM, Xen** *let's add software mimicking hardware's complexity to the mix, telling everyone it's for security and simplicity*
|
- **Qemu + KVM, Xen** *let's add software mimicking hardware's complexity to the mix, telling everyone it's for security and simplicity*
|
||||||
- **docker** *I don't know how to do simple applications nor packages, so I ship my whole dev environment*<br />
|
- **docker** *I don't know how to do simple applications nor packages, so I give to you my whole dev environment*<br />
|
||||||
Note: we have to admit, packaging on most OS is painful for absolute no good reason.
|
Note: we have to admit, packaging on most OS is painful for absolutely no good reason.
|
||||||
- **Chef and Puppet** *the 500 MB running Ruby's code on our virtual machines just to check for new configuration is okay, memory is cheap right?*<br />
|
- **Chef and Puppet** *the 500 MB running Ruby code on our virtual machines just to check for configuration updates is okay 'cause memory is cheap, right?*<br />
|
||||||
We talk about the importance of security from time to time, but running a software designed by people telling [it's okay not to free memory][ruby-memory-bp] is far from being wise.
|
We talk about the importance of security from time to time, but running a software designed by people telling [it's okay not to free memory][ruby-memory-bp] is far from being wise.
|
||||||
- **Ansible** *templating your applications… from another machine*<br />
|
- **Ansible** *templating your applications… from another machine*<br />
|
||||||
As Chef and Puppet, ansible provides templating for applications, this time configuration propagation is **way** simpler since it uses well-known, trusted and loved ssh.
|
As Chef and Puppet, ansible provides templating for applications, this time configuration propagation is **way** simpler since it uses well-known, trusted and loved ssh.
|
||||||
|
@ -637,8 +639,8 @@ To overcome drawbacks of having simplistic tools, sys-admins developed all kind
|
||||||
As for many tools in IT: this is not for simple users.
|
As for many tools in IT: this is not for simple users.
|
||||||
|
|
||||||
Simple users:
|
Simple users:
|
||||||
1. should be only required to provide absolute necessary information for their services
|
1. should only have to provide absolutely necessary information for their services
|
||||||
2. should be able to run as many services they want
|
2. should be able to run as many services as they want
|
||||||
3. shouldn't have to learn configuration syntax for their services
|
3. shouldn't have to learn configuration syntax for their services
|
||||||
4. shouldn't be afraid of updates
|
4. shouldn't be afraid of updates
|
||||||
5. ... and should have BACKUPS! Where are they? We should have that **by default** on our systems over 20 years ago.
|
5. ... and should have BACKUPS! Where are they? We should have that **by default** on our systems over 20 years ago.
|
||||||
|
@ -649,7 +651,7 @@ And advanced users should have an uncomplicated CLI tool to do that.
|
||||||
**Let's take an example with `service`** *better than a thousand words*
|
**Let's take an example with `service`** *better than a thousand words*
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# We want a wordpress service, proxied by a nginx and using postgresql as DBMS
|
# We want a wordpress service, proxied by an nginx and using postgresql as DBMS
|
||||||
|
|
||||||
# 1. we add an nginx
|
# 1. we add an nginx
|
||||||
$ service add nginx
|
$ service add nginx
|
||||||
|
@ -682,34 +684,41 @@ A bit of explanation:
|
||||||
The configuration of a service is made when `service` starts it.
|
The configuration of a service is made when `service` starts it.
|
||||||
In this case:
|
In this case:
|
||||||
1. `nginx` is started as a proxy for `example.com`
|
1. `nginx` is started as a proxy for `example.com`
|
||||||
2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
|
2. `postgresql` is started, its internal directories and files are created, then a user, a database and a password for `wordpress` are created
|
||||||
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
|
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
|
||||||
|
|
||||||
Stopping a service also stops its dependencies, if they aren't required elsewhere (or that we explicitely said to keep them running).
|
Stopping a service also stops its dependencies, unless specified otherwise.
|
||||||
|
Of course, a service is not stopped if it is required elsewhere.
|
||||||
|
|
||||||
|
|
||||||
Here are a few functionalities `service` brings.
|
Here are a few functionalities `service` brings.
|
||||||
1. **uncomplicated domain-wise service configuration**
|
1. **uncomplicated service configuration with shared information**<br />
|
||||||
|
Services can share:
|
||||||
|
- passwords (which are auto-generated)
|
||||||
|
- user names (which follow a naming convention)
|
||||||
|
- port numbers (auto-attributed unless specified)
|
||||||
|
- ... etc.
|
||||||
|
|
||||||
2. **templates** *configuration files are generated by templates and user data*<br />
|
2. **templates** *configuration files are generated by templates and a few parameters*<br />
|
||||||
When we want a software, for instance a blog, we want to provide the minimum informations it requires to work and that's it.
|
When we want a software, for instance a blog, we want to provide the minimum informations it requires to work and that's it.
|
||||||
When `service` starts a service, it verifies if its configuration file is installed and up-to-date, or create it.
|
When `service` starts a service, it verifies if its configuration file is installed and up-to-date, and creates it otherwise.
|
||||||
Users shouldn't need to manually change the configuration.
|
Users shouldn't need to manually change the configuration.
|
||||||
<u>Syntax may change at any time without breaking a single service</u>, since the configuration will smoothly be regenerated with useful information at start-up.
|
<u>Syntax may change at any time without breaking a single service</u>, since the configuration will smoothly be regenerated with useful information at start-up.
|
||||||
|
|
||||||
3. **environments**<br />
|
3. **environments**<br />
|
||||||
Each service can be installed in a specific environment (read: a custom rootfs, a virtual machine, etc.).
|
Each service can be installed in a specific environment, such as a custom rootfs, a virtual machine, etc.
|
||||||
|
|
||||||
4. **automatic user and group creation**<br />
|
4. **automatic user and group creation**<br />
|
||||||
Each service needs to be separated from other users on the system, for security reasons.
|
Each service needs to run with a unique user, dedicated to it, different from other users on the system for security reasons.
|
||||||
Let's make it the default.
|
Let's make it the default behavior.
|
||||||
|
|
||||||
5. **tokens** *service a needs b, b needs c, when a starts, b and c start and are configured*<br />
|
5. **tokens** *service a needs b, b needs c, when a starts, b and c are configured then started*<br />
|
||||||
`service` knows the relation between services, and use it to configure them through the templates.
|
`service` knows the relations between services, and uses them to configure services through the templates.
|
||||||
*everything is smoother now*
|
*everything is smoother now*
|
||||||
|
|
||||||
6. **automatic backup solution**<br />
|
6. **automatic backup solution**<br />
|
||||||
Since we know each running database, service configuration and data directories, we can backup everything once a backup server has been configured.
|
Since we know each running database, service configuration and data directories, we can backup everything.
|
||||||
|
We just need a backup server to be configured.
|
||||||
Of course, we can add a lot of parameters to have a fine-grained backup solution.
|
Of course, we can add a lot of parameters to have a fine-grained backup solution.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
@ -719,8 +728,7 @@ $ backup add ssh:user@example.com:/srv/backup
|
||||||
|
|
||||||
7. **unified way to configure the system** *best user experience possible*<br />
|
7. **unified way to configure the system** *best user experience possible*<br />
|
||||||
It alleviates the need for manual configuration.
|
It alleviates the need for manual configuration.
|
||||||
For example, adding a Wordpress service will automatically change the `nginx` configuration, create a new database and a new user in `mariadb` for this specific service.
|
The CLI tool is the same for every provided service.<br />
|
||||||
If several `nginx` are required, ports will be registered and automatically managed for each instance, no need for user input.<br />
|
|
||||||
Behind the scene, it's a simple token system with configuration templating!<br />
|
Behind the scene, it's a simple token system with configuration templating!<br />
|
||||||
<u>No heavy machinery here, and we'll keep it that way.</u>
|
<u>No heavy machinery here, and we'll keep it that way.</u>
|
||||||
|
|
||||||
|
@ -762,7 +770,7 @@ ports: http
|
||||||
%configuration gitea.cfg
|
%configuration gitea.cfg
|
||||||
```
|
```
|
||||||
|
|
||||||
*We **currently** uses a bit more configuration for the database, but we will get there at some point.*
|
*We **currently** use a bit more configuration for the database, but we will get there at some point.*
|
||||||
|
|
||||||
Now, a quick look at its configuration template. *just a sample, we'll skip most, don't worry*<br />
|
Now, a quick look at its configuration template. *just a sample, we'll skip most, don't worry*<br />
|
||||||
Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
|
Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
|
||||||
|
@ -960,7 +968,8 @@ So, we need a language for both administration dashboard and online services, he
|
||||||
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
|
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
|
||||||
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
|
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
|
||||||
|
|
||||||
**Problem:**
|
### Current set of problems
|
||||||
|
|
||||||
- what happens when two programs need a different version of a library?<br />
|
- what happens when two programs need a different version of a library?<br />
|
||||||
The installation of both may no be possible.
|
The installation of both may no be possible.
|
||||||
See python from version 2 to 3 as an example: developers knew it will break OS systems.
|
See python from version 2 to 3 as an example: developers knew it will break OS systems.
|
||||||
|
@ -971,18 +980,46 @@ In the long run, software and libraries change, which is no big deal since maint
|
||||||
*see companies running decade-old OSs and databases*
|
*see companies running decade-old OSs and databases*
|
||||||
|
|
||||||
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
|
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
|
||||||
|
|
||||||
|
### What is slotting?
|
||||||
|
|
||||||
|
Slotting is a lot like repositories, except that repositories provide packages in the same prefixes than your base system.
|
||||||
|
|
||||||
|
**Without slotting**<br />
|
||||||
|
Let's take an example with simple repositories.
|
||||||
|
You add a non-official repository for <blue>my-overly-awesome-game</blue> to your Debian system.
|
||||||
|
This newly installed program will be in `/usr/bin`, as every other program.
|
||||||
|
1. what if the game requires libraries?<br />
|
||||||
|
These libraries are installed in `/usr/lib`.
|
||||||
|
2. what if the game requires libraries that are not in the official repository?<br />
|
||||||
|
Either the repository for <blue>my-overly-awesome-game</blue> provides them directly, or you can find another repository providing them.
|
||||||
|
In <u>both cases</u> these libraries will end-up in `/usr/lib`.
|
||||||
|
|
||||||
|
|
||||||
|
**With slotting**<br />
|
||||||
|
With slotting, the program will be in `/usr/`<blue>my-overly-awesome-game</blue>`/bin`.
|
||||||
|
1. What if requires libraries?
|
||||||
|
These libraries will be installed in your base system so any of your non-official slot can use them.
|
||||||
|
2. What if the required libraries aren't available in the official `baguette` slot?
|
||||||
|
Either the game slot provides them, or they are in another slot.
|
||||||
|
In <u>both cases</u> the base system won't change a bit.
|
||||||
|
|
||||||
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
|
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
|
||||||
Here, the slot is `baguette`.
|
Here, the slot is `baguette`.
|
||||||
Any package outside the official ones are in another named slot.
|
Any package outside the official ones are in another named slot.
|
||||||
|
|
||||||
Wanna support Python 2.7 **for life**? Just maintain a `python-2.7` slot.
|
Wanna support Python 2.7 **for life**?
|
||||||
It will be available for everyone.
|
Just maintain a `python-2.7` slot and tell the world!
|
||||||
|
If BaguetteOS do not provide required libraries for your slot, just add them in your slot.
|
||||||
|
|
||||||
**This is nothing new, however not often used, and still maybe the best way to handle the problem.**
|
**This is nothing new, however not used directly in OSs, and still maybe the best way to handle the problem.**
|
||||||
|
|
||||||
Others are doing it: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br />
|
|
||||||
|
### Why not use X?
|
||||||
|
|
||||||
|
Others are doing slotting too: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br />
|
||||||
They all use slotting... *but*.
|
They all use slotting... *but*.
|
||||||
Since they are is *yet another package manager for your system*, you need to install all your snap software dependencies from their own dependency tree.
|
Since they are *yet another package manager for your system*, you need to install all your software dependencies from their own dependency tree.
|
||||||
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
|
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
|
||||||
|
|
||||||
**Having an alternate package manager for your system is a fancy way of saying:**<br />
|
**Having an alternate package manager for your system is a fancy way of saying:**<br />
|
||||||
|
|
Loading…
Reference in New Issue