Your application or your library lacks a build system? Here a [tool to create makefiles](#build.zsh).
Your application or your library lacks a build system? Here is a [tool to create makefiles](#build.zsh).
It works on any language. *yes, even that one*
**Stable and development versions: same thing.** *slotting, again and again*<br/>
@ -303,53 +303,61 @@ firefox-79.0-r8.pkg
# 3. BaguetteOS: custom tools
**Simple ideas, simple implementations.**<br/>
Keep reminding to yourself while reading that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br/>
Keep reminding to yourself while reading this section that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br/>
Also, they can easily run on other systems: nothing here is OS specific.
[Feel free to provide a feedback.](#contact)
Here a few pointers:
Here is a few pointers:
- [spec format](#spec-format)
- [package](#package)
- [packaging](#packaging)
- [service](#service)
- [build.zsh](#build.zsh)
- [… and a few other tools](#other-tools)
---
### Main BaguetteOS tools
## Main BaguetteOS tools
<aname="spec-format"></a>
**Spec files.** *our declarative format*<br/>
Before presenting our tools, here a file format named `spec` that we tend to use when relevant.
It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`).
The `spec format` is only composed of variables, lists, code block and named sections.
Here a quick example.
### Spec files: our declarative format
Before presenting our tools, here is a file format named `spec` that we use when relevant.
It is *declarative*: <u>we do not write instructions on *how* to do things</u> (copy this file here, download this, etc.).
Instead, <u>we describe what something is</u>: the URL of the project is `https://example.com/xxx`, for example.
The `spec format` is only composed of variables, lists, code blocks and named sections.
Here is a quick example.
```yaml
# This is a comment
# This is a simple variable instanciation
variable: value
my-variable: value
# This is a inlined list
list: a, b, c
# This is an inlined list
my-list: a, b, c
# This is a multiline list
list:
my-list:
- a
- b
- c
```
This looks a lot like YAML, but now we add `code block` to it: because sometimes we <u>do</u> want to tell instructions.
Up to this point, this looks a lot like YAML, but now we add `code blocks` to it: because sometimes we <u>do</u> want to tell instructions.
```yaml
# We have the URL of the tarballs for a software
software-url: 'https://example.com/my-software/'
# ... and we want to get its last version number
# "@watch" is a block, the name "watch" has a meaning in the application reading the file
Sometimes, we want to refer to a file (or directory) and add metadata to it: here`named sections`.
Sometimes, we want to refer to a file (or directory) and add metadata to it through`named sections`.
```yaml
# as for "@watch", "%configuration" has a meaning in the application reading the file
# as for "@watch", "%configuration" is a keyword for the application
# this time, the block has an arbitrary name
%configuration postgresql.conf
# within a named section, we find simple declarations as outside the block
# within a named section, variables are declared in the same way as outside the block
name: database configuration
creation-command: my-script.sh -a -b -c
```
Next, the usage in practice: [packaging](#packaging), [service](#service).
[Come back to top](#top)
[Back to top](#top)
---
<aname="package"></a>
**[Package][package]: our package manager.**<br/>
`Package` covers the basics: install, remove, search and provide informations about a package.
### [Package][package]: our package manager
`Package` covers the basics: install, remove, search and provide information about a package.
`Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example.
Package provides slotting by default: no need for custom environments for each software.
Packages format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database format contains `world`, `installed`, `[package-name]/[slot]/manifest` and `[package-name]/[slot]/meta.spec`.
Package format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database format contains:
- `world`, the file containing the list of available packages
- `installed`, the list of installed packages
- `[package-name]/[slot]/manifest`, the manifest of the installed package
- `[package-name]/[slot]/meta.spec`, the meta-data of the installed package
Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on).
`Package` configuration consists of:
- a list of repositories
- authorized package signing keys
- packaging variables (cflags, makeflags, and so on)
Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code.
Finally, `Package` can easily be expanded, as it is only a few hundred lines of Crystal code.
### [Packaging][packaging]: the way to create packages for BaguetteOS
Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries.
As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries.
As BaguetteOS is designed to provide quickly usable systems, we choose to provide binaries.
`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
`Packaging` has a few advantages compared to most used packaging tools:
- **declarative recipes abstract OS specifics**<br/>
The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies.
This only requires to provide a back-end for the target package manager.
- **auto-split in different packages**<br/>
The same recipe may work for many **native packaging systems** (on many OSs), as long as packagers provide the right target running dependencies.
This only requires to provide a back-end for the package format of the target package manager (a back-end to write `.deb` files, for instance).
- **packages are split automatically**<br/>
We need to separate binaries, libraries and documentation in different packages, so we can only install what's needed.
Slow and testing systems only require strict minimum.
- **auto-strip of binaries and libraries**<br/>
- **binaries and libraries are stripped by default**<br/>
By default, a running system does not require debug symbols in its binaries.
- **recipe readability is great**<br/>
A few variable declarations are better than a dozen lines of code.
- **trivial shell script patterns become automated**<br/>
`Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project.
- **tooling may evolve, very few recipes will be require to change**<br/>
`Autotools` and `cmake` build systems are auto-detected; packagers should only provide specific parameters for each project.
- **tooling may change, recipes won't**<br/>
Everybody wants to change its build system?
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`.
- **it creates hash'ed and signed packages by default**<br/>
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just for `packaging`.
- **packages are hashed and signed by default, no extra-tooling**<br/>
You need your own set of cryptographic keys, which is created at first use.
- **it creates repositories (automatic at first compilation), helping people maintaining their own set of tools**<br/>
- **repositories are created automatically at first compilation, helping people maintaining their own set of tools**<br/>
Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
That is that simple.
It's <u>that</u> simple.
<imgsrc="/shell-scripting-is-lava.png"alt="shell script is lava"/>
<aname="packaging-build-env"></a>
**`Packaging` build environments**<br/>
`Packages` creates build environments to test packages before validation.
**`Packaging`'s build environments**<br/>
`Packaging` uses`package` to create build environments and to test packages before validation.
It works as follow:
1. creation of a `/tmp/packaging/build-UUID/` directory
1. a `/tmp/packaging/build-UUID/` directory is created
2. sources are downloaded, extracted then compiled<br/>
Recipes and [`packaging` host configuration](#packaging-host-config) may add parameters to it: adding steps before compilation, changing `configure` arguments, etc.
The package is compiled for a specific slot, by default `/usr/baguette`.
Recipes and [`packaging` host configuration](#packaging-host-config) may change parameters to the build: adding steps before compilation, changing `configure` arguments, changing the default slotting (`/usr/baguette`), etc.
3. compiled applications and libraries are put in `/tmp/packaging/build-UUID/root` which is used to create the final package
`Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
In this case, `package` only installs the minimal set of binaries required by the package to build.
Build environments are low-cost since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
`Packaging` only installs the minimal set of binaries required by the package to build.
Besides the target application, a building environment size is <u>only a few kilobytes</u>.
# prefixes for `packaging` running environment and child processes
# = where to search for binaries and libraries for the build
prefixes:
# the first prefix is the default slot used for building application
- /usr/baguette/
- /
# list of environment variables we want to have when building
environment:
# you may choose another compiler, provide some CFLAGS, etc.
# we may choose another compiler, provide some CFLAGS, etc.
- CC: clang
- CFLAGS: -Os -Wall
# next three have special meaning
# next three parameters have special meaning
# to provide parameters to the `./configure` script when building
- configure: --disable-nls --without-gettext
@ -474,11 +491,11 @@ environment:
package-manager: package
```
That's it. You know all about `packaging` configuration.
These parameters may be override by recipes.
These parameters may be overridden by recipes.
<aname="recipes"></a>
**`Packaging` recipes.** *we need to create packages*<br/>
A recipe is the way to reproduce something; here we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
A recipe is the way to reproduce something; here, we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
This means at least having a name for the software and a version (they appear in the package name) and sources (software code).
Let's take an example.
@ -592,12 +609,12 @@ This way, bootstrapping an application and providing it to any system <u>with th
It's actually what we did during the Baguette's bootstrap.
Providing universal recipes could even become a game for patient system administrators.
[Come back to top](#top)
[Back to top](#top)
---
<aname="service"></a>
**[Service][service]: service management.** *not just `kill` or start/stop/status wrapper*<br/>
### [Service][service]: service management. <side-note>*not just `kill` or start/stop/status wrapper*</side-note>
Service management often comes with:
- default configuration files, users should learn how to configure them and do it manually
- default user and group, so two instances may have security-involved issues
@ -655,7 +672,7 @@ $ service start wordpress
```
A bit of explanation:
1. first, we add `nginx` to the list of service we want on our system
1. first, we add `nginx` to the list of services we want on our system
2. same thing with `postgresql`
3. then we add `wordpress` and we pass parameters to the service configuration: `domain`, `http` and `database`<br/>
`domain` for the domain name, `http` for the HTTP proxy and then the `database` back-end.
@ -668,7 +685,7 @@ A bit of explanation:
2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
Stopping a service also stops its dependency, if they aren't required elsewhere (or that we explicitely said to keep them running).
Stopping a service also stops its dependencies, if they aren't required elsewhere (or that we explicitely said to keep them running).
Don't like Makefiles? <red>We may even add some other back-ends.</red>
[Back to top](#top)
---
<aname="libipc"></a>
**[LibIPC][libipc]: an IPC communication library** *nothing new, yet it still feels fresh*<br/>
### [LibIPC][libipc]: an IPC communication library <side-note>*nothing new, yet it still feels fresh*</side-note>
We use this communication library between our services.
@ -833,33 +912,34 @@ Remote remote communications are transparent.
- any client can join remote services via any communication protocol
- any service is implicitly accessible from anywhere, anyhow
[Back to top](#top)
---
<aname="build.zsh"></a>
<red>TODO.</red>
**[Build.zsh][build.zsh]: makefile creation.** *for mere mortals*<br/>
Build.zsh creates a makefile from a simple declarative configuration file.
It can replace most build systems.
<aname="other-tools"></a>
## Other tools
---
### [tap-aggregator][tap-aggregator]: quality assurance & test results aggregation
<red>TODO.</red>
**[tap-aggregator][tap-aggregator]: quality assurance & test results aggregation**<br/>
[Back to top](#top)
---
<red>TODO: better explanation.</red>
**[Webhooksd][webhooksd]: verify recipes.**<br/>
### [Webhooksd][webhooksd]: verify recipes.
Webhooksd provides an automatic verification of the recipes, based on new application or library version.
Paired with a build system, new recipes received in the repository create packages for a couple of architectures (x86_64, ARM, others will follow).
[Back to top](#top)
---
## Still in discussion
For the simple users, we want to provide an unified web interface to manage the system and online services.
We currently use `Crystal` to work on service back-ends for a variety of projects, we are satisfied on a technical level and we are productive, it is highly probable we continue using it.
The front-end is still in discussion: we currently use `livescript` and it is way more usable than plain JS, but it is painful to debug.
So, we need a language for both administration dashboard and online services, here some examples we find interesting for the job:
So, we need a language for both administration dashboard and online services, here are some examples we find interesting for the job:
- Purescript
- haskell-like syntax, with a smaller set of notions
- strongly typed, with type inference
@ -880,21 +960,62 @@ So, we need a language for both administration dashboard and online services, he
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
<red>TODO</red>
**Problem:** what happens when two programs need a different version of a library?<br/>
**Problem:** what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?<br/>
**Problem:** what happens when you want to provide a **very** long term support for your users (see companies running decade-old databases)?
**Problem:**
- what happens when two programs need a different version of a library?<br/>
The installation of both may no be possible.
See python from version 2 to 3 as an example: developers knew it will break OS systems.
So, they provided by themselves new names for their binaries (`python-2.7`), and libraries are *by default* packaged into a directory specific for a python version, such as `/usr/lib/python3.8/` and this is a form of slotting.
This is mostly done for languages, by what about other packaged applications and libraries?
- what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?<br/>
- what happens when you want to provide a **very** long term support for your users?
*see companies running decade-old OSs and databases*
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
Here, the slot is `baguette`.
Any package outside the official ones are in another named slot.
<red>TODO</red>
Wanna support Python 2.7 **for life**? Just maintain a `python-2.7` slot.
It will be available for everyone.
**This is nothing new, however not often used, and still maybe the best way to handle the problem.**
**This in nothing new, however not often used, and still maybe the best way to handle the problem.**
Others are doing it: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br/>
They all use slotting... *but*.
Since they are is *yet another package manager for your system*, you need to install all your snap software dependencies from their own dependency tree.
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
**Having an alternate package manager for your system is a fancy way of saying:**<br/>
*"I trust all the developers more than my OS maintainers. And fuck testing, who needs that anyway."*
- **snap**<br/>
Snap maintains a `/snap` directory on your system, everything is in there.
They even go to a slotting **per recipe version**, which is *kinda* per application version.
See the readme in `/snap/README`, does it ring a bell?
```
/snap/bin - Symlinks to snap applications.
/snap/<snapname>/<revision> - Mountpoint for snap content.
/snap/<snapname>/current - Symlink to current revision, if enabled.
```
- **flatpak**: same as `snap`
- cpanm, pip, stack, cabal, go... and other **developer tools**<br/>
These are tools for developers to overcome the relatively slow process of providing a library in an OS.
But this process is **slow for a reason**, taking shortcuts is not safe.
<imgsrc="/can-i-haz-my-firefox.png"alt="wanna the updates at all cost"class="meanie-img"/>
<u>*But, BaguetteOS... I still need my last version of Blah!*</u> We gotcha buddy.<br/>
You wanna go fast? Try sonic the good slot: `/usr/sonic`.
With this slot, the BaguetteOS maintainers provide the last versions of a variety of applications and libraries.
### How slotting works in BaguetteOS
<red>TODO: this documentation is not available right now. Sure the project is still awesome anyway!</red>
### How use and change slots used
<red>TODO: this documentation is not available right now. Sure the project is still awesome anyway!</red>
### BaguetteOS file system hierarchy
@ -910,30 +1031,19 @@ Any package outside the official ones are in another named slot.
- `/bad`: things that cannot be properly installed or slotted somewhere else
<aname="roadmap"></a>
# Roadmap
<red>TODO</red>
# 5. Roadmap
We currently aim at providing a rootfs with our tools, when we will have enough spare time to contribute.
**Web interface** is for later: we need more time to design its graphical components.
**Web interface is for later**: we need more time to design its graphical components.
On the other hand, back-end should be straightforward.
<aname="contact"></a>
# Contact
# 6. Contact
Do not hesitate to come on our [Mattermost][mattermost].
We will soon have bridges on XMPP and IRC, stay tuned!
# Just dropped things
80211d
networkctl
firewalld
ajouter des outils de gestion (suppression, modification) de préfixes dans l’environnement (PATH, LD_LIBRARY_PATH, PKGCONFIG_PATH, etc.). À faire a minima dans /etc/profile.
partitionnement et formatage des partitions
Avoir un programme qui permet de retrouver de quel paquet provient un fichier (un reverse apk manifest).
<red>This page isn't ready for public review.</red>
# dnsmanager (and netlib.re)
dnsmanager is a web interface to enable users to register DNS names and manage their zone. It is the software powering [netlib.re](https://netlib.re), a service to provide names for everyone on the Internet.
[netlib.re](https://netlib.re) is kindly operated by [Alsace Réseau Neutre](https://arn-fai.net), a neutral and non-profit Internet Service Provider based in Alsace, France. Don't be shy, come and ask questions!
Philippe PITTOLI
4 years ago