BaguetteOS: new pages (not yet referenced).

master
Karchnu 3 years ago
parent e3a80653b1
commit 080a2ee5bb
  1. 73
      content/baguette/index.md
  2. 884
      content/baguette/our-tooling/index.md
  3. 249
      content/baguette/overview/index.md
  4. 228
      content/baguette/slotting/index.md
  5. 237
      content/baguette/technical-choices/index.md
  6. 8
      content/projects/index.md
  7. 211
      content/projects/specification-file-format/index.md
  8. 4
      sass/custom.scss

@ -34,7 +34,7 @@ If the OS is simple, there is no need to hack it.
**Unified interface is better than features.**<br />
Our web interface will directly provide the basic features such as mail, instant messaging, etc.
The list of features covered by our interface will grow-up continuously.
The list of features covered by our interface will grow continuously.
We do not want a patchwork of very different software, each of them having its own particularities.
**Online services.** *day-to-day use*<br />
@ -44,9 +44,10 @@ The web interface should cover online services, providing an unified experience
The web interface should handle basic system and network configurations, such as adding users, dhcp, DNS, backups, etc.
**Well-known, reliable software.** *for real*<br />
BaguetteOS relies on robust and independent software.
BaguetteOS relies on software known for robustness (opensmtpd, ssh, etc.).
Final users may not be familiar to these software, and <u>they won't have to</u> since BaguetteOS will provide a user interface.
At no point the user should be forced to reinstall, a misconfiguration has to be easily fixable.
We use static compilation for *(at least)* system tools: there is almost no chance an update break the system (yes, *almost*, people are creative these days).
We use static compilation for *(at least)* system tools: there is almost no chance for an update to break the system (yes, *almost*, people are creative these days).
**Hardware support.** *new or old, fast or slow, it doesn't matter*<br />
We provide support for RPi and other small cards: if our kernel runs on it, it has to work.
@ -119,7 +120,7 @@ Switching OS version is just a few environment variables away.
BaguetteOS does not suffer from cumbersome historical decisions: no overly engineered package format, no stupidly complex patchwork of mismatch tools.
**Easy-to-write documentation.** *and hopefully less procrastination*<br />
Online documentation and man pages are written in Markdown (thanks to `zola` and `scdoc`).
Online documentation and man pages are written in Markdown (thanks to `zola` and [scdoc][scdoc]).
Every tool has a man page: no man page = no integration in base.
## Inspiration
@ -127,19 +128,20 @@ Every tool has a man page: no man page = no integration in base.
- [CRUX][crux], [alpine][alpine]: simple-to-understand Linux systems
- [OpenBSD][openbsd]: security-oriented system, therefore simple
- [PFsense][pfsense]: advanced networking administration through a simple website
- [Plan9][plan9] and [Inferno][inferno]: OS with an *everything is a file* philosophy *no seriously guys*
- [Plan9][plan9] and [Inferno][inferno]: OS with an *everything is a file* philosophy *no, seriously guys*
- [suckless][suckless] and [cat-v][cat-v]: documentation and tools, all about simplicity, code readability and re-usability
- [morpheus][morpheus]: OS based on statically compiled tools
## Why not use another system?
This section could be expanded.
An operating system is mostly a kernel handling the hardware and a bunch of small programs (named `coreutils`) doing basic tasks (copy a file, list the content of a directory, print the date, etc.).
Additionaly, an OS also has a package manager (such as `apt` on Debian-like systems).
**A few reasons why none of the candidates cover it all.**<br />
**A few reasons why none of the candidates cover it all.** *some are mentionned bellow*<br />
1. **we want slotting**<br />
So we could change the way we install and maintain packages.
2. **we want fast install and startup on dumb devices**<br />
2. **we want fast install and startup on really simple devices** <side-note>*see Raspberry Pi and such*</side-note><br />
Coreutils shrank to bare minimum, thanks to `toybox`.
We won't require a full-feature system with several hundred megabytes of disk-space used.
3. **documentation is important, but not for all installations**<br />
@ -153,6 +155,8 @@ So we need to run on any available kernel.
We don't accept to follow updates from an upstream OS that could break our system at any time.
*(Note: this section could be expanded in the future.)*
---
**Now, let's take a look at each candidate.**
@ -286,7 +290,48 @@ This is the one reaching the sweet spot between these parameters:
We are also looking at [Zig][zig] for low-level stuff. Wait & see.
## naming convention
## BaguetteOS conventions
### Documentation
Manual pages have to be provided for each software.
`BaguetteOS` tools have man-pages generated with [scdoc][scdoc], letting developers use the markdown format to create man-pages.
Scdoc isn't perfect, it ignores meta-data of man-pages (dates, authors name, etc.) and only provides a way to create man-pages with titles and style.
Scdoc should be seen as a way to encourage developers writing documentation, without the complexity of the `roff` format.
Meta-data are ignored in most usage anyway, <u>we should not let them be an obstacle to documentation</u>.
Additional documentation will be provided on this website, for a more educational approach and informal details.
### Service name, user and group names
Default name for a service instance is the name of the project: `nginx`, `wordpress`, etc.
Arbitrary names may be given to service instances.
Example:
```sh
# service add blog.example.com type=wordpress domain=blog.example.com
```
Default environment is `root`.
When a service is configured, we create a user (and a group) under which the service will run, providing a clean separation between all running services.
User and group names for these services are:
```sh
user = $environment.$service
group = $environment.$service
```
Example:
```sh
service = nginx
environment = root (the default environment)
user = root.nginx
group = root.nginx
```
### Package names
```sh
name = application name
@ -299,6 +344,8 @@ Example:
firefox-79.0-r8.pkg
```
###
<a name="custom-tools"></a>
<a name="top"></a>
# 3. BaguetteOS: custom tools
@ -309,7 +356,7 @@ Also, they can easily run on other systems: nothing here is OS specific.
[Feel free to provide a feedback.](#contact)
Here is a few pointers:
A few pointers:
- [spec format](#spec-format)
- [package](#package)
- [packaging](#packaging)
@ -391,13 +438,13 @@ Next, the usage in practice: [packaging](#packaging), [service](#service).
Package provides slotting by default: no need for custom environments for each software.
Package format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database format contains:
The database of the package manager contains:
- `world`, the file containing the list of available packages
- `installed`, the list of installed packages
- `[package-name]/[slot]/manifest`, the manifest of the installed package
- `[package-name]/[slot]/meta.spec`, the meta-data of the installed package
`Package` configuration consists of:
The configuration of `Package` is made of:
- a list of repositories
- authorized package signing keys
- packaging variables (cflags, makeflags, and so on)
@ -1323,6 +1370,8 @@ On the other hand, back-end should be straightforward.
[working-service-asciinema]: https://asciinema.org/a/0p2vGNA1TUmvq0s61Lu0r4TN6
[ruby-memory-bp]: https://web.archive.org/web/20160329122617/http://blog.rubybestpractices.com/posts/ewong/005-Avoiding-system-calls.html
[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
[pledge]: https://man.openbsd.org/pledge.2
[unveil]: https://man.openbsd.org/unveil.2

@ -0,0 +1,884 @@
+++
title = "BaguetteOS: custom tools"
paginate_by = 5
+++
<a name="top"></a>
**Simple ideas, simple implementations.**<br />
Keep reminding to yourself while reading this section that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br />
Also, they can easily run on other systems: nothing here is OS specific.
[Feel free to provide a feedback.](/contact/)
A few pointers:
- [BaguetteOS configuration file format](#spec-format)
- [Package: our package manager](#package)
- [Packaging: the way to create packages for BaguetteOS](#packaging)
- [Service: configuration should be simple too](#service)
- [service use](#service-use)
- [service example](#service-example)
- [service current implementation](#service-current-implementation)
- [service future](#service-future)
- [build.zsh](#build.zsh)
- [… and a few other tools](#other-tools)
- [Things still in discussion](#still-in-discussion)
<a name="spec-format"></a>
# Spec files: our declarative format
Before presenting our tools, here is a file format named `spec` that we use when relevant.
It is *declarative*: <u>we do not write instructions on *how* to do things</u> (copy this file here, download this, etc.).
Instead, <u>we describe what something is</u>: the URL of the project is `https://example.com/xxx`, for example.
The `spec format` is only composed of variables, lists, code blocks and sections.
Here is a quick example.
```yaml
# This is a comment
# This is a simple variable instanciation
my-variable: value
# This is an inlined list
my-list: a, b, c
# This is a multiline list
my-list:
- a
- b
- c
# Interpolation is performed with %{variable}
url: https://example.com/
software-url: %{url}/my-software-v0.2.tar.gz
# "@an-example-of-code-block" is a Code block.
@an-example-of-code-block
# Code blocks allow inserting simple shell scripting in the configuration file.
curl %{url} -o- | \
sed "s/_my-software.tar.gz//" | \
tail -1
# The scope is the indentation.
# Not the same indentation: not in the block anymore.
# "%an-example-of-section" is a Section, allowing to add metadata to a "target".
# "the-target" is the target, receiving metadata.
%an-example-of-section the-target
# Within a section, variables are declared in the same way as outside the block.
name: The name of the target.
list-of-random-info-about-the-target:
- a
- b
- c
```
Variables and lists look like YAML.
`code blocks` allow to insert `shell scripting`, because sometimes we <u>do</u> want to tell instructions.
Finally, `sections` allow to add metadata to a target (a file or directory).
Note for developers: this file format can be read line-by-line, which is very simple to implement.
Next, the usage in practice: [packaging](#packaging), [service](#service).
[See the dedicated page][specfileformat] for a more in-depth explanation of the specification file format.
[Back to top](#top)
---
<a name="package"></a>
# [Package][package]: our package manager
`Package` covers the basics: install, remove, search and provide information about a package.
`Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example.
Package provides slotting by default: no need for custom environments for each software.
Package format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database of the package manager contains:
- `world`, the file containing the list of available packages
- `installed`, the list of installed packages
- `[package-name]/[slot]/manifest`, the manifest of the installed package
- `[package-name]/[slot]/meta.spec`, the meta-data of the installed package
The configuration of `Package` is made of:
- a list of repositories
- authorized package signing keys
- packaging variables (cflags, makeflags, and so on)
Finally, `Package` can easily be expanded, as it is only a few hundred lines of Crystal code.
[Back to top](#top)
---
<a name="packaging"></a>
# [Packaging][packaging]: the way to create packages for BaguetteOS
Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries.
As BaguetteOS is designed to provide quickly usable systems, we choose to provide binaries.
`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
`Packaging` has a few advantages compared to most used packaging tools:
- **declarative recipes abstract OS specifics**<br />
The same recipe may work for many **native packaging systems** (on many OSs), as long as packagers provide the right target running dependencies.
This only requires to provide a back-end for the package format of the target package manager (a back-end to write `.deb` files, for instance).
- **packages are split automatically**<br />
We need to separate binaries, libraries and documentation in different packages, so we can only install what's needed.
Slow and testing systems only require strict minimum.
- **binaries and libraries are stripped by default**<br />
By default, a running system does not require debug symbols in its binaries.
- **recipe readability is great**<br />
A few variable declarations are better than a dozen lines of code.
- **trivial shell script patterns become automated**<br />
`Autotools` and `cmake` build systems are auto-detected; packagers should only provide specific parameters for each project.
- **tooling may change, recipes won't**<br />
Everybody wants to change its build system?
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just for `packaging`.
- **packages are hashed and signed by default, no extra-tooling**<br />
You need your own set of cryptographic keys, which is created at first use.
- **repositories are created automatically at first compilation, helping people maintaining their own set of tools**<br />
Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
It's <u>that</u> simple.
<img src="/shell-scripting-is-lava.png" alt="shell script is lava" />
<a name="packaging-build-env"></a>
## `Packaging`'s build environments
`Packaging` uses `package` to create build environments and to test packages before validation.
It works as follow:
1. a `/tmp/packaging/build-UUID/` directory is created
2. sources are downloaded, extracted then compiled<br />
Recipes and [`packaging` host configuration](#packaging-host-config) may change parameters to the build: adding steps before compilation, changing `configure` arguments, changing the default slotting (`/usr/baguette`), etc.
3. compiled applications and libraries are put in `/tmp/packaging/build-UUID/root` which is used to create the final package
Build environments are low-cost since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
`Packaging` only installs the minimal set of binaries required by the package to build.
Besides the target application, a building environment size is <u>only a few kilobytes</u>.
<a name="packaging-host-config"></a>
## `Packaging` configuration <side-note>*common configuration for your packages*</side-note>
`Packaging` may be configured globally for your system with the file `/etc/packaging.conf` which contains the following:
```YAML
# Configuration file for `packaging`
# where to send built packages
packages-directory: /usr/local/pkg/
# where to download sources
sources-directory: /usr/local/src/
# the slot we want for our packages
slotting: /usr/baguette
# prefixes for `packaging` running environment and child processes
# = where to search for binaries and libraries for the build
prefixes:
- /usr/baguette/
- /
# list of environment variables we want to have when building
environment:
# we may choose another compiler, provide some CFLAGS, etc.
- CC: clang
- CFLAGS: -Os -Wall
# next three parameters have special meaning
# to provide parameters to the `./configure` script when building
- configure: --disable-nls --without-gettext
# to provide parameters to the `make` command
- make:
# to provide parameters to the final `make install` command
- make install:
# wanna build for another system? not a problem, just add the back-end (we currently have `apk` and `package`)
package-manager: package
```
That's it. You know all about `packaging` configuration.
These parameters may be overridden by recipes.
<a name="recipes"></a>
## `Packaging` recipes <side-note>*we need to create packages*</side-note>
A recipe is the way to reproduce something; here, we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
This means at least having a name for the software and a version (they appear in the package name) and sources (software code).
Let's take an example.
```yaml
# GNU Hello program
name: hello # software name
version: 2.10 # software version
release: 2 # recipe release: incremented when the recipe changes for the current version of the software
# the description will appear in the package information we can retrieve with `package`
description: "This is the GNU Hello program."
# sources may be multiple: you may want to add arbitrary files along the tarball (patches for example)
sources: https://ftp.gnu.org/gnu/hello/hello-%{version}.tar.gz
# we provide running dependencies: the program needs these to `run`
dependencies:
- gettext
# we provide build dependencies: these programs are needed to compile our recipe, not to run the application
build-dependencies:
- make
# if we want to add or override compilation options
options:
- configure: --disable-nls
# feels like déjà vu, right?
# "watch" code block helps to check if the recipe covers the last software version
@watch
curl 'https://ftp.gnu.org/gnu/hello/' -o- 2>/dev/null |
sed -n "/hello-.*\.tar\.gz/{s/\.tar\.gz.*//;s/.*hello-//;p}" |
tail -1
```
This was [a real example][recipe-hello], and not the simplest one.
Most of our recipes currently look like this:
```YAML
name: dhcpcd
version: 8.0.3
sources: https://roy.marples.name/downloads/dhcpcd/dhcpcd-%{version}.tar.xz
description: "dhcp server"
```
That's it.
Yes, we can add a few meta-data, but this is a working recipe.
Configuration, compilation and packaging are done without needing anything else.
The only required parameters are `name`, `version` and `sources`.
<a name="packaging-manual-operations"></a>
Sometimes, developers are assholes and force you to fix their build system.
When manual operations <u>really are required</u>, you can use `@configure`, `@build` and `@install` code blocks.
Let's see an example with a special snowflake… like `perl` which has a non-standard build system.
```YAML
# in our example, we want to configure the build
@configure
# we set some script execution parameters: exit at any error, print everything you do
set -e -x
# we currently are in `/tmp/packaging/build-UUID/`
# now we enter the directory created by the tarball extraction, where the build occurs
cd perl-%{version} # we can interpolate variables with this: %{}
# Perl needs a few environment variables for configuration
BUILD_ZLIB=0
BUILD_BZIP2=0
BZIP2_LIB=/usr/lib
BZIP2_INCLUDE=/usr/include
export BUILD_ZLIB BUILD_BZIP2 BZIP2_LIB BZIP2_INCLUDE
# now we can launch the `configure` script…
# … but wait kids, that's not `configure`, that is `Configure` (see the capitalized C?)
# this is a totally different and very special script, but you should love it anyway
./Configure -des -Dcccdlflags='-fPIC' \
-Dcccdlflags='-fPIC' \
-Dccdlflags='-rdynamic' \
# ... some long and uninteresting list of very specific parameters because we are Perl, we are historic and stuff
```
Now you know how to deal with `@configure`, `@build` and `@install`: these are code blocks allowing you to fix this kind of problems.
Non standard build operations happen from time to time, and code blocks help you overcome this.
If a lot of packages have the same workarounds, we might add detection and integration into `packaging`, so that only specifics are kept into recipes.
If you want to investigate a bit more, you can check [our recipe repository][recipes].
Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
<a name="packaging-future"></a>
## Future of `packaging` <side-note>*let's be even more declarative*</side-note>
As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
Sometimes we have to confront ourselves to poorly designed build systems, but we can hack a bit.
In the future, manual operations should be reduced even more by adding:
- a few other parameters to the environment<br />
This implies to check for patterns in the recipes and to provide workarounds.
- new code blocks *before* the `configuration`, `build` and `install` steps<br />
This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) while still keeping automatic operations.
Another possible addition to `packaging` could be to take cross-OS recipe as first-class citizen.
It was part of the design process of spec files: `sections` can be used to discriminate information based on the target OS.
An example:
```YAML
%OS Ubuntu-20.04
dependencies: libxxx
```
```YAML
%OS Debian-12
dependencies: lib-with-specific-naming-standard-for-reasons
```
This way, bootstrapping an application and providing it to any system <u>with its own tools</u> could be easy.
It's actually what we did during the Baguette's bootstrap.
Providing universal recipes could even become a game for patient system administrators.
[Back to top](#top)
---
<a name="service"></a>
# [Service][service]: service management. <side-note>*not just `kill` or start/stop/status wrapper*</side-note>
Service management often comes with:
- default configuration files, users should learn how to configure them and do it manually
- default user and group, so two instances may have security-involved issues
- a single possible instance, otherwise the configuration has to be heavily changed
- root-only management, simple users rarely run their own services (except on systemd, kudos for once)
- no domain management
These shortcomings imply manual configuration, scripting to manage databases and users, specific tooling for each database and service: this is heavy machinery.
To overcome drawbacks of having simplistic tools, sys-admins developed all kind of monstrous architectures.
- **LXC** *it's basically a chroot with network and software limits*<br />
LXC is *kinda* reasonable, and may be useful in some cases, but it provides no simple way of configuring our services.
- **Qemu + KVM, Xen** *let's add software mimicking hardware's complexity to the mix, telling everyone it's for security and simplicity*<br />
These programs make the administration simple for sys-admins: no need to configure thoroughly users, groups, etc.
Everyone is root and handles its administration as (s)he wants.
Qemu and alike also help big companies to have a large computing capacity which they can rent when they don't need it.
At no point Qemu or Xen help getting your services up and running, and [they are not made for security][cveqemu].
Yes, running broken programs may be better within Qemu than on plain non security-oriented OS.
But.
This is still <u>way less efficient than fixing the application</u>.
Running applications as simple users, compiling them with sane default options ([RETGUARD][retguard] for example) and providing a few syscalls (like [pledge][pledge] and [unveil][unveil]) to catch errors and most security holes is simple nowadays, let's use that.<br />
*But, I can't do that in my company 'cause reasons.*<br />
That's okay if you don't want to do that in your company, we are here for <u>**final users**</u> and advanced users, mostly.
We don't aim at fixing everybody's problems.
- **docker** *I don't know how to do simple applications nor packages, so I give to you my whole dev environment*<br />
Note: we have to admit, packaging on most OS is painful for absolutely no good reason.
- **Chef and Puppet** *the 500 MB running Ruby code on our virtual machines just to check for configuration updates is okay 'cause memory is cheap, right?*<br />
We talk about the importance of security from time to time, but running a software designed by people telling [it's okay not to free memory][ruby-memory-bp] is far from being wise.
- **Ansible** *templating your applications… from another machine*<br />
As Chef and Puppet, ansible provides templating for applications, this time configuration propagation is **way** simpler since it uses well-known, trusted and loved ssh.
Still, templating is done for remote machines, as it is intended for server deployments: this is **a sys-admin tool**.
The introduction page already talks about `cloud provisionning` and `intra-service orchestration` on the first line, telling that you really need to setup SSH keys, etc.
As for many tools in IT: this is not for simple users.
Simple users:
1. should only have to provide absolutely necessary information for their services
2. should be able to run as many services as they want
3. shouldn't have to learn configuration syntax for their services
4. shouldn't be afraid of updates
5. ... and should have BACKUPS! Where are they? We should have that **by default** on our systems over 20 years ago.
And advanced users should have an uncomplicated CLI tool to do that.
<a name="service-use"></a>
## Let's take an example with `service` <side-note>*better than a thousand words*</side-note>
```sh
# We want a wordpress service, proxied by an nginx and using postgresql as DBMS
# THIS IS THE "VERBOSE" VERSION
# 1. we add an nginx
$ service add nginx
```
```sh
# 2. we add the database
$ service add postgresql
```
```sh
# 3. we add the wordpress
# by default, it uses available http proxy and database
$ service add wordpress domain=example.com http=nginx database=postgresql
```
```sh
# 4. we start the wordpress
$ service start wordpress
```
A bit of explanation:
1. first, we add `nginx` to the list of services we want on our system
2. same thing with `postgresql`
3. then we add `wordpress` and we pass parameters to the service configuration: `domain`, `http` and `database`<br />
`domain` for the domain name, `http` for the HTTP proxy and then the `database` back-end.
Up to this point, nothing even started.
4. Finally, we start the service.<br />
Since `service` knows the dependence graph, it starts other services before wordpress (which actually doesn't have any binary to run *per se*).
The configuration of a service is made when `service` starts it.
In this case:
1. `nginx` is started as a proxy for `example.com`
2. `postgresql` is started, its internal directories and files are created, then a user, a database and a password for `wordpress` are created
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
Stopping a service also stops its dependencies, unless specified otherwise.
Of course, a service is not stopped if it is required elsewhere.
Wanna see the less verbose version?
```zsh
$ service add wordpress domain=example.com
```
```zsh
$ service start wordpress
```
And <u>that's it</u>.
1. Services have tokens.
2. Tokens are used by default.
3. BaguetteOS provides **default** services for each token.
4. If a service is added and its dependencies aren't satisfied, we add other services.
5. (Bonus) If a service isn't installed, we ask nicely if the user wants to install it.<br />
*(This is in discussion.)*
Here are a few functionalities `service` brings.
1. **uncomplicated service configuration with shared information**<br />
Services can share:
- passwords (which are auto-generated)
- user names (which follow a naming convention)
- port numbers (auto-attributed unless specified)
- ... etc.
2. **templates** *configuration files are generated by templates and a few parameters*<br />
When we want a software, for instance a blog, we want to provide the minimum informations it requires to work and that's it.
When `service` starts a service, it verifies if its configuration file is installed and up-to-date, and creates it otherwise.
Users shouldn't need to manually change the configuration.
<u>Syntax may change at any time without breaking a single service</u>, since the configuration will smoothly be regenerated with useful information at start-up.
3. **environments**<br />
Each service can be installed in a specific environment, such as a custom rootfs, a virtual machine, etc.
4. **automatic user and group creation**<br />
Each service needs to run with a unique user, dedicated to it, different from other users on the system for security reasons.
Let's make it the default behavior.
5. **tokens** *service a needs b, b needs c, when a starts, b and c are configured then started*<br />
`service` knows the relations between services, and uses them to configure services through the templates.
*everything is smoother now*
6. **automatic backup solution**<br />
Since we know each running database, service configuration and data directories, we can backup everything.
We just need a backup server to be configured.
Of course, we can add a lot of parameters to have a fine-grained backup solution.
```sh
# not currently done, but will look like this
$ backup add ssh:user@example.com:/srv/backup
```
7. **unified way to configure the system** *best user experience possible*<br />
It alleviates the need for manual configuration.
The CLI tool is the same for every provided service.<br />
Behind the scene, it's a simple token system with configuration templating!<br />
<u>No heavy machinery here, and we'll keep it that way.</u>
8. **... and understandable tooling output** *for god's sake!*<br />
```sh
$ sudo service show my-awesome-gitea-instance
Name: my-awesome-gitea-instance
Type: gitea
Environment: root (prefix)
Consumes:
- database root/postgresql
- http root/nginx
Ports:
- http: 49155
```
<a name="service-example"></a>
## `Service`: a service example
A service is defined by its description in a `.spec` file and its template(s).
Let's use `gitea` as an example, the `git` web server handling [our repositories][baguette-gitea].
```YAML
# `gitea.spec`
# The command which is run to start the service.
command: gitea -C . -w . -c gitea.cfg
# The tokens needed by the service. Here http is optional.
consumes: database, http?
# Does the service require a domain name to be configured?
# If so, service will require a "domain" token, such as `service add $service domain=example.com`
# This name may be used in the service configuration, or its (forward or backward) dependencies.
requires-domain: true
# The port(s) the service requires.
# By default, if a service requires a port named "http", its value will be 80.
ports: http
# `%configuration` indicates the name of the configuration template.
# For example, if the template file is `/usr/baguette/service/templates/gitea.cfg.j2`:
%configuration gitea.cfg
```
*We **currently** use a bit more configuration for the database, but we will get there at some point.*
Now, a quick look at the `gitea` configuration template. <side-note>*just a sample, we'll skip most, don't worry*</side-note><br />
Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
First, let's say we created a `gitea` service this way:
```zsh
service add gitea domain=example.com database=postgresql
```
Now, we want to see the template used to configure `gitea`.
```YAML
[database]
# `service` passes data to the templates.
# `providers` is the list of services behind the tokens used.
# For example, behind "providers.database" there is a `postgresql` service.
# Gitea needs to establish a connection to its database.
# The database ports can be accessed with "database.ports".
# The port used to connect to the database is named "main" by convention.
# See the definition of the postgresql service for details.
HOST = 127.0.0.1:{{ providers.database.ports.main }}
# Here, the template requires the database name.
# By convention, databases are named: $environment_$service_db.
# In our example, `service` creates the database "root_gitea_db", as the default environment is "root".
# `service.id` is '$environment/$service', which uniquely identifies the service.
# In our example, `service.id` is "root/gitea".
# The database name will be obtained by replacing "/" by "_" and adding "_db" to the service.id variable.
# This is performed this way:
NAME = {{ service.id | replace("/", "_") }}_db
# By convention the user name is identical to the database name without the "_db" suffix.
USER = {{ service.id | replace("/", "_") }}
# The `random_password` function creates a unique password the first time it is called.
# This function takes a service id in parameter.
# After that, it provides the same password each time it is called with the same service id.
# `random_password` enables sharing a password between two service configurations.
PASSWD = {{ random_password( service.id ) }}
[repository]
# service.root indicates the service working directory: /srv/<environment>/<service>/
ROOT = {{ service.root }}/repositories
[server]
# service.domain = provided domain token value
# service.ports.http = provided ports.http token value
ROOT_URL = http://{{ service.domain }}:{{ service.ports.http }}/
```
<a name="service-current-implementation"></a>
## Current implementation of `service`
It [currently works][working-service-asciinema], and we just need to add more services!
Every new corner case will be thoroughly investigated to provide the smoothest experience possible with BaguetteOS.
We currently use it for the nginx providing this very website.
Currently, there are no fancy environments, just plain directories.
<a name="service-future"></a>
## Future of `service`
First, we want to work on databases, to export data and get a simple backup service.
Then, we could let users manage their own services, but this is of little interest in practice since services are running under different users already.
The most useful contributions right now would be to provide new service configurations.
[Back to top](#top)
---
<a name="build.zsh"></a>
# [Build.zsh][build.zsh]: makefile creation. <side-note>*for mere mortals*</side-note>
A `Makefile` is very useful to share your project, <u>whatever your project is</u>.
No matter if it is an application or a library, whatever the language used, etc.
Build.zsh creates a makefile from a simple declarative configuration file.
A `Makefile` should:
- **compile and link the application**
- **handle dependencies** and **rebuild only the updated parts of the application** *for incremental builds*
- **allow users to rebuild any part of the project independently**
- **install the application**<br />
The default installation root directory is `/usr/local` but it can be changed easily (with an environment variable or through configuration).
- **create a tarball of the project**<br />
This tarball includes the application, its man-pages, the Makefile to build it, etc.
- **have a `make help`** *everybody needs help*
- *(BONUS)* have colors. I like colors.
<a name="buildzsh-how-to-makefile"></a>
## How-to generate a Makefile using `Build.zsh`
```zsh
$ ls
project.zsh # we need to create this file, we'll cover that in a moment
```
```zsh
$ build.zsh -c # -c is for colors. I like colors. Really.
```
```zsh
$ ls
project.zsh
Makefile # tadaaaaa you made a Makefile
```
You can invoke your newly created Makefile with `make help` to know what it can do for you.
For example, it can create a tarball with `make dist`.
<a name="buildzsh-projectzsh"></a>
## How-to make a `project.zsh` for `build.zsh`
```zsh
# File `project.zsh`
package=my-application # Name of your package.
version=2.18.3 # Version of the package (will follow the application version, probably).
# targets = all the applications to compile in the project.
targets=(my-application)
# The language of these applications is then specified.
# Here, my-application is coded in Crystal.
# `build.zsh` comes with a number of back-ends: https://git.baguette.netlib.re/Baguette/build.zsh/src/branch/master/build
type[my-application]=crystal
# The sources of the application.
sources[my-application]=src/main.cr
# The application depends on a certain number of files.
# `make` will re-compile "my-application" if one of these files is modified.
depends[my-application]="$(find src -type f | grep -v main.cr)"
# Finally, the list of the files to be included in the tarball (made through `make dist`).
# Here, the tarball will include:
# the application binary,
# man-pages in man/ directory,
# this project.zsh and the generated Makefile.
dist=(my-application man/*[1-9] project.zsh Makefile)
```
<a name="buildzsh-real-life-example"></a>
## `Build.zsh`: a real-life example
```zsh
package=build_zsh # Name of the package.
version=0.2.1 # Version of the package.
targets=(build.zsh) # The things to build or install.
type[build.zsh]=script # How they’re built.
# Using a for loop to add more targets.
# In this example, we’re registering scripts for installation.
for i in build/*.zsh; do
targets+=($i)
type[$i]=script
# Installation in a non-default directory.
install[$i]='$(SHAREDIR)/build.zsh'
# Targets marked as “auto” won’t appear in `make help`.
auto[$i]=true
done
# Files to add to tarballs through `make dist`.
dist=(build.zsh.in build/*.zsh project.zsh Makefile)
```
<a name="buildzsh-another-example"></a>
## Another real-life example <side-note>*I like examples*</side-note>
This time, we want to build a Makefile for a C library.
```zsh
package=libipc # Name of the package.
version=0.5.1 # Version of the package.
# Our targets are the library and its documentation.
targets=(libipc man/libipc.7)
# The libipc target is a library ("for the C language" is implied).
# The `library` type automatically adds two targets:
# `target`.so of type `sharedlib`
# `target`.a of type `staticlib`
type[libipc]=library
# Sources are added by default to the tarball.
sources[libipc]=$(ls src/*.c)
depends[libipc]=$(ls src/*.h)
# Add extra compilation flags.
variables+=(CFLAGS "-Wall -Wextra -g")
# Let's add some CFLAGS, with another syntax.
cflags[libipc]="-std=c11"
# The man/libipc.7 target is a man-page generated with `scdoc`.
# `scdocman` is one of the many back-ends of build.zsh: https://git.baguette.netlib.re/Baguette/build.zsh
type[man/libipc.7]=scdocman
# Files to add in the tarball, generated with `make dist`.
dist=(libipc.so libipc.a) # The library in both shared and static versions.
dist+=(man/*.scd man/*.[1-9]) # Manual pages (and sources).
dist+=(Makefile project.zsh) # The generated Makefile and this project.zsh.
```
This example shows how to work with C libraries, but it wouldn't have changed much for an application: only the `type` of the target would change.
This `project.zsh` comes from the [LibIPC repository][libipc] (a library we'll see soon).
<a name="buildzsh-future"></a>
## What about now?
Now you do not have any good reason avoiding Makefiles for your projects!
***But, I don't like Makefiles!***<br />
You like the functionalities we presented here, but you want an alternative to Makefiles?
<red>We may add some other back-ends.</red>
Stay tuned!
[Back to top](#top)
---
<a name="libipc"></a>
# [LibIPC][libipc]: an IPC communication library <side-note>*nothing new, yet it still feels fresh*</side-note>
We use this communication library between our services.
1. Applications should talk to each other
2. We need services, not libraries<br />
Therefore, languages are irrelevant: you can use *any* library in *any* language.
LibIPC is currently used for the administration dashboard, the web interface for the services, [for a kanban][todod] and several other tools we use for collaboration.
It provides a way to communicate between clients and services using simple unix sockets behind the scene.
C library with Crystal bindings (other languages coming soon)
```Crystal
require "ipc.cr"
server = IPC::Service.new "MyService"
server.loop do |message|
# ...
end
```
That's easy to write [even in plain C](https://git.baguette.netlib.re/Baguette/libipc/src/branch/master/examples/pongd.c).
LibIPC explanation goes beyond the scope of this page… <red>and may even deserve a whole website on its own</red> but the tool is awesome and performances are crazy (we have to tell the world!).
[Just go and check!][libipc]
<red>Explain remote communications.</red>
Remote communications are transparent.
- clients and services do not need remote communication
- any client can join remote services via any communication protocol
- any service is implicitly accessible from anywhere, with any protocol
[Back to top](#top)
---
<a name="other-tools"></a>
# Other tools
## [tap-aggregator][tap-aggregator]: quality assurance & test results aggregation
[Back to top](#top)
---
## [Webhooksd][webhooksd]: verify recipes.
Webhooksd provides an automatic verification of the recipes, based on new application or library version.
Paired with a build system, new recipes received in the repository create packages for a couple of architectures (x86\_64, ARM, others will follow).
[Back to top](#top)
---
<a name="still-in-discussion"></a>
# Still in discussion
For the simple users, we want to provide an unified web interface to manage the system and online services.
We currently use `Crystal` to work on service back-ends for a variety of projects, we are satisfied on a technical level and we are productive, it is highly probable we continue using it.
The front-end is still in discussion: we currently use `livescript` and it is way more usable than plain JS, but it is painful to debug.
So, we need a language for both administration dashboard and online services, here are some examples we find interesting for the job:
- Purescript
- haskell-like syntax, with a smaller set of notions
- strongly typed, with type inference
- good documentation
- useful compilation errors
- no runtime error
- Elm
- as Purescript but with way less documentation (but reading the code is sometimes enough here)
- less generic code (functions such as `fold` and `map` have hardcoded type), which feels a bit hacky
- still very young
- WASM
- seems to be a very young tech, with no real good language or documentation
- Zig has wasm as a Tier 1 support, we should investigate
And we should implement a generic framework, QML was the way all along (but without all the historic tooling and **without C++** it would be awesome!).
# Next
Next step is [to use][usebaguetteos] or even [become a contributor][contributor].
[usebaguetteos]: /baguette
[contributor]: /projects
[specfileformat]: /projects/specification-file-format/
[service]: https://git.baguette.netlib.re/Baguette/service
[package]: https://git.baguette.netlib.re/Baguette/package
[packaging]: https://git.baguette.netlib.re/Baguette/packaging
[build.zsh]: https://git.baguette.netlib.re/Baguette/build.zsh
[libipc]: https://git.baguette.netlib.re/Baguette/libipc
[todod]: https://git.baguette.netlib.re/Baguette/todod
[webhooksd]: https://git.baguette.netlib.re/Baguette/
[tap-aggregator]: https://git.baguette.netlib.re/Baguette/tap-aggregator
[baguette-gitea]: https://git.baguette.netlib.re/
[baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc
[baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile
[recipes]: https://git.baguette.netlib.re/Baguette/recipes/
[recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec
[recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec
[recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec
[autotools]: https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
[toybox]: http://www.landley.net/toybox/
[cruxinit]: https://crux.nu/gitweb/?p=ports/core.git;a=blob;f=rc/rc;h=26b8ca08d67208ceda4d4004c8333d362bcdc689;hb=HEAD
[ksh]: https://github.com/att/ast
[zsh]: https://www.zsh.org/
[libarchive]: https://libarchive.org/
[bsdcpio]: https://libarchive.org/
[bsdtar]: https://libarchive.org/
[m4]: https://www.gnu.org/software/m4/m4.html
[gmake]: https://www.gnu.org/software/make/
[proot]: https://man.openbsd.org/proot
[crinja]: https://github.com/straight-shoota/crinja
[zig]: https://ziglang.org/
[openbsd]: https://openbsd.org/
[pfsense]: https://www.pfsense.org/
[alpine]: https://alpinelinux.org/
[crux]: https://crux.nu/
[inferno]: http://www.vitanuova.com/inferno/index.html
[plan9]: https://9p.io/plan9/index.html
[morpheus]: https://morpheus.2f30.org/
[suckless]: https://suckless.org/
[cat-v]: http://cat-v.org/
[working-service-asciinema]: https://asciinema.org/a/0p2vGNA1TUmvq0s61Lu0r4TN6
[ruby-memory-bp]: https://web.archive.org/web/20160329122617/http://blog.rubybestpractices.com/posts/ewong/005-Avoiding-system-calls.html
[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
[pledge]: https://man.openbsd.org/pledge.2
[unveil]: https://man.openbsd.org/unveil.2
[retguard]: https://undeadly.org/cgi?action=article&sid=20170819230157
[cveqemu]: https://www.cvedetails.com/vulnerability-list.php?vendor_id=7506&product_id=0&version_id=0&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=276&sha=6055b0330a499f6aed7620adb79dc0cc143e50bc

@ -0,0 +1,249 @@
+++
title = "BaguetteOS: quick overview"
paginate_by = 5
+++
BaguetteOS status: Work In Progress.
A beta will be available circa mid-2020 for advanced users.
End of the year for simple users.
<red>Warning:</red> this documentation is currently mostly oriented towards advanced unix users.
We provide insights on system administration, expose our ideas and point of view.
Documentation for simple users will be split in another page, soon.
- [Objectives for **simple users**](#simple-users)
- [Objectives for **advanced users**](#advanced-users)
- [Objectives for **contributors**](#contributors)
- [Inspiration](#inspiration)
- [Why not use X?](#why-not-x)
<a name="simple-users"></a>
## Objectives, for simple users
BaguetteOS aims at providing a simple unix-like system, with an **unified web interface**.
**No command-line required for simple users.** *let users be just users*<br />
Simplicity should not only come from an interface, but be inherent to the whole system.
If the OS is simple, there is no need to hack it.
**Unified interface is better than features.**<br />
Our web interface will directly provide the basic features such as mail, instant messaging, etc.
The list of features covered by our interface will grow continuously.
We do not want a patchwork of very different software, each of them having its own particularities.
**Online services.** *day-to-day use*<br />
The web interface should cover online services, providing an unified experience for main usages: mails, calendar, instant messaging, personal website, file sharing, etc.
**One-click management.** *service installs, updates, etc.*<br />
The web interface should handle basic system and network configurations, such as adding users, dhcp, DNS, backups, etc.
**Well-known, reliable software.** *for real*<br />
BaguetteOS relies on software known for robustness (opensmtpd, ssh, etc.).
Final users may not be familiar to these software, and <u>they won't have to</u> since BaguetteOS will provide a user interface.
At no point the user should be forced to reinstall, a misconfiguration has to be easily fixable.
We use static compilation for *(at least)* system tools: there is almost no chance for an update to break the system (yes, *almost*, people are creative these days).
**Hardware support.** *new or old, fast or slow, it doesn't matter*<br />
We provide support for RPi and other small cards: if our kernel runs on it, it has to work.
Minimal hardware requirement should be around 100 MB RAM, 50 MB disk.
**Documentation.** *simple, reliable, useful, all-in-one-place*<br />
Similar to the OpenBSD FAQ: updated, complete, concise and well-written.
**Constrained devices use case.** *wanna see what small systems can do?*<br />
By default, we try to provide the smallest memory footprint: we do not provide manuals nor runtime libraries when not required.
Our programs will never require you to download development libraries, nor alternative package managers.
We choose all of our tools with size in mind.
As a result, our system can be installed quickly even on slow machines.
<a name="advanced-users"></a>
## Objectives, for advanced users
**A knowable OS.** *simplicity at (almost) all cost*<br />
Any interested user should be able to understand the role of every part of the base system: no compromise.
This means having a very small and consistent set of tools, easy to learn, easy to remember.
**Basic system and network management.** *with the simplest tools ever*<br />
We provide a web interface that should handle basic system and network configurations, such as adding users, firewall management, dhcp, DNS, backups, etc.
CLI tools are available to manage services, they are designed to be simple, consistent and reliable.
**Officially supported services.** *so you are sure to get them working*<br />
We use some services for our own personal usage, so we will provide support for them.
For instance: gitea, postgresql, etc.
**One need, one tool.** *this time for real*<br />
Installing an application or a library is done by [package][package].
Other methods are not supported **and the base system will never require them**.
We avoid to rely on `pip`, `cpanm`, or other third party package manager and dependency tree.
Starting, stopping, or configuring a service is done by [service][service].
This program alone is used to manage services on the OS.
Users should not be required to manually configure each software; instead, most of the configuration should be done upstream using templates.
Users should be able to change the default configuration through command-line options.
Manual configuration is the last option.
**Slotting.** *any software can be installed on any machine at any time, no extra tooling*<br />
[Slotting](#slotting) by default helps to install many programs, even with peculiar library version requirements.
The same program can be installed in several versions without conflicts.
See more in the [technical section](#technical-choices).
<a name="contributors"></a>
## Objectives, for contributors
**Simple to contribute to.** *you can focus on providing recipes for packages, we handle the tooling*<br />
Packaging applications and libraries should be possible with the fewest and simplest tooling possible.
BaguetteOS provides [a simple tool to package applications](#packaging) (`packaging`) which allows simple recipes for your packages.
`packaging` handles [slotting](#slotting), compiling, stripping binaries and libraries, splitting the result into different packages (`-man`, `-lib`, `-doc`, etc.), authenticating, etc.
All that, just by typing
```sh
$ packaging app
```
nothing more.
<img src="/meanie-makefile.png" alt="meanie meanie dev" class="meanie-img" />
**No Makefile?** *no problem*<br />
Your application or your library lacks a build system? Here is a [tool to create makefiles](#build.zsh).
It works on any language. *yes, even that one*
**Stable and development versions: same thing.** *slotting, again and again*<br />
One of the coolest aspect of slotting is: you don't need to change your system *at all* to be on the development version of the OS.
The newest version is just slotted in `/usr/baguette-dev` and that's it.
Switching OS version is just a few environment variables away.
**New OS, open to explore** *we are open to new ideas*<br />
BaguetteOS does not suffer from cumbersome historical decisions: no overly engineered package format, no stupidly complex patchwork of mismatch tools.
**Easy-to-write documentation.** *and hopefully less procrastination*<br />
Online documentation and man pages are written in Markdown (thanks to `zola` and [scdoc][scdoc]).
Every tool has a man page: no man page = no integration in base.
## Inspiration
- [CRUX][crux], [alpine][alpine]: simple-to-understand Linux systems
- [OpenBSD][openbsd]: security-oriented system, therefore simple
- [PFsense][pfsense]: advanced networking administration through a simple website
- [Plan9][plan9] and [Inferno][inferno]: OS with an *everything is a file* philosophy *no, seriously guys*
- [suckless][suckless] and [cat-v][cat-v]: documentation and tools, all about simplicity, code readability and re-usability
- [morpheus][morpheus]: OS based on statically compiled tools
## Why not use another system?
An operating system is mostly a kernel handling the hardware and a bunch of small programs (named `coreutils`) doing basic tasks (copy a file, list the content of a directory, print the date, etc.).
Additionaly, an OS also has a package manager (such as `apt` on Debian-like systems).
**A few reasons why none of the candidates cover it all.** *some are mentionned bellow*<br />
1. **we want slotting**<br />
So we could change the way we install and maintain packages.
2. **we want fast install and startup on really simple devices** <side-note>*see Raspberry Pi and such*</side-note><br />
Coreutils shrank to bare minimum, thanks to `toybox`.
We won't require a full-feature system with several hundred megabytes of disk-space used.
3. **documentation is important, but not for all installations**<br />
Your 42nd test install on a virtual machine doesn't need a manual for CLI tools you won't use since you do everything through the web interface.
Software and its documentation will be split: manual pages won't be provided by default.
4. **we want automatic tests, and to enable people to test our system**<br />
We want easy chroot installs, on any system.
5. **we want to run on tiny ARM boxes, old laptops, top-notch servers**<br />
So we need to run on any available kernel.
6. **we want to control software distribution releases**<br />
We don't accept to follow updates from an upstream OS that could break our system at any time.
*(Note: this section could be expanded in the future.)*
---
**Now, let's take a look at each candidate.**
**OpenBSD.** *we will get there quick, but will focus on Linux a bit before* <br />
We definitively want to use OpenBSD, currently we just focus on Linux for hardware compatibility reasons (and out of habits) but it's not set in stone.
We love OpenBSD big time, some of us use it daily.
We aim at providing rootfs for running our system under an OpenBSD kernel and environment.<br />
*(also, snapshots could be great, guys)*
**PFsense.** *network focused* <br />
Great system, does the job very well, but won't fit our objectives as it is way too focused on networking.
**CRUX and Alpine.** *great source of inspiration* <br />
We **do** use the CRUX's `rc` script, and as Alpine it is a source of inspiration for package recipes.
However, since we have to change all packages to get slotting, the service manager to have tokenized services, the packaging software to get declarative recipes (...), why even bother use those systems?
**GUIX (and Nix).** *not simple enough*<br />
GUIX approach of package management is interesting, having to learn a language to make recipes isn't.
And that sums up this technology pretty well.
It's great, but not to put everywhere.
Every part of BaguetteOS is simple to understand, GUIX is not simple *enough*.
But keep it up guys, it's still awesome.
**Plan9, Inferno, morpheus, etc.** *kinda abandoned systems* <br />
That's it.
# Wanna see more?
[See the technical choices](/baguette/technical-choices/)
[service]: https://git.baguette.netlib.re/Baguette/service
[package]: https://git.baguette.netlib.re/Baguette/package
[packaging]: https://git.baguette.netlib.re/Baguette/packaging
[build.zsh]: https://git.baguette.netlib.re/Baguette/build.zsh
[libipc]: https://git.baguette.netlib.re/Baguette/libipc
[todod]: https://git.baguette.netlib.re/Baguette/todod
[webhooksd]: https://git.baguette.netlib.re/Baguette/
[tap-aggregator]: https://git.baguette.netlib.re/Baguette/tap-aggregator
[baguette-gitea]: https://git.baguette.netlib.re/
[baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc
[baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile
[recipes]: https://git.baguette.netlib.re/Baguette/recipes/
[recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec
[recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec
[recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec
[autotools]: https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
[toybox]: http://www.landley.net/toybox/
[cruxinit]: https://crux.nu/gitweb/?p=ports/core.git;a=blob;f=rc/rc;h=26b8ca08d67208ceda4d4004c8333d362bcdc689;hb=HEAD
[ksh]: https://github.com/att/ast
[zsh]: https://www.zsh.org/
[libarchive]: https://libarchive.org/
[bsdcpio]: https://libarchive.org/
[bsdtar]: https://libarchive.org/
[m4]: https://www.gnu.org/software/m4/m4.html
[gmake]: https://www.gnu.org/software/make/
[proot]: https://man.openbsd.org/proot
[crinja]: https://github.com/straight-shoota/crinja
[zig]: https://ziglang.org/
[openbsd]: https://openbsd.org/
[pfsense]: https://www.pfsense.org/
[alpine]: https://alpinelinux.org/
[crux]: https://crux.nu/
[inferno]: http://www.vitanuova.com/inferno/index.html
[plan9]: https://9p.io/plan9/index.html
[morpheus]: https://morpheus.2f30.org/
[suckless]: https://suckless.org/
[cat-v]: http://cat-v.org/
[working-service-asciinema]: https://asciinema.org/a/0p2vGNA1TUmvq0s61Lu0r4TN6
[ruby-memory-bp]: https://web.archive.org/web/20160329122617/http://blog.rubybestpractices.com/posts/ewong/005-Avoiding-system-calls.html
[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
[pledge]: https://man.openbsd.org/pledge.2
[unveil]: https://man.openbsd.org/unveil.2
[retguard]: https://undeadly.org/cgi?action=article&sid=20170819230157
[cveqemu]: https://www.cvedetails.com/vulnerability-list.php?vendor_id=7506&product_id=0&version_id=0&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=276&sha=6055b0330a499f6aed7620adb79dc0cc143e50bc

@ -0,0 +1,228 @@
+++
title = "Slotting: what it is, why it matters"
paginate_by = 5
+++
- [Overview](#overview)
- [Technical choices](#technical-choices)
- [Custom tools](#custom-tools)
- [Slotting](#slotting)
- [Roadmap](#roadmap)
- [BaguetteOS File-System Hierarchy](#baguetteos-fsh)
<a name="slotting"></a>
# Slotting: providing software the right way
The usual way to provide software is to package it into a distribution, then provide it as *the* OS version of the software.
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
Maintainers' job is to verify that all programs have the right library versions under their OS.
### Current set of problems
- What happens when two programs need a different version of a library?<br />
The installation of both may not be possible without workarounds.
See python from version 2 to 3 as an example.
To make it work, OSs have given new names for their binaries (`python-2.7` and `python-3.5` for example).
Libraries are *by default* packaged into a directory specific for a python version, such as `/usr/lib/python3.5/`.
This is mostly done for languages, but what about other packaged applications and libraries?
- What happens when two libraries are compatible but you want both on your system (see libressl and openssl)?<br />
One of them could be provided in another path, such as `/usr/lib/libressl`.
- What happens when you want to provide a **very** long-term support for your users?
*see companies running decade-old OSs and databases*
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
### What is slotting?
Slotting is a way to use prefixes (paths, directories) to separate execution environments: a program **A**, requiring libraries **B and C** can be installed this way:
```sh
/usr/awesome-slot/bin/A
/usr/awesome-slot/lib/B
/usr/awesome-slot/lib/C
```
In this example, the `slot` is named **awesome-slot**, providing an execution environment for A <u>no matter</u> the OS version of *B* and *C*.
**Without slotting** <side-note>*basically, your life sucks*</side-note><br />
Let's take an example with software provided by a non-official repository on a Linux distribution.
You add a non-official repository for <blue>my-overly-awesome-game</blue> to your Debian system.
This newly installed program will be in `/usr/bin`, as every other program.
1. What if the game requires libraries?<br />
These libraries are installed in `/usr/lib`.
2. What if the game requires libraries that are not in the official repository?<br />
Either the repository for <blue>my-overly-awesome-game</blue> provides them directly, or you can find another repository providing them.
In <u>both cases</u> these libraries will end-up in `/usr/lib`.
**With slotting** <side-note>*your're awesome*</side-note><br />
With slotting, the program will be in `/usr/`<blue>my-overly-awesome-game</blue>`/bin`.
1. What if the game requires libraries?<br />
If these libraries are available in the `BaguetteOS` repository, they will be installed in your base system.
2. What if the game requires libraries that aren't available in the official `BaguetteOS` repository?<br />
Either the game slot provides them, or they are in another slot.
In <u>both cases</u> the base system won't change.
Besides essential programs such as `coreutils` which are in `/bin` and `/sbin`, all official OS packages are installed in the slot named `baguette` (`/usr/baguette/`).
Any non-official package is in another slot.
Wanna support Python 2.7 **for life**?
Just maintain a `python-2.7` slot and tell the world!
If BaguetteOS does not provide the libraries required for the continuous support of your application, just add them in your slot.
**Slotting is nothing new, however it is usually not used directly in OSs, whereas it may be the best way to handle the problem.**
### Why not use *X*?
Others are doing slotting too: snap, flatpak, cpanm, pip, go, stack, cabal... <side-note>*the list seems endless*</side-note><br />
They all use slotting... *but*.
Since they are *yet another package manager for your system*, you need to install all your software dependencies from their own dependency tree.
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
**Having an alternate package manager for your system is a fancy way of saying:**<br />
*"I trust all the developers more than my OS maintainers. And fuck testing, who needs that anyway."*
- **snap**<br />
Snap maintains a `/snap` directory on your system, everything is in there.
They even go to a slotting **per recipe version**, which is *kinda* per application version.
See the readme in `/snap/README`, does it ring a bell?
```
/snap/bin - Symlinks to snap applications.
/snap/<snapname>/<revision> - Mountpoint for snap content.
/snap/<snapname>/current - Symlink to current revision, if enabled.
```
- **flatpak**: same as `snap`
- cpanm, pip, stack, cabal, go... and other **developer tools**<br />
These are tools for developers to overcome the relatively slow process of providing a library in an OS.
But this process is **slow for a reason**, taking shortcuts is not safe.
<img src="/can-i-haz-my-firefox.png" alt="wanna the updates at all cost" class="meanie-img" />
<u>*But, BaguetteOS... I still need my last version of Blah!*</u> We gotcha buddy.<br />
You wanna go fast? Try sonic the fast slot: `/usr/sonic`.
With this slot, the BaguetteOS maintainers provide the last versions of a variety of applications and libraries.
You will have bleeding-edge technologies and bugs. You're welcome!
### How slotting works in BaguetteOS
**Applications and libraries provided by BaguetteOS.**<br />
For all official OS versions of the applications and libraries, `BaguetteOS` will provide them in `/usr/baguette`, the `baguette` slot.
In case several versions of a library are provided, they will be slotted.
For example, `LLVM` is provided in several versions (8, 9 and 10), only the most recent is in `baguette`.
```zsh
$ ls /usr
/usr/baguette
/usr/llvm-8
/usr/llvm-9
```
**Applications and libraries provided by third parties.**<br />
`BaguetteOS` allows third parties to provide their applications and libraries easily by creating repositories, but they have to be slotted.
For example, to provide a specific `nodejs` version, the following convention must be used:
```zsh
/usr/$application-$version/
```
### How to use slots and install new repositories
**Use a slot.**<br />
`BaguetteOS` comes with a `/etc/profile` script, adding the functions `prefix_add` and `prefix_del` to your shell.
For example, if you want to use an application in the slot `my-awesome-app`, type:
```sh
$ prefix_add my-awesome-app
```
This will change your `$PATH`, allowing you to run applications in `/usr/my-awesome-app`:
```sh
$ echo $PATH
/bin:/usr/baguette/bin:/usr/local/bin:/usr/my-awesome-app/bin
```
**Install a new repository.**<br />
A new repository can be used after adding its address in `/etc/package.conf`:
```
# File `/etc/package.conf`
# Default repositories
https://repos.baguette.netlib.re/$arch/
# Add your repositories here
https://repos.my-awesome-app.com/$arch/
```
You can then update your list of packages and install your application:
```zsh
# package update
# package install my-awesome-app
```
That's all folks!
[service]: https://git.baguette.netlib.re/Baguette/service
[package]: https://git.baguette.netlib.re/Baguette/package
[packaging]: https://git.baguette.netlib.re/Baguette/packaging
[build.zsh]: https://git.baguette.netlib.re/Baguette/build.zsh
[libipc]: https://git.baguette.netlib.re/Baguette/libipc
[todod]: https://git.baguette.netlib.re/Baguette/todod
[webhooksd]: https://git.baguette.netlib.re/Baguette/
[tap-aggregator]: https://git.baguette.netlib.re/Baguette/tap-aggregator
[baguette-gitea]: https://git.baguette.netlib.re/
[baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc
[baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile
[recipes]: https://git.baguette.netlib.re/Baguette/recipes/
[recipe-hello]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/hello/recipe.spec
[recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec
[recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec
[autotools]: https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
[toybox]: http://www.landley.net/toybox/
[cruxinit]: https://crux.nu/gitweb/?p=ports/core.git;a=blob;f=rc/rc;h=26b8ca08d67208ceda4d4004c8333d362bcdc689;hb=HEAD
[ksh]: https://github.com/att/ast
[zsh]: https://www.zsh.org/
[libarchive]: https://libarchive.org/
[bsdcpio]: https://libarchive.org/
[bsdtar]: https://libarchive.org/
[m4]: https://www.gnu.org/software/m4/m4.html
[gmake]: https://www.gnu.org/software/make/
[proot]: https://man.openbsd.org/proot
[crinja]: https://github.com/straight-shoota/crinja
[zig]: https://ziglang.org/
[openbsd]: https://openbsd.org/
[pfsense]: https://www.pfsense.org/
[alpine]: https://alpinelinux.org/
[crux]: https://crux.nu/
[inferno]: http://www.vitanuova.com/inferno/index.html
[plan9]: https://9p.io/plan9/index.html
[morpheus]: https://morpheus.2f30.org/
[suckless]: https://suckless.org/
[cat-v]: http://cat-v.org/
[working-service-asciinema]: https://asciinema.org/a/0p2vGNA1TUmvq0s61Lu0r4TN6
[ruby-memory-bp]: https://web.archive.org/web/20160329122617/http://blog.rubybestpractices.com/posts/ewong/005-Avoiding-system-calls.html
[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
[pledge]: https://man.openbsd.org/pledge.2
[unveil]: https://man.openbsd.org/unveil.2
[retguard]: https://undeadly.org/cgi?action=article&sid=20170819230157
[cveqemu]: https://www.cvedetails.com/vulnerability-list.php?vendor_id=7506&product_id=0&version_id=0&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=276&sha=6055b0330a499f6aed7620adb79dc0cc143e50bc

@ -0,0 +1,237 @@
+++
title = "BaguetteOS: technical choices"
paginate_by = 5
+++
- [Base system](#base)
- [Developers tools](#development)
- [Supported languages for the base system](#languages)
- [Conventions](#conventions)
<a name="base"></a>
# Base system
**Linux kernel**, but we are lurking on the OpenBSD one.<br />
Linux is compatible with most hardware and software, it is fast and we can easily compile a custom version to remove most of the bloat for server usage.
Still, we don't want to rely on Linux-specific components.
At some point, our system will be kernel-agnostic and will be able to run on any BSD as well.
OpenBSD has `pledge` and `unveil` syscalls, which is an elegant way to provide a guarantee on the software behavior.
**Musl.** *reasonable libc for Linux*<br />
It has a reasonable amount of features, it is efficient, provides reasonable binary sizes and static compilation.
Musl is simple, reliable and remove all glibc-specific functions.
Others can be added easily, which is useful for compatibility and comparisons, through [slotting](#slotting).
**Bootable system and rootfs available.**<br />
A bootable system to install in virtual machines or bare metal, a rootfs to use BaguetteOS from any other OS, including non-Linux ones.
**SysV-style init + [CRUX-like /etc/{rc,mdev.conf,...}][baguette-rc]**. *easy to read, easy to adapt*<br />
The init could come from toybox or another minimalist project.
The [rc script from CRUX][cruxinit] is simple to understand and to adapt to any requirements, so we use it.
We also provide some other scripts, like for [profile][baguette-profile] so we can easily manage slotting.
No systemd BS.
**Toybox.** *the megabyte coreutils*<br />
[Toybox][toybox] combines common unix command line utilities together into a single BSD-licensed executable.
It is designed to be simple even to read, and is standards-compliant.
For the base system, that's all we need.
**ksh and zsh**. *the first for scripts and root, the other for users*<br />
[Ksh][ksh] is a very stable and reliable shell from AT&T, trusted by the paranoid people of OpenBSD.
That's a safe choice for a base system and the root account.
On the other hand, we do use [zsh][zsh] daily, as for many users, so we may use it for development or build scripts but not in the base system.
**[Service][service] for service management** *tokenized service description, templating and dumb cli tools for the win* <br />
[See custom tools.](#custom-tools)
**[Package][package] for package management** *simple-to-use, efficient, dead simple code* <br />
[See custom tools.](#custom-tools)
**OpenSSH.** *as we all know and love*<br />
This is required for almost all usages, and for debug.
Let's try not to shoot ourselves in the foot.
That's all you need for starters. Web administrative interface will be added to the base when ready.
<a name="development"></a>
# Development, build tools
**Default building tools** *every tool needed to bootstrap*<br />
Clang (+ LLVM) is the default C (and C++) compiler.
[Libarchive][libarchive] is required for tarballs, packages, webhooks from `packaging`, and both [bsdcpio][bsdcpio] and [bsdtar][bsdtar] (sane implementations of `cpio` and `tar`).
[Auto-tools][autotools] are also required (for SysV init and libarchive).
[m4][m4] and [gnu-make][gmake] are required for compatibility reasons.
**Documentation.**<br />
<