From a05f0f33086e6a1dd720aa7e841f373ce95e261d Mon Sep 17 00:00:00 2001
From: Philippe PITTOLI
Date: Sun, 26 Apr 2020 17:05:54 +0200
Subject: [PATCH] Site review up to recipes.
---
content/_index.md | 8 +
content/baguette/index.md | 322 +++++++++++++++++++++++++-------------
content/projects/index.md | 6 +
3 files changed, 231 insertions(+), 105 deletions(-)
diff --git a/content/_index.md b/content/_index.md
index fe3e66a..89336d5 100644
--- a/content/_index.md
+++ b/content/_index.md
@@ -3,3 +3,11 @@ title = "Baguette - OS, tools and stuff"
paginate_by = 5
+++
+
+Hello! Here a quick links to our currently expanding documentation.
+
+- [BaguetteOS: the French operating system.][baguetteos]
+- [Our projects, beyond the OS.][projects]
+
+[baguetteos]: /baguette
+[projects]: /projects
diff --git a/content/baguette/index.md b/content/baguette/index.md
index 6aebf45..cbb3a9e 100644
--- a/content/baguette/index.md
+++ b/content/baguette/index.md
@@ -106,7 +106,7 @@ nothing more.
**No Makefile?** *no problem*
-Your application or your library lacks a build system? Here a [tool to create makefiles](#build.zsh).
+Your application or your library lacks a build system? Here is a [tool to create makefiles](#build.zsh).
It works on any language. *yes, even that one*
**Stable and development versions: same thing.** *slotting, again and again*
@@ -303,53 +303,61 @@ firefox-79.0-r8.pkg
# 3. BaguetteOS: custom tools
**Simple ideas, simple implementations.**
-Keep reminding to yourself while reading that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).
+Keep reminding to yourself while reading this section that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).
Also, they can easily run on other systems: nothing here is OS specific.
[Feel free to provide a feedback.](#contact)
-Here a few pointers:
+Here is a few pointers:
- [spec format](#spec-format)
- [package](#package)
- [packaging](#packaging)
- [service](#service)
+- [build.zsh](#build.zsh)
- [… and a few other tools](#other-tools)
---
-### Main BaguetteOS tools
+## Main BaguetteOS tools
-**Spec files.** *our declarative format*
-Before presenting our tools, here a file format named `spec` that we tend to use when relevant.
-It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`).
-The `spec format` is only composed of variables, lists, code block and named sections.
-Here a quick example.
+### Spec files: our declarative format
+Before presenting our tools, here is a file format named `spec` that we use when relevant.
+It is *declarative*: we do not write instructions on *how* to do things (copy this file here, download this, etc.).
+Instead, we describe what something is: the URL of the project is `https://example.com/xxx`, for example.
+The `spec format` is only composed of variables, lists, code blocks and named sections.
+Here is a quick example.
```yaml
# This is a comment
# This is a simple variable instanciation
-variable: value
+my-variable: value
-# This is a inlined list
-list: a, b , c
+# This is an inlined list
+my-list: a, b, c
# This is a multiline list
-list:
+my-list:
- a
- b
- c
```
-This looks a lot like YAML, but now we add `code block` to it: because sometimes we do want to tell instructions.
+Up to this point, this looks a lot like YAML, but now we add `code blocks` to it: because sometimes we do want to tell instructions.
```yaml
# We have the URL of the tarballs for a software
software-url: 'https://example.com/my-software/'
# ... and we want to get its last version number
-# "@watch" is a block, the name "watch" has a meaning in the application reading the file
+# and its tarballs on the website look like this:
+# 'https://example.com/my-software/2.6_my-software.tar.gz'
+# 'https://example.com/my-software/2.7_my-software.tar.gz'
+# 'https://example.com/my-software/2.8_my-software.tar.gz'
+# ...
+# "@watch" is a block
+# in the application reading the file, "watch" is a keyword meaning "command to run to check for new tarball versions"
@watch
# the following is simple shell scripting
curl %{software-url} -o- 2>/dev/null | \
@@ -357,83 +365,89 @@ software-url: 'https://example.com/my-software/'
tail -1
```
-Sometimes, we want to refer to a file (or directory) and add metadata to it: here `named sections`.
+Sometimes, we want to refer to a file (or directory) and add metadata to it through `named sections`.
```yaml
-# as for "@watch", "%configuration" has a meaning in the application reading the file
+# as for "@watch", "%configuration" is a keyword for the application
# this time, the block has an arbitrary name
%configuration postgresql.conf
- # within a named section, we find simple declarations as outside the block
+ # within a named section, variables are declared in the same way as outside the block
name: database configuration
creation-command: my-script.sh -a -b -c
```
Next, the usage in practice: [packaging](#packaging), [service](#service).
-[Come back to top](#top)
+[Back to top](#top)
---
-**[Package][package]: our package manager.**
-`Package` covers the basics: install, remove, search and provide informations about a package.
+### [Package][package]: our package manager
+`Package` covers the basics: install, remove, search and provide information about a package.
`Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example.
Package provides slotting by default: no need for custom environments for each software.
-Packages format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
-The database format contains `world`, `installed`, `[package-name]/[slot]/manifest` and `[package-name]/[slot]/meta.spec`.
+Package format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
+The database format contains:
+- `world`, the file containing the list of available packages
+- `installed`, the list of installed packages
+- `[package-name]/[slot]/manifest`, the manifest of the installed package
+- `[package-name]/[slot]/meta.spec`, the meta-data of the installed package
-Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on).
+`Package` configuration consists of:
+- a list of repositories
+- authorized package signing keys
+- packaging variables (cflags, makeflags, and so on)
-Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code.
+Finally, `Package` can easily be expanded, as it is only a few hundred lines of Crystal code.
-[Come back to top](#top)
+[Back to top](#top)
---
-**[Packaging][packaging]: creates packages.**
+### [Packaging][packaging]: the way to create packages for BaguetteOS
Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries.
-As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries.
+As BaguetteOS is designed to provide quickly usable systems, we choose to provide binaries.
`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
`Packaging` has a few advantages compared to most used packaging tools:
- **declarative recipes abstract OS specifics**
- The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies.
- This only requires to provide a back-end for the target package manager.
-- **auto-split in different packages**
+ The same recipe may work for many **native packaging systems** (on many OSs), as long as packagers provide the right target running dependencies.
+ This only requires to provide a back-end for the package format of the target package manager (a back-end to write `.deb` files, for instance).
+- **packages are split automatically**
We need to separate binaries, libraries and documentation in different packages, so we can only install what's needed.
Slow and testing systems only require strict minimum.
-- **auto-strip of binaries and libraries**
+- **binaries and libraries are stripped by default**
By default, a running system does not require debug symbols in its binaries.
- **recipe readability is great**
A few variable declarations are better than a dozen lines of code.
- **trivial shell script patterns become automated**
- `Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project.
-- **tooling may evolve, very few recipes will be require to change**
+ `Autotools` and `cmake` build systems are auto-detected; packagers should only provide specific parameters for each project.
+- **tooling may change, recipes won't**
Everybody wants to change its build system?
- *(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`.
-- **it creates hash'ed and signed packages by default**
+ *(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just for `packaging`.
+- **packages are hashed and signed by default, no extra-tooling**
You need your own set of cryptographic keys, which is created at first use.
-- **it creates repositories (automatic at first compilation), helping people maintaining their own set of tools**
+- **repositories are created automatically at first compilation, helping people maintaining their own set of tools**
Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
- That is that simple.
+ It's that simple.
-**`Packaging` build environments**
-`Packages` creates build environments to test packages before validation.
+**`Packaging`'s build environments**
+`Packaging` uses `package` to create build environments and to test packages before validation.
It works as follow:
-1. creation of a `/tmp/packaging/build-UUID/` directory
+1. a `/tmp/packaging/build-UUID/` directory is created
2. sources are downloaded, extracted then compiled
- Recipes and [`packaging` host configuration](#packaging-host-config) may add parameters to it: adding steps before compilation, changing `configure` arguments, etc.
- The package is compiled for a specific slot, by default `/usr/baguette`.
+ Recipes and [`packaging` host configuration](#packaging-host-config) may change parameters to the build: adding steps before compilation, changing `configure` arguments, changing the default slotting (`/usr/baguette`), etc.
3. compiled applications and libraries are put in `/tmp/packaging/build-UUID/root` which is used to create the final package
-`Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
-In this case, `package` only installs the minimal set of binaries required by the package to build.
+Build environments are low-cost since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
+`Packaging` only installs the minimal set of binaries required by the package to build.
Besides the target application, a building environment size is only a few kilobytes.
@@ -448,19 +462,22 @@ packages-directory: /usr/local/pkg/
# where to download sources
sources-directory: /usr/local/src/
+# the slot we want for our packages
+slotting: /usr/baguette
+
# prefixes for `packaging` running environment and child processes
+# = where to search for binaries and libraries for the build
prefixes:
- # the first prefix is the default slot used for building application
- /usr/baguette/
- /
# list of environment variables we want to have when building
environment:
- # you may choose another compiler, provide some CFLAGS, etc.
+ # we may choose another compiler, provide some CFLAGS, etc.
- CC: clang
- CFLAGS: -Os -Wall
- # next three have special meaning
+ # next three parameters have special meaning
# to provide parameters to the `./configure` script when building
- configure: --disable-nls --without-gettext
@@ -474,11 +491,11 @@ environment:
package-manager: package
```
That's it. You know all about `packaging` configuration.
-These parameters may be override by recipes.
+These parameters may be overridden by recipes.
**`Packaging` recipes.** *we need to create packages*
-A recipe is the way to reproduce something; here we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
+A recipe is the way to reproduce something; here, we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
This means at least having a name for the software and a version (they appear in the package name) and sources (software code).
Let's take an example.
@@ -592,12 +609,12 @@ This way, bootstrapping an application and providing it to any system with th
It's actually what we did during the Baguette's bootstrap.
Providing universal recipes could even become a game for patient system administrators.
-[Come back to top](#top)
+[Back to top](#top)
---
-**[Service][service]: service management.** *not just `kill` or start/stop/status wrapper*
+### [Service][service]: service management. *not just `kill` or start/stop/status wrapper*
Service management often comes with:
- default configuration files, users should learn how to configure them and do it manually
- default user and group, so two instances may have security-involved issues
@@ -655,7 +672,7 @@ $ service start wordpress
```
A bit of explanation:
-1. first, we add `nginx` to the list of service we want on our system
+1. first, we add `nginx` to the list of services we want on our system
2. same thing with `postgresql`
3. then we add `wordpress` and we pass parameters to the service configuration: `domain`, `http` and `database`
`domain` for the domain name, `http` for the HTTP proxy and then the `database` back-end.
@@ -668,7 +685,7 @@ A bit of explanation:
2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
- Stopping a service also stops its dependency, if they aren't required elsewhere (or that we explicitely said to keep them running).
+ Stopping a service also stops its dependencies, if they aren't required elsewhere (or that we explicitely said to keep them running).
Here are a few functionalities `service` brings.
@@ -714,7 +731,7 @@ Name: my-awesome-gitea-instance
Type: gitea
Environment: root (prefix)
Consumes:
- - postgresql root/postgresql
+ - database root/postgresql
- http root/nginx
Ports:
- http: 49155
@@ -747,32 +764,40 @@ ports: http
*We **currently** uses a bit more configuration for the database, but we will get there at some point.*
-Now, a quick look at its configuration template. *we'll skip most, don't worry*
-Templating is done with `Jinja2` templates.
-
-TODO: explain a bit more this example
+Now, a quick look at its configuration template. *just a sample, we'll skip most, don't worry*
+Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
```YAML
-# This is my template comment
-
[database]
-DB_TYPE = postgres
-
-HOST = 127.0.0.1:{{ providers.postgresql.ports.postgresql }}
+# providers = list of services behind the tokens
+# for example: behind 'database' there is a `postgresql` service
+# and we want the main port to use (for a database, 'ports.main' = port to access it remotely)
+# See the definition of the postgresql service for details.
+HOST = 127.0.0.1:{{ providers.database.ports.main }}
+# we have to enter the database name
+# by default, databases are configured with a name convention as follow: environment_service_db
+# service.id is 'environment/service', which identifies uniquely the service
+# to get the right database name, we should take the service.id template variable and replace the "/" by "_" then add "_db"
NAME = {{ service.id | replace("/", "_") }}_db
+
+# by default, the user name is identical to the database name, but without the "_db" suffix
USER = {{ service.id | replace("/", "_") }}
+
+# we have created the `random_password` function, taking a service id in parameter
+# `random_password` creates a unique password the first time it is called
+# after that, it provides the same password each time it is called with the same service id
+# `random_password` enables sharing a password between two service configurations
PASSWD = {{ random_password( service.id ) }}
[repository]
+# service.root indicates the service working directory: /srv///
ROOT = {{ service.root }}/repositories
[server]
-SSH_DOMAIN = {{ service.domain }}
-DOMAIN = {{ service.domain }}
-HTTP_PORT = {{ service.ports.http }}
+# service.domain = provided domain token value
+# service.ports.http = provided ports.http token value
ROOT_URL = http://{{ service.domain }}:{{ service.ports.http }}/
-LFS_CONTENT_PATH = {{ service.root }}/data/lfs
```
@@ -790,15 +815,69 @@ Then, we could let users manage their own services, but this is of little intere
The most useful thing to do right now is to provide new services.
-[Come back to top](#top)
+[Back to top](#top)
---
-
-### Other tools
+
+### [Build.zsh][build.zsh]: makefile creation. *for mere mortals*
+Build.zsh creates a makefile from a simple declarative configuration file.
+A `Makefile` should:
+- **build the application**
+ This means a few things:
+ - compile and link your applications
+ - handle dependencies between files to build
+ - rebuild only updated parts of the application, for incremental builds
+ - allow users to rebuid any part of your project independently
+- **install the application**
+ The default installation root directory is `/usr/local` but you should be able to change that easily (and with an environment variable).
+- **create a tarball of the project**
+ Very useful to share your project, whatever your project is.
+ No matter if it is an application, a library, the language used, etc.
+- **have a `make help`**
+
+Here is a quick example of what `build.zsh` can do:
+```zsh
+package=my-application
+version=2.18.3
+
+# target = all the
+target=()
+```
+
+Here is a real-life example:
+```zsh
+package=build_zsh # Name of the package.
+version=0.2.1 # Version of the package.
+
+targets=(build.zsh) # The things to build or install.
+type[build.zsh]=script # How they’re built.
+
+# Using a for loop to add more targets.
+# In this example, we’re registering scripts for installation.
+for i in build/*.zsh; do
+ targets+=($i)
+ type[$i]=script
+
+ # Installation in a non-default directory.
+ install[$i]='$(SHAREDIR)/build.zsh'
+
+ # Targets marked as “auto” won’t appear in `make help`.
+ auto[$i]=true
+done
+
+# Files to add to tarballs through `make dist`.
+dist=(build.zsh.in build/*.zsh project.zsh Makefile)
+```
+
+Don't like Makefiles? We may even add some other back-ends.
+
+[Back to top](#top)
+
+---
-**[LibIPC][libipc]: an IPC communication library** *nothing new, yet it still feels fresh*
+### [LibIPC][libipc]: an IPC communication library *nothing new, yet it still feels fresh*
We use this communication library between our services.
@@ -833,33 +912,34 @@ Remote remote communications are transparent.
- any client can join remote services via any communication protocol
- any service is implicitly accessible from anywhere, anyhow
----
-
-
-TODO.
-**[Build.zsh][build.zsh]: makefile creation.** *for mere mortals*
-Build.zsh creates a makefile from a simple declarative configuration file.
-It can replace most build systems.
+[Back to top](#top)
---
-TODO.
-**[tap-aggregator][tap-aggregator]: quality assurance & test results aggregation**
+
+## Other tools
+
+### [tap-aggregator][tap-aggregator]: quality assurance & test results aggregation
+
+[Back to top](#top)
---
-TODO: better explanation.
-**[Webhooksd][webhooksd]: verify recipes.**
+### [Webhooksd][webhooksd]: verify recipes.
Webhooksd provides an automatic verification of the recipes, based on new application or library version.
Paired with a build system, new recipes received in the repository create packages for a couple of architectures (x86_64, ARM, others will follow).
+[Back to top](#top)
+
+---
+
## Still in discussion
For the simple users, we want to provide an unified web interface to manage the system and online services.
We currently use `Crystal` to work on service back-ends for a variety of projects, we are satisfied on a technical level and we are productive, it is highly probable we continue using it.
The front-end is still in discussion: we currently use `livescript` and it is way more usable than plain JS, but it is painful to debug.
-So, we need a language for both administration dashboard and online services, here some examples we find interesting for the job:
+So, we need a language for both administration dashboard and online services, here are some examples we find interesting for the job:
- Purescript
- haskell-like syntax, with a smaller set of notions
- strongly typed, with type inference
@@ -880,21 +960,62 @@ So, we need a language for both administration dashboard and online services, he
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
-TODO
-
-**Problem:** what happens when two programs need a different version of a library?
-**Problem:** what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?
-**Problem:** what happens when you want to provide a **very** long term support for your users (see companies running decade-old databases)?
+**Problem:**
+- what happens when two programs need a different version of a library?
+ The installation of both may no be possible.
+ See python from version 2 to 3 as an example: developers knew it will break OS systems.
+ So, they provided by themselves new names for their binaries (`python-2.7`), and libraries are *by default* packaged into a directory specific for a python version, such as `/usr/lib/python3.8/` and this is a form of slotting.
+ This is mostly done for languages, by what about other packaged applications and libraries?
+- what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?
+- what happens when you want to provide a **very** long term support for your users?
+ *see companies running decade-old OSs and databases*
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
Here, the slot is `baguette`.
Any package outside the official ones are in another named slot.
-TODO
+Wanna support Python 2.7 **for life**? Just maintain a `python-2.7` slot.
+It will be available for everyone.
-**This in nothing new, however not often used, and still maybe the best way to handle the problem.**
+**This is nothing new, however not often used, and still maybe the best way to handle the problem.**
+Others are doing it: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*
+They all use slotting... *but*.
+Since they are is *yet another package manager for your system*, you need to install all your snap software dependencies from their own dependency tree.
+You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
+
+**Having an alternate package manager for your system is a fancy way of saying:**
+*"I trust all the developers more than my OS maintainers. And fuck testing, who needs that anyway."*
+
+- **snap**
+ Snap maintains a `/snap` directory on your system, everything is in there.
+ They even go to a slotting **per recipe version**, which is *kinda* per application version.
+ See the readme in `/snap/README`, does it ring a bell?
+```
+ /snap/bin - Symlinks to snap applications.
+ /snap// - Mountpoint for snap content.
+ /snap//current - Symlink to current revision, if enabled.
+```
+- **flatpak**: same as `snap`
+- cpanm, pip, stack, cabal, go... and other **developer tools**
+ These are tools for developers to overcome the relatively slow process of providing a library in an OS.
+ But this process is **slow for a reason**, taking shortcuts is not safe.
+
+
+
+*But, BaguetteOS... I still need my last version of Blah!* We gotcha buddy.
+You wanna go fast? Try sonic the good slot: `/usr/sonic`.
+With this slot, the BaguetteOS maintainers provide the last versions of a variety of applications and libraries.
+
+
+### How slotting works in BaguetteOS
+
+TODO: this documentation is not available right now. Sure the project is still awesome anyway!
+
+### How use and change slots used
+
+TODO: this documentation is not available right now. Sure the project is still awesome anyway!
### BaguetteOS file system hierarchy
@@ -910,30 +1031,19 @@ Any package outside the official ones are in another named slot.
- `/bad`: things that cannot be properly installed or slotted somewhere else
-# Roadmap
-
-TODO
+# 5. Roadmap
We currently aim at providing a rootfs with our tools, when we will have enough spare time to contribute.
-**Web interface** is for later: we need more time to design its graphical components.
+**Web interface is for later**: we need more time to design its graphical components.
On the other hand, back-end should be straightforward.
-# Contact
+# 6. Contact
Do not hesitate to come on our [Mattermost][mattermost].
We will soon have bridges on XMPP and IRC, stay tuned!
-# Just dropped things
-
-80211d
-networkctl
-firewalld
-ajouter des outils de gestion (suppression, modification) de préfixes dans l’environnement (PATH, LD_LIBRARY_PATH, PKGCONFIG_PATH, etc.). À faire a minima dans /etc/profile.
-partitionnement et formatage des partitions
-Avoir un programme qui permet de retrouver de quel paquet provient un fichier (un reverse apk manifest).
-
[mattermost]: https://team.baguette.netlib.re/
[service]: https://git.baguette.netlib.re/Baguette/service
@@ -969,6 +1079,8 @@ Avoir un programme qui permet de retrouver de quel paquet provient un fichier (u
[proot]: https://man.openbsd.org/proot
+[crinja]: https://github.com/straight-shoota/crinja
+
[zig]: https://ziglang.org/
[openbsd]: https://openbsd.org/
diff --git a/content/projects/index.md b/content/projects/index.md
index 1bd1ad5..52e8774 100644
--- a/content/projects/index.md
+++ b/content/projects/index.md
@@ -2,6 +2,12 @@
title = "Projects"
+++
+
+This page isn't ready for public review.
+
+
+# dnsmanager (and netlib.re)
+
dnsmanager is a web interface to enable users to register DNS names and manage their zone. It is the software powering [netlib.re](https://netlib.re), a service to provide names for everyone on the Internet.
[netlib.re](https://netlib.re) is kindly operated by [Alsace Réseau Neutre](https://arn-fai.net), a neutral and non-profit Internet Service Provider based in Alsace, France. Don't be shy, come and ask questions!