Service rewrite.

content/baguette/index.md

@@ -15,12 +15,13 @@ paginate_by = 5
 # 1. Concise overview
 
 BaguetteOS status: Work In Progress.
+
 A beta will be available circa mid-2020 for advanced users.
+End of the year for simple users.
 
-
+<red>Warning:</red> this documentation is currently mostly oriented toward advanced unix users.
-<red>TODO: explications partie Custom Tools, factorisation, exemples de code et de commandes, liens vers d'autres projets.</red>
+We provide insights on system administration, expose our ideas and point of view.
-
+Documentation for simple users will be split in another page, soon.
-<red>Warning:</red> this documentation is currently oriented toward advanced unix users.
 
 ## Objectives, for simple users
 
@@ -93,14 +94,14 @@ The same program can be installed in several versions without conflicts.
 
 **Simple to contribute to.** *you can focus on providing recipes for packages, we handle the tooling*<br />
 We want fewer and simpler tools as possible, even for packaging applications and libraries.
-BaguetteOS ships [a simple tool to package applications][packaging] and it only requires you to create a very simple recipe for your package.
+BaguetteOS ships [a simple tool to package applications](#packaging) and it only requires you to create a very simple recipe for your package.
 It handles [slotting](#slotting), compiling, stripping binaries and libraries, splitting the result into different packages (`-man`, `-lib`, `-doc`, etc.) then authenticate them and finally recreate the index.
 All that, just by typing `packaging app`, nothing more.
 
 <img src="/meanie-makefile.png" alt="meanie meanie dev" class="meanie-img" />
 
 **No Makefile?** *no problem*<br />
-Your application or your library lacks a build system? Here a [tool to create makefiles][build.zsh].
+Your application or your library lacks a build system? Here a [tool to create makefiles](#build.zsh).
 It works on any language, yes, even that one.
 
 **Stable and development versions: same thing.** *slotting, again and again*<br />
@@ -108,15 +109,13 @@ One of the coolest aspect of slotting is: you don't need to change your system *
 The development of the newest version of the OS is just slotted in `/usr/baguette-dev` and that's it.
 At any point in time you can execute programs from your stable OS, or if you will, change a few environment variables and be in dev.
 
-**New OS, open to explore**<br />
+**New OS, open to explore** *we are open to new ideas*<br />
 BaguetteOS do not suffer from cumbersome historical decisions: no overly engineered package format, no stupidly complex patchwork of mismatch tools.
-BaguetteOS is easy enough to understand then even try to surpass it.
 
 **Easy to write documentation.** *easy to write, hopefully less procrastination*<br />
 Online documentation is written in Markdown (thanks Zola), and man pages too thanks to `scdoc`.
 Every tool is shipped with a man page: no man page, no integration in base.
 
-
 ## Inspiration
 
 - [CRUX][crux], [alpine][alpine]: simple to understand Linux systems
@@ -239,6 +238,28 @@ Our software man-pages are written with `scdoc` so anyone can contribute.
 **[Packaging][packaging] for packaging software and libraries.** *dead simple, intuitive*<br />
 [See custom tools.](#custom-tools)
 
+**Slotting.** *custom file system hierarchy*<br />
+Our FS is not FHS-compliant, partially because of the origin-based slotting.
+There is a strict separation between core system and third party software.<br />
+[More information on slotting.](#slotting)
+- `/usr/baguette` for core system programs
+- `/usr/bad` for non slot-able software
+- `/usr/third-party` for other software
+
+## Languages
+
+We are reluctant to add new languages to the base system.
+We will limit the number of languages required for a system bootstrap.
+For now, bootstrapping requires: `C`, `perl`, `m4`, `python` (for stupid reasons).
+
+However, we think that we can overcome C limitations, explore new possibilities.
+Now, more than ever, we have better alternatives for all non kernel-related tooling.
+That being said: we do not want dynamic languages.
+We need:
+- simple, small and efficient binaries
+- less possible dependencies (not to download half `cpan` or `pypi` for any freaking software)
+
+
 **Crystal language for system tools.** *syntax and productivity of Ruby, the speed of C*<br />
 It is as simple to learn as a dynamic (oriented object) language, while at the same time being almost as fast as C.
 Technically, Crystal is strongly typed which catches errors at compile-time, but with type inference so it is not cumbersome to use.
@@ -248,8 +269,9 @@ Technical choices are reasonable and documented.
 Finally, Crystal has a large library with all we need for our system components.
 
 There is not much of a drawback here.
-Yes, this is a language you have to learn to work with us on a couple of projects (the web interface back-end, the service manager, the package manager and packaging) but it increases our productivity like crazy.
+Yes, this is a language you have to learn to work with us on a couple of projects but you can learn it in about a few days to a week and it increases our productivity like crazy.
-We heard about `nim` and a ton of other languages, but this is the one which reach the sweet spot between these parameters:
+We heard about `nim` and a ton of other languages, lot of them are great candidates, but **choices needed to be made**.
+This is the one reaching the sweet spot between these parameters:
 
 - productivity (the package manager was mostly done in a few days, [and is just a few hundred lines long][package])
 - easy learning (a developer with basic notions of oriented-object can read our code, no black magic here)
@@ -257,23 +279,38 @@ We heard about `nim` and a ton of other languages, but this is the one which rea
 - reasonably deployable (no host dependencies)
 - execution speed
 
+We are also looking at [Zig][zig] for low-level stuff. Wait & see.
 
-**Slotting.** *custom file system hierarchy*<br />
+## naming convention
-Our FS is not FHS-compliant, partially because of the origin-based slotting.
+
-There is a strict separation between core system and third party software.<br />
+```sh
-[More information on slotting.](#slotting)
+name = application name
-- `/usr/baguette` for core system programs
+version = application version
-- `/usr/bad` for non slot-able software
+release = recipe version
-- `/usr/third-party` for other software
+$name-$version-r$release.pkg
+```
 
 <a name="custom-tools"></a>
+<a name="top"></a>
 # 3. BaguetteOS: custom tools
 
-All our tools are designed to be simple to use, to understand, to read.
+**Simple ideas, simple implementations.**<br />
-<u>Feel free to provide a feedback.</u>
+Keep reminding to yourself while reading that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br />
+Also, they can easily run on other systems: nothing here is OS specific.
+
+[Feel free to provide a feedback.](#contact)
+
+Here a few pointers:
+- [spec format](#spec-format)
+- [package](#package)
+- [packaging](#packaging)
+- [service](#service)
+- [… and a few other tools](#other-tools)
 
 ---
 
+### Main BaguetteOS tools
+
 <a name="spec-format"></a>
 **Spec files.** *our declarative format*<br />
 Before presenting our tools, here a file format named `spec` that we tend to use when relevant.
@@ -325,9 +362,9 @@ Sometimes, we want to refer to a file (or directory) and add metadata to it: her
 
 Next, the usage in practice: [packaging](#packaging), [service](#service).
 
----
+[Come back to top](#top)
 
-<red>TODO: explains why it's different / better than other package managers.</red>
+---
 
 <a name="package"></a>
 **[Package][package]: our package manager.**<br />
@@ -343,6 +380,8 @@ Package's configuration is a list of repositories, authorized package signing ke
 
 Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code.
 
+[Come back to top](#top)
+
 ---
 
 <a name="packaging"></a>
@@ -351,19 +390,24 @@ Any OS needs a way to create packages to share software, either by sharing sourc
 As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries.
 `Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
 `Packaging` has a few advantages compared to most used packaging tools:
-- declarative recipes abstact OS specifics<br />
+- **declarative recipes abstract OS specifics**<br />
   The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies.
-  This only requires to provide a backend for the target package manager.
+  This only requires to provide a back-end for the target package manager.
-- recipe readability is great<br />
+- **auto-split in different packages**<br />
+  We need to separate binaries, libraries and documentation in different packages, so we can only install what's needed.
+  Slow and testing systems only require strict minimum.
+- **auto-strip of binaries and libraries**<br />
+  By default, a running system does not require debug symbols in its binaries.
+- **recipe readability is great**<br />
   A few variable declarations are better than a dozen lines of code.
-- trivial shell script patterns become automated<br />
+- **trivial shell script patterns become automated**<br />
   `Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project.
-- tooling may evolve, very few recipes will be require to change<br />
+- **tooling may evolve, very few recipes will be require to change**<br />
   Everybody wants to change its build system?
   *(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`.
-- it creates hash'ed and signed packages by default<br />
+- **it creates hash'ed and signed packages by default**<br />
   You need your own set of cryptographic keys, which is created at first use.
-- it creates repositories (automatic at first compilation), helping people maintaining their own set of tools<br />
+- **it creates repositories (automatic at first compilation), helping people maintaining their own set of tools**<br />
   Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
   That is that simple.
 
@@ -418,7 +462,7 @@ environment:
 	# to provide parameters to the final `make install` command
 	- make install:
 
-# wanna build for another system? not a problem, just add the backend (we currently have `apk` and `package`)
+# wanna build for another system? not a problem, just add the back-end (we currently have `apk` and `package`)
 package-manager: package
 ```
 That's it. You know all about `packaging` configuration.
@@ -515,9 +559,6 @@ When a lot of packages have the same workarounds, we think about detection and i
 If you want to investigate a bit more, you can check [our recipe repository][recipes].
 Feel free to improve these recipes with meta-data, `@watch` code blocks… contributions are welcome.
 
-<a name="packaging-own-repository"></a>
-<red>TODO: talk about creating a repository.</red>
-
 <a name="packaging-future"></a>
 **Future of `packaging`.** *let's be even more declarative*<br />
 As we saw, `packaging` allows maintainers to create very simple and **readable** recipes.
@@ -528,53 +569,229 @@ In the future, manual operations should be reduced even more:
 - by adding new code blocks *before* the `configuration`, `build` and `install` steps<br />
   This would allow performing a few hacks in the directory (a quick `sed` in a file for instance) and still keeping automatic operations: the best of the two worlds.
 
----
+Another possible addition to `packaging` could be to take cross-OS recipes as first-class citizen.
+It was part of the design process of spec files: `named sections` can be used to discriminate information based on the target OS.
+An example:
 
-<red>Factorisation.</red>
+```YAML
+%OS Ubuntu-20.04
+	dependencies: libxxx
+%OS Debian-12
+	dependencies: lib-with-specific-naming-standard-for-reasons
+```
+
+This way, bootstrapping an application and providing it to any system <u>with their own tools</u> could be easy.
+It's actually what we did during the Baguette's bootstrap.
+Providing universal recipes could even become a game for patient system administrators.
+
+[Come back to top](#top)
+
+---
 
 <a name="service"></a>
 **[Service][service]: service management.** *not just `kill` or start/stop/status wrapper*<br />
 Service management often comes with:
 - default configuration files, users should learn how to configure them and do it manually
+- default user and group, so two instances may have security-involved issues
 - a single possible instance, otherwise the configuration has to be heavily changed
 - root-only management, simple users rarely run their own services (except on systemd, kuddos for once)
 - no domain management
 
 These shortcomings imply manual configuration, scripting to manage databases and users, specific tooling for each database and service: this is heavy machinery.
+To overcome drawbacks of having simplistic tools, sys-admins developed all kind of monstrous architectures.
+- **LXC** *chroot + network + software limits*
+- **Qemu + KVM, Xen** *let's add software mimicking hardware's complexity to the mix, telling everyone it's for security and simplicity*
+- **docker** *I don't know how to do simple applications nor packages, so I ship my whole dev environment*<br />
+  Note: we have to admit, packaging on most OS is painful for absolute no good reason.
+- **Chef and Puppet** *the 500 MB running Ruby's code on our virtual machines just to check for new configuration is okay, memory is cheap right?*<br />
+  We talk about the importance of security from time to time, but running a software designed by people telling [it's okay not to free memory][ruby-memory-bp] is far from being wise.
+- **Ansible** *templating your applications… from another machine*<br />
+  As Chef and Puppet, ansible provides templating for applications, this time configuration propagation is **way** simpler since it uses well-known, trusted and loved ssh.
+  Still, templating is done for remote machines, as it is intended for server deployments: this is **a sys-admin tool**.
+  The introduction page already talks about `cloud provisionning` and `intra-service orchestration` on the first line, telling that you really need to setup SSH keys, etc.
+  As for many tools in IT: this is not for simple users.
+
 Simple users:
 1. should be only required to provide absolute necessary information for their services
 2. should be able to run as many services they want
 3. shouldn't have to learn configuration syntax for their services
 4. shouldn't be afraid of updates
+5. ... and should have BACKUPS! Where are they? We should have that **by default** on our systems over 20 years ago.
+
+And advanced users should have an uncomplicated CLI tool to do that.
+
+
+**Let's take an example with `service`** *better than a thousand words*
+
+```sh
+# We want a wordpress service, proxied by a nginx and using postgresql as DBMS
+
+# 1. we add an nginx
+$ service add nginx
+```
+
+```sh
+# 2. we add the database
+$ service add postgresql
+```
+
+```sh
+# 3. we add the wordpress
+#    by default, it uses available http proxy and database, but let's be "verbose"
+$ service add wordpress domain=example.com http=nginx database=postgresql
+```
+
+```sh
+# 4. we start the wordpress
+$ service start wordpress
+```
+
+A bit of explanation:
+1. first, we add `nginx` to the list of service we want on our system
+2. same thing with `postgresql`
+3. then we add `wordpress` and we pass parameters to the service configuration: `domain`, `http` and `database`<br />
+	`domain` for the domain name, `http` for the HTTP proxy and then the `database` back-end.
+	Up to this point, nothing even started.
+4. Finally, we start the service.<br />
+	Since `service` knows the dependence graph, it starts other services before wordpress (which actually doesn't have any binary to run *per se*).
+	The configuration of a service is made when `service` starts it.
+	In this case:
+	1. `nginx` is started as a proxy for `example.com`
+	2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
+	3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
+
+	Stopping a service also stops its dependency, if they aren't required elsewhere (or that we explicitely said to keep them running).
+
 
 Here are a few functionalities `service` brings.
-1. **domain-wise service configuration**<br />
+1. **uncomplicated domain-wise service configuration**
-   Example: `service add wordpress domain=example.com`
 
 2. **templates** *configuration files are generated by templates and user data*<br />
    When we want a software, for instance a blog, we want to provide the minimum informations it requires to work and that's it.
-   When `service` is invoked to start a service, it verifies if a configuration file is installed or create it.
+   When `service` starts a service, it verifies if its configuration file is installed and up-to-date, or create it.
    Users shouldn't need to manually change the configuration.
    <u>Syntax may change at any time without breaking a single service</u>, since the configuration will smoothly be regenerated with useful information at start-up.
 
 3. **environments**<br />
-   Each service can be installed in a specific environment (read: a custom rootfs).
+   Each service can be installed in a specific environment (read: a custom rootfs, a virtual machine, etc.).
-   Example: `service add wordpress example.com testing-env`
 
-4. **unified way to configure the system**<br />
+4. **automatic user and group creation**<br />
+   Each service needs to be separated from other users on the system, for security reasons.
+   Let's make it the default.
+
+5. **tokens** *service a needs b, b needs c, when a starts, b and c start and are configured*<br />
+   `service` knows the relation between services, and use it to configure them through the templates.
+   *everything is smoother now*
+
+6. **automatic backup solution**<br />
+   Since we know each running database, service configuration and data directories, we can backup everything once a backup server has been configured.
+   Of course, we can add a lot of parameters to have a fine-grained backup solution.
+
+```sh
+# not currently done, but will look like this
+$ backup add ssh:user@example.com:/srv/backup
+```
+
+7. **unified way to configure the system** *best user experience possible*<br />
    It alleviates the need for manual configuration.
    For example, adding a Wordpress service will automatically change the `nginx` configuration, create a new database and a new user in `mariadb` for this specific service.
    If several `nginx` are required, ports will be registered and automatically managed for each instance, no need for user input.<br />
    Behind the scene, it's a simple token system with configuration templating!<br />
-   No heavy machinery here, and we'll keep it that way.
+   <u>No heavy machinery here, and we'll keep it that way.</u>
 
+8. **... and understandable tooling output** *for god's sake!*<br />
+```sh
+$ sudo service show my-awesome-gitea-instance
+Name:            my-awesome-gitea-instance
+Type:            gitea
+Environment:     root (prefix)
+Consumes:
+  - postgresql   root/postgresql
+  - http         root/nginx
+Ports:
+  - http:        49155
+```
+
+**`Service`: a service example.**<br />
+A service configuration has two parts: the service description and its configuration file template(s).
+Let's use `gitea` as example, the `git` web server handling [our repositories][baguette-gitea].
+
+```YAML
+# `gitea.spec`
+
+# The actual command to run to start the service
+command: gitea -C . -w . -c gitea.cfg
+
+# what tokens the service needs, see that http is optional
+consumes: database, http?
+
+# does the service requires a domain name to be configured?
+# this name may be used in the service configuration, or its (forward|backward) dependencies
+requires-domain: true
+
+# the service requires a simgle port to be open
+# by default, 80
+ports: http
+
+# `%configuration` indicates the name of the configuration template
+%configuration gitea.cfg
+```
+
+*We **currently** uses a bit more configuration for the database, but we will get there at some point.*
+
+Now, a quick look at its configuration template. *we'll skip most, don't worry*<br />
+Templating is done with `Jinja2` templates.
+
+<red>TODO: explain a bit more this example</red>
+
+```YAML
+# This is my template comment
+
+[database]
+DB_TYPE  = postgres
+
+HOST     = 127.0.0.1:{{ providers.postgresql.ports.postgresql }}
+
+NAME     = {{ service.id | replace("/", "_") }}_db
+USER     = {{ service.id | replace("/", "_") }}
+PASSWD   = {{ random_password( service.id ) }}
+
+[repository]
+ROOT = {{ service.root }}/repositories
+
+[server]
+SSH_DOMAIN       = {{ service.domain }}
+DOMAIN           = {{ service.domain }}
+HTTP_PORT        = {{ service.ports.http }}
+ROOT_URL         = http://{{ service.domain }}:{{ service.ports.http }}/
+LFS_CONTENT_PATH = {{ service.root }}/data/lfs
+```
+
+
+**Current implementation of `service`**<br />
+It [currently works][working-service-asciinema], and we just need to add more services!
+Every new corner case will be thoroughly investigated to provide the smoothest experience possible under BaguetteOS.
+We currently use it for a few services, such as the nginx providing this very website.
+
+Currently, there are no fancy environments, just plain directories.
+
+**Future of `service`**<br />
+First, we want to work on databases, to export data and get a simple backup service.
+
+Then, we could let users manage their own services, but this is of little interest in practice since services are running under different users already.
+
+The most useful thing to do right now is to provide new services.
+
+[Come back to top](#top)
 
 ---
 
-<red>Better introduction.</red>
+<a name="other-tools"></a>
+### Other tools
 
 <a name="libipc"></a>
 **[LibIPC][libipc]: an IPC communication library** *nothing new, yet it still feels fresh*<br />
+We use this communication library between our services.
 
 
 1. Applications should talk to each other
@@ -584,12 +801,6 @@ Here are a few functionalities `service` brings.
 LibIPC is currently used for the administration dashboard, the web interface for the services, [for a kanban][todod] and several other tools we use for collaboration.
 It provides a way to communicate between clients and services using simple unix sockets behind the scene.
 
-<red>Explain remote communications.</red>
-Remote remote communications are transparent.
-    - clients and services do not need remote communication
-    - any client can join remote services via any communication protocol
-    - any service is implicitly accessible from anywhere, anyhow
-
 C library with Crystal bindings (other languages coming soon)
 
 
@@ -602,12 +813,21 @@ server.loop do |message|
 end
 ```
 
-<red>TODO: show that's easy to write even in plain C.</red>
+That's easy to write [even in plain C](https://git.baguette.netlib.re/Baguette/libipc/src/branch/master/examples/pongd.c).
 
-<red>TODO: performances are crazy, we have to tell the world.</red>
+
+Its explanation goes beyond the scope of this page… but the tool is awesome and performances are crazy (we have to tell the world!).
+[Just go and check!][libipc]
+
+<red>Explain remote communications.</red>
+Remote remote communications are transparent.
+    - clients and services do not need remote communication
+    - any client can join remote services via any communication protocol
+    - any service is implicitly accessible from anywhere, anyhow
 
 ---
 
+<a name="build.zsh"></a>
 <red>TODO.</red>
 **[Build.zsh][build.zsh]: makefile creation.** *for mere mortals*<br />
 Build.zsh creates a makefile from a simple declarative configuration file.
@@ -688,16 +908,37 @@ Any package outside the official ones are in another named slot.
 
 We currently aim at providing a rootfs with our tools, when we will have enough spare time to contribute.
 
+**Web interface** is for later: we need more time to design its graphical components.
+On the other hand, back-end should be straightforward.
 
+<a name="contact"></a>
+# Contact
+
+Do not hesitate to come on our [Mattermost][mattermost].
+We will soon have bridges on XMPP and IRC, stay tuned!
+
+# Just dropped things
+
+80211d
+networkctl
+firewalld
+ajouter des outils de gestion (suppression, modification) de préfixes dans l’environnement (PATH, LD_LIBRARY_PATH, PKGCONFIG_PATH, etc.). À faire a minima dans /etc/profile.
+partitionnement et formatage des partitions
+Avoir un programme qui permet de retrouver de quel paquet provient un fichier (un reverse apk manifest).
+
+[mattermost]: https://team.baguette.netlib.re/
 
 [service]: https://git.baguette.netlib.re/Baguette/service
 [package]: https://git.baguette.netlib.re/Baguette/package
 [packaging]: https://git.baguette.netlib.re/Baguette/packaging
 [build.zsh]: https://git.baguette.netlib.re/Baguette/build.zsh
 [libipc]: https://git.baguette.netlib.re/Baguette/libipc
+[todod]: https://git.baguette.netlib.re/Baguette/todod
 [webhooksd]: https://git.baguette.netlib.re/Baguette/
 [tap-aggregator]: https://git.baguette.netlib.re/Baguette/tap-aggregator
 
+[baguette-gitea]: https://git.baguette.netlib.re/
+
 [baguette-rc]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc
 [baguette-profile]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/rc/profile
 
@@ -706,6 +947,7 @@ We currently aim at providing a rootfs with our tools, when we will have enough
 [recipe-dhcpcd]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/dhcpcd/recipe.spec
 [recipe-alsautils]: https://git.baguette.netlib.re/Baguette/recipes/src/branch/master/alsa-utils/recipe.spec
 
+[autotools]: https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
 [toybox]: http://www.landley.net/toybox/
 
 [cruxinit]: https://crux.nu/gitweb/?p=ports/core.git;a=blob;f=rc/rc;h=26b8ca08d67208ceda4d4004c8333d362bcdc689;hb=HEAD
@@ -719,6 +961,8 @@ We currently aim at providing a rootfs with our tools, when we will have enough
 
 [proot]: https://man.openbsd.org/proot
 
+[zig]: https://ziglang.org/
+
 [openbsd]: https://openbsd.org/
 [pfsense]: https://www.pfsense.org/
 [alpine]: https://alpinelinux.org/
@@ -729,3 +973,6 @@ We currently aim at providing a rootfs with our tools, when we will have enough
 [suckless]: https://suckless.org/
 [cat-v]: http://cat-v.org/
 
+[working-service-asciinema]: https://asciinema.org/a/0p2vGNA1TUmvq0s61Lu0r4TN6
+[ruby-memory-bp]: https://web.archive.org/web/20160329122617/http://blog.rubybestpractices.com/posts/ewong/005-Avoiding-system-calls.html
+