Baguette/website
3
1
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Site review up to recipes.

Browse Source
This commit is contained in:
Philippe PITTOLI 2020-04-26 17:05:54 +02:00
parent b63da4db60
commit a05f0f3308
3 changed files with 231 additions and 105 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
content/_index.md
View File

@ -3,3 +3,11 @@ title = "Baguette - OS, tools and stuff"
paginate_by = 5
+++
Hello! Here a quick links to our <red>currently expanding</red> documentation.
- [BaguetteOS: the French operating system.][baguetteos]
- [Our projects, beyond the OS.][projects]
[baguetteos]: /baguette
[projects]: /projects

322
content/baguette/index.md
View File

@ -106,7 +106,7 @@ nothing more.
<img src="/meanie-makefile.png" alt="meanie meanie dev" class="meanie-img" />
**No Makefile?** *no problem*<br />
Your application or your library lacks a build system? Here a [tool to create makefiles](#build.zsh).
Your application or your library lacks a build system? Here is a [tool to create makefiles](#build.zsh).
It works on any language. *yes, even that one*
**Stable and development versions: same thing.** *slotting, again and again*<br />
@ -303,53 +303,61 @@ firefox-79.0-r8.pkg
# 3. BaguetteOS: custom tools
**Simple ideas, simple implementations.**<br />
Keep reminding to yourself while reading that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br />
Keep reminding to yourself while reading this section that our tools are implemented within just a few hundred lines of code (up to 1500 lines for `service` and `libipc`).<br />
Also, they can easily run on other systems: nothing here is OS specific.
[Feel free to provide a feedback.](#contact)
Here a few pointers:
Here is a few pointers:
- [spec format](#spec-format)
- [package](#package)
- [packaging](#packaging)
- [service](#service)
- [build.zsh](#build.zsh)
- [… and a few other tools](#other-tools)
---
### Main BaguetteOS tools
## Main BaguetteOS tools
<a name="spec-format"></a>
**Spec files.** *our declarative format*<br />
Before presenting our tools, here a file format named `spec` that we tend to use when relevant.
It is *declarative*: we tend not to tell instructions how to do things (copy this file here, download this, etc.) but to describe something (url of the project is `https://example.com/xxx`).
The `spec format` is only composed of variables, lists, code block and named sections.
Here a quick example.
### Spec files: our declarative format
Before presenting our tools, here is a file format named `spec` that we use when relevant.
It is *declarative*: <u>we do not write instructions on *how* to do things</u> (copy this file here, download this, etc.).
Instead, <u>we describe what something is</u>: the URL of the project is `https://example.com/xxx`, for example.
The `spec format` is only composed of variables, lists, code blocks and named sections.
Here is a quick example.
```yaml
# This is a comment
# This is a simple variable instanciation
variable: value
my-variable: value
# This is a inlined list
list: a, b , c
# This is an inlined list
my-list: a, b, c
# This is a multiline list
list:
my-list:
- a
- b
- c
```
This looks a lot like YAML, but now we add `code block` to it: because sometimes we <u>do</u> want to tell instructions.
Up to this point, this looks a lot like YAML, but now we add `code blocks` to it: because sometimes we <u>do</u> want to tell instructions.
```yaml
# We have the URL of the tarballs for a software
software-url: 'https://example.com/my-software/'
# ... and we want to get its last version number
# "@watch" is a block, the name "watch" has a meaning in the application reading the file
# and its tarballs on the website look like this:
# 'https://example.com/my-software/2.6_my-software.tar.gz'
# 'https://example.com/my-software/2.7_my-software.tar.gz'
# 'https://example.com/my-software/2.8_my-software.tar.gz'
# ...
# "@watch" is a block
# in the application reading the file, "watch" is a keyword meaning "command to run to check for new tarball versions"
@watch
# the following is simple shell scripting
curl %{software-url} -o- 2>/dev/null | \
@ -357,83 +365,89 @@ software-url: 'https://example.com/my-software/'
tail -1
```
Sometimes, we want to refer to a file (or directory) and add metadata to it: here `named sections`.
Sometimes, we want to refer to a file (or directory) and add metadata to it through `named sections`.
```yaml
# as for "@watch", "%configuration" has a meaning in the application reading the file
# as for "@watch", "%configuration" is a keyword for the application
# this time, the block has an arbitrary name
%configuration postgresql.conf
# within a named section, we find simple declarations as outside the block
# within a named section, variables are declared in the same way as outside the block
name: database configuration
creation-command: my-script.sh -a -b -c
```
Next, the usage in practice: [packaging](#packaging), [service](#service).
[Come back to top](#top)
[Back to top](#top)
---
<a name="package"></a>
**[Package][package]: our package manager.**<br />
`Package` covers the basics: install, remove, search and provide informations about a package.
### [Package][package]: our package manager
`Package` covers the basics: install, remove, search and provide information about a package.
`Package` can create minimal rootfs, to bootstrap BaguetteOS on another system or to create test environments for example.
Package provides slotting by default: no need for custom environments for each software.
Packages format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database format contains `world`, `installed`, `[package-name]/[slot]/manifest` and `[package-name]/[slot]/meta.spec`.
Package format is a simple `tar` archive containing a `meta.spec` file describing all meta-data about the package (hash, manifest, etc.) and `files.tar.xz` with the files to install.
The database format contains:
- `world`, the file containing the list of available packages
- `installed`, the list of installed packages
- `[package-name]/[slot]/manifest`, the manifest of the installed package
- `[package-name]/[slot]/meta.spec`, the meta-data of the installed package
Package's configuration is a list of repositories, authorized package signing keys and packaging variables (cflags, makeflags, and so on).
`Package` configuration consists of:
- a list of repositories
- authorized package signing keys
- packaging variables (cflags, makeflags, and so on)
Finally, `Package` can easily be expanded, as it only relies on a few hundred lines of Crystal code.
Finally, `Package` can easily be expanded, as it is only a few hundred lines of Crystal code.
[Come back to top](#top)
[Back to top](#top)
---
<a name="packaging"></a>
**[Packaging][packaging]: creates packages.**<br />
### [Packaging][packaging]: the way to create packages for BaguetteOS
Any OS needs a way to create packages to share software, either by sharing sources that need to be compiled or by sharing pre-compiled binaries.
As BaguetteOS is design to provide quickly usable systems, we choose to provide binaries.
As BaguetteOS is designed to provide quickly usable systems, we choose to provide binaries.
`Packaging` uses simple, declarative recipe files with the `spec format` [as we saw earlier](#spec-format).
`Packaging` has a few advantages compared to most used packaging tools:
- **declarative recipes abstract OS specifics**<br />
The same recipe may work for many **native packaging systems** (many OSs), given that packagers provide the right target running dependencies.
This only requires to provide a back-end for the target package manager.
- **auto-split in different packages**<br />
The same recipe may work for many **native packaging systems** (on many OSs), as long as packagers provide the right target running dependencies.
This only requires to provide a back-end for the package format of the target package manager (a back-end to write `.deb` files, for instance).
- **packages are split automatically**<br />
We need to separate binaries, libraries and documentation in different packages, so we can only install what's needed.
Slow and testing systems only require strict minimum.
- **auto-strip of binaries and libraries**<br />
- **binaries and libraries are stripped by default**<br />
By default, a running system does not require debug symbols in its binaries.
- **recipe readability is great**<br />
A few variable declarations are better than a dozen lines of code.
- **trivial shell script patterns become automated**<br />
`Autotools` and `cmake` build systems are detected; packagers should only provide specific parameters for each project.
- **tooling may evolve, very few recipes will be require to change**<br />
`Autotools` and `cmake` build systems are auto-detected; packagers should only provide specific parameters for each project.
- **tooling may change, recipes won't**<br />
Everybody wants to change its build system?
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just `packaging`.
- **it creates hash'ed and signed packages by default**<br />
*(Besides possibly broken tools and possible workarounds,)* this is not a problem for the recipe, just for `packaging`.
- **packages are hashed and signed by default, no extra-tooling**<br />
You need your own set of cryptographic keys, which is created at first use.
- **it creates repositories (automatic at first compilation), helping people maintaining their own set of tools**<br />
- **repositories are created automatically at first compilation, helping people maintaining their own set of tools**<br />
Change the first `prefix` in your [packaging configuration](#packaging-host-config), compile your first package and you have your repository.
That is that simple.
It's <u>that</u> simple.
<img src="/shell-scripting-is-lava.png" alt="shell script is lava" />
<a name="packaging-build-env"></a>
**`Packaging` build environments**<br />
`Packages` creates build environments to test packages before validation.
**`Packaging`'s build environments**<br />
`Packaging` uses `package` to create build environments and to test packages before validation.
It works as follow:
1. creation of a `/tmp/packaging/build-UUID/` directory
1. a `/tmp/packaging/build-UUID/` directory is created
2. sources are downloaded, extracted then compiled<br />
Recipes and [`packaging` host configuration](#packaging-host-config) may add parameters to it: adding steps before compilation, changing `configure` arguments, etc.
The package is compiled for a specific slot, by default `/usr/baguette`.
Recipes and [`packaging` host configuration](#packaging-host-config) may change parameters to the build: adding steps before compilation, changing `configure` arguments, changing the default slotting (`/usr/baguette`), etc.
3. compiled applications and libraries are put in `/tmp/packaging/build-UUID/root` which is used to create the final package
`Packaging` uses `package` to create low-cost build environments since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
In this case, `package` only installs the minimal set of binaries required by the package to build.
Build environments are low-cost since we hardlink binaries into the building rootfs, which is inspired by the [proot][proot] tool on OpenBSD.
`Packaging` only installs the minimal set of binaries required by the package to build.
Besides the target application, a building environment size is <u>only a few kilobytes</u>.
<a name="packaging-host-config"></a>
@ -448,19 +462,22 @@ packages-directory: /usr/local/pkg/
# where to download sources
sources-directory: /usr/local/src/
# the slot we want for our packages
slotting: /usr/baguette
# prefixes for `packaging` running environment and child processes
# = where to search for binaries and libraries for the build
prefixes:
# the first prefix is the default slot used for building application
- /usr/baguette/
- /
# list of environment variables we want to have when building
environment:
# you may choose another compiler, provide some CFLAGS, etc.
# we may choose another compiler, provide some CFLAGS, etc.
- CC: clang
- CFLAGS: -Os -Wall
# next three have special meaning
# next three parameters have special meaning
# to provide parameters to the `./configure` script when building
- configure: --disable-nls --without-gettext
@ -474,11 +491,11 @@ environment:
package-manager: package
```
That's it. You know all about `packaging` configuration.
These parameters may be override by recipes.
These parameters may be overridden by recipes.
<a name="recipes"></a>
**`Packaging` recipes.** *we need to create packages*<br />
A recipe is the way to reproduce something; here we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
A recipe is the way to reproduce something; here, we want to create a package, the recipe should provide all data necessary to be able to reproduce the package.
This means at least having a name for the software and a version (they appear in the package name) and sources (software code).
Let's take an example.
@ -592,12 +609,12 @@ This way, bootstrapping an application and providing it to any system <u>with th
It's actually what we did during the Baguette's bootstrap.
Providing universal recipes could even become a game for patient system administrators.
[Come back to top](#top)
[Back to top](#top)
---
<a name="service"></a>
**[Service][service]: service management.** *not just `kill` or start/stop/status wrapper*<br />
### [Service][service]: service management. <side-note>*not just `kill` or start/stop/status wrapper*</side-note>
Service management often comes with:
- default configuration files, users should learn how to configure them and do it manually
- default user and group, so two instances may have security-involved issues
@ -655,7 +672,7 @@ $ service start wordpress
```
A bit of explanation:
1. first, we add `nginx` to the list of service we want on our system
1. first, we add `nginx` to the list of services we want on our system
2. same thing with `postgresql`
3. then we add `wordpress` and we pass parameters to the service configuration: `domain`, `http` and `database`<br />
`domain` for the domain name, `http` for the HTTP proxy and then the `database` back-end.
@ -668,7 +685,7 @@ A bit of explanation:
2. `postgresql` is started, its internal directories and files are created, then an user, a database and a password for `wordpress` are created
3. a directory is created in `/srv/root/wordpress/` and wordpress files are copied into it, then its configuration is generated
Stopping a service also stops its dependency, if they aren't required elsewhere (or that we explicitely said to keep them running).
Stopping a service also stops its dependencies, if they aren't required elsewhere (or that we explicitely said to keep them running).
Here are a few functionalities `service` brings.
@ -714,7 +731,7 @@ Name: my-awesome-gitea-instance
Type: gitea
Environment: root (prefix)
Consumes:
- postgresql root/postgresql
- database root/postgresql
- http root/nginx
Ports:
- http: 49155
@ -747,32 +764,40 @@ ports: http
*We **currently** uses a bit more configuration for the database, but we will get there at some point.*
Now, a quick look at its configuration template. *we'll skip most, don't worry*<br />
Templating is done with `Jinja2` templates.
<red>TODO: explain a bit more this example</red>
Now, a quick look at its configuration template. *just a sample, we'll skip most, don't worry*<br />
Templating is done with `Jinja2` templates: [see more about Jinja2 templating][crinja].
```YAML
# This is my template comment
[database]
DB_TYPE = postgres
HOST = 127.0.0.1:{{ providers.postgresql.ports.postgresql }}
# providers = list of services behind the tokens
# for example: behind 'database' there is a `postgresql` service
# and we want the main port to use (for a database, 'ports.main' = port to access it remotely)
# See the definition of the postgresql service for details.
HOST = 127.0.0.1:{{ providers.database.ports.main }}
# we have to enter the database name
# by default, databases are configured with a name convention as follow: environment_service_db
# service.id is 'environment/service', which identifies uniquely the service
# to get the right database name, we should take the service.id template variable and replace the "/" by "_" then add "_db"
NAME = {{ service.id | replace("/", "_") }}_db
# by default, the user name is identical to the database name, but without the "_db" suffix
USER = {{ service.id | replace("/", "_") }}
# we have created the `random_password` function, taking a service id in parameter
# `random_password` creates a unique password the first time it is called
# after that, it provides the same password each time it is called with the same service id
# `random_password` enables sharing a password between two service configurations
PASSWD = {{ random_password( service.id ) }}
[repository]
# service.root indicates the service working directory: /srv/<environment>/<service>/
ROOT = {{ service.root }}/repositories
[server]
SSH_DOMAIN = {{ service.domain }}
DOMAIN = {{ service.domain }}
HTTP_PORT = {{ service.ports.http }}
# service.domain = provided domain token value
# service.ports.http = provided ports.http token value
ROOT_URL = http://{{ service.domain }}:{{ service.ports.http }}/
LFS_CONTENT_PATH = {{ service.root }}/data/lfs
```
@ -790,15 +815,69 @@ Then, we could let users manage their own services, but this is of little intere
The most useful thing to do right now is to provide new services.
[Come back to top](#top)
[Back to top](#top)
---
<a name="other-tools"></a>
### Other tools
<a name="build.zsh"></a>
### [Build.zsh][build.zsh]: makefile creation. <side-note>*for mere mortals*</side-note>
Build.zsh creates a makefile from a simple declarative configuration file.
A `Makefile` should:
- **build the application**<br />
This means a few things:
- compile and link your applications
- handle dependencies between files to build
- rebuild only updated parts of the application, for incremental builds
- allow users to rebuid any part of your project independently
- **install the application**<br />
The default installation root directory is `/usr/local` but you should be able to change that easily (and with an environment variable).
- **create a tarball of the project**<br />
Very useful to share your project, <u>whatever your project is</u>.
No matter if it is an application, a library, the language used, etc.
- **have a `make help`**
Here is a quick example of what `build.zsh` can do:
```zsh
package=my-application
version=2.18.3
# target = all the
target=()
```
Here is a real-life example:
```zsh
package=build_zsh # Name of the package.
version=0.2.1 # Version of the package.
targets=(build.zsh) # The things to build or install.
type[build.zsh]=script # How they’re built.
# Using a for loop to add more targets.
# In this example, we’re registering scripts for installation.
for i in build/*.zsh; do
targets+=($i)
type[$i]=script
# Installation in a non-default directory.
install[$i]='$(SHAREDIR)/build.zsh'
# Targets marked as “auto” won’t appear in `make help`.
auto[$i]=true
done
# Files to add to tarballs through `make dist`.
dist=(build.zsh.in build/*.zsh project.zsh Makefile)
```
Don't like Makefiles? <red>We may even add some other back-ends.</red>
[Back to top](#top)
---
<a name="libipc"></a>
**[LibIPC][libipc]: an IPC communication library** *nothing new, yet it still feels fresh*<br />
### [LibIPC][libipc]: an IPC communication library <side-note>*nothing new, yet it still feels fresh*</side-note>
We use this communication library between our services.
@ -833,33 +912,34 @@ Remote remote communications are transparent.
- any client can join remote services via any communication protocol
- any service is implicitly accessible from anywhere, anyhow
---
<a name="build.zsh"></a>
<red>TODO.</red>
**[Build.zsh][build.zsh]: makefile creation.** *for mere mortals*<br />
Build.zsh creates a makefile from a simple declarative configuration file.
It can replace most build systems.
[Back to top](#top)
---
<red>TODO.</red>
**[tap-aggregator][tap-aggregator]: quality assurance & test results aggregation**<br />
<a name="other-tools"></a>
## Other tools
### [tap-aggregator][tap-aggregator]: quality assurance & test results aggregation
[Back to top](#top)
---
<red>TODO: better explanation.</red>
**[Webhooksd][webhooksd]: verify recipes.**<br />
### [Webhooksd][webhooksd]: verify recipes.
Webhooksd provides an automatic verification of the recipes, based on new application or library version.
Paired with a build system, new recipes received in the repository create packages for a couple of architectures (x86_64, ARM, others will follow).
[Back to top](#top)
---
## Still in discussion
For the simple users, we want to provide an unified web interface to manage the system and online services.
We currently use `Crystal` to work on service back-ends for a variety of projects, we are satisfied on a technical level and we are productive, it is highly probable we continue using it.
The front-end is still in discussion: we currently use `livescript` and it is way more usable than plain JS, but it is painful to debug.
So, we need a language for both administration dashboard and online services, here some examples we find interesting for the job:
So, we need a language for both administration dashboard and online services, here are some examples we find interesting for the job:
- Purescript
- haskell-like syntax, with a smaller set of notions
- strongly typed, with type inference
@ -880,21 +960,62 @@ So, we need a language for both administration dashboard and online services, he
The usual way to provide software is to maintain a version of a software or a library, package it into a distribution, then provide it as *the* OS version of the software.
In the long run, software and libraries change, which is no big deal since maintainers verify the consistency of the different versions provided by the OS.
<red>TODO</red>
**Problem:** what happens when two programs need a different version of a library?<br />
**Problem:** what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?<br />
**Problem:** what happens when you want to provide a **very** long term support for your users (see companies running decade-old databases)?
**Problem:**
- what happens when two programs need a different version of a library?<br />
The installation of both may no be possible.
See python from version 2 to 3 as an example: developers knew it will break OS systems.
So, they provided by themselves new names for their binaries (`python-2.7`), and libraries are *by default* packaged into a directory specific for a python version, such as `/usr/lib/python3.8/` and this is a form of slotting.
This is mostly done for languages, by what about other packaged applications and libraries?
- what happens when two libraries are compatible but you want both on your system (see libressl and openssl)?<br />
- what happens when you want to provide a **very** long term support for your users?
*see companies running decade-old OSs and databases*
BaguetteOS has a simple and safe way to let users and maintainers provide packages: `slotting`.
Official OS packages are installed under `/usr/baguette/`, for non-essential programs.
Here, the slot is `baguette`.
Any package outside the official ones are in another named slot.
<red>TODO</red>
Wanna support Python 2.7 **for life**? Just maintain a `python-2.7` slot.
It will be available for everyone.
**This in nothing new, however not often used, and still maybe the best way to handle the problem.**
**This is nothing new, however not often used, and still maybe the best way to handle the problem.**
Others are doing it: snap, flatpak, cpanm, pip, go, stack, cabal, ... *the list seems endless*<br />
They all use slotting... *but*.
Since they are is *yet another package manager for your system*, you need to install all your snap software dependencies from their own dependency tree.
You have now a shit-ton of ways to get software updates, and for *almost* all of them, it's just giving up the idea of having people testing the software before providing it.
**Having an alternate package manager for your system is a fancy way of saying:**<br />
*"I trust all the developers more than my OS maintainers. And fuck testing, who needs that anyway."*
- **snap**<br />
Snap maintains a `/snap` directory on your system, everything is in there.
They even go to a slotting **per recipe version**, which is *kinda* per application version.
See the readme in `/snap/README`, does it ring a bell?
```
/snap/bin - Symlinks to snap applications.
/snap/<snapname>/<revision> - Mountpoint for snap content.
/snap/<snapname>/current - Symlink to current revision, if enabled.
```
- **flatpak**: same as `snap`
- cpanm, pip, stack, cabal, go... and other **developer tools**<br />
These are tools for developers to overcome the relatively slow process of providing a library in an OS.
But this process is **slow for a reason**, taking shortcuts is not safe.
<img src="/can-i-haz-my-firefox.png" alt="wanna the updates at all cost" class="meanie-img" />
<u>*But, BaguetteOS... I still need my last version of Blah!*</u> We gotcha buddy.<br />
You wanna go fast? Try sonic the good slot: `/usr/sonic`.
With this slot, the BaguetteOS maintainers provide the last versions of a variety of applications and libraries.
### How slotting works in BaguetteOS
<red>TODO: this documentation is not available right now. Sure the project is still awesome anyway!</red>
### How use and change slots used
<red>TODO: this documentation is not available right now. Sure the project is still awesome anyway!</red>
### BaguetteOS file system hierarchy
@ -910,30 +1031,19 @@ Any package outside the official ones are in another named slot.
- `/bad`: things that cannot be properly installed or slotted somewhere else
<a name="roadmap"></a>
# Roadmap
<red>TODO</red>
# 5. Roadmap
We currently aim at providing a rootfs with our tools, when we will have enough spare time to contribute.
**Web interface** is for later: we need more time to design its graphical components.
**Web interface is for later**: we need more time to design its graphical components.
On the other hand, back-end should be straightforward.
<a name="contact"></a>
# Contact
# 6. Contact
Do not hesitate to come on our [Mattermost][mattermost].
We will soon have bridges on XMPP and IRC, stay tuned!
# Just dropped things
80211d
networkctl
firewalld
ajouter des outils de gestion (suppression, modification) de préfixes dans l’environnement (PATH, LD_LIBRARY_PATH, PKGCONFIG_PATH, etc.). À faire a minima dans /etc/profile.
partitionnement et formatage des partitions
Avoir un programme qui permet de retrouver de quel paquet provient un fichier (un reverse apk manifest).
[mattermost]: https://team.baguette.netlib.re/
[service]: https://git.baguette.netlib.re/Baguette/service
@ -969,6 +1079,8 @@ Avoir un programme qui permet de retrouver de quel paquet provient un fichier (u
[proot]: https://man.openbsd.org/proot
[crinja]: https://github.com/straight-shoota/crinja
[zig]: https://ziglang.org/
[openbsd]: https://openbsd.org/

6
content/projects/index.md
View File

@ -2,6 +2,12 @@
title = "Projects"
+++
<red>This page isn't ready for public review.</red>
# dnsmanager (and netlib.re)
dnsmanager is a web interface to enable users to register DNS names and manage their zone. It is the software powering [netlib.re](https://netlib.re), a service to provide names for everyone on the Internet.
[netlib.re](https://netlib.re) is kindly operated by [Alsace Réseau Neutre](https://arn-fai.net), a neutral and non-profit Internet Service Provider based in Alsace, France. Don't be shy, come and ask questions!