.ds VERSION 0.5.1
.ds POINT
.so macros.roff
.de dq
\[lq]\\$1\[rq]\c
.shift
\&\\$1
..
.de TREE1
.QP
.ps -3
.vs -3
.KS
.ft CW
.b1
.nf
..
.de TREE2
.ft
.fi
.b2
.ps
.vs
.KE
.QE
..
.de CLASS
.I \\$*
..
.de FUNCTION_CALL
.I "\\$*"
..
.
.de COMMAND
.I "\\$*"
..
.de DIRECTORY
.ps -3
\f[CW]\\$*\f[]
.ps +3
..
.de PRIMARY_KEY
.I \\$1 \\$2 \\$3
..
.de FOREIGN_KEY
.I \\$1 \\$2 \\$3
..
.
.
. \" The document starts here.
.
.TITLE Document Oriented DataBase (DODB)
.AUTHOR Philippe PITTOLI
.ABSTRACT1
DODB is a document-oriented database library, enabling a very simple way to store applications' data without external dependencies by storing serialized
.I documents
(basically any data type) in plain files.
The objective is to avoid complex traditional relational databases and to explore a more straightforward way to handle data, to have a tool anyone can
.B "read and understand entirely" .
To speed-up searches, attributes of these documents can be used as indexes.
DODB can provide a filesystem representation of those indexes to enable off-application data manipulation with the most basic tools, such as
.I ls
or even a file explorer.

This document briefly presents DODB and its main differences with other database engines.
Limits of such approach are discussed.
An experiment is described and analyzed to understand the performance that can be expected.
.ABSTRACT2
.SINGLE_COLUMN
.br
.po +11.5c
.nf
Document sync'ed with DODB \*[VERSION]
.fi
.br
.po
.SECTION Introduction to DODB
A database consists in managing data, enabling queries to add, to retrieve, to modify and to delete a piece of information.
These actions are grouped under the acronym CRUD: creation, retrieval, update and deletion.
CRUD operations are the foundation for the most basic databases.
Yet, almost every single database engine goes far beyond this minimalistic set of features.

Although everyone using the filesystem of their computer as some sort of database (based on previous definition) by storing raw data (files) in a hierarchical manner (directories), computer science classes introduce a particularly convoluted way of managing data.
Universities all around the world teach about Structured Query Language (SQL) and relational databases.
These two concepts are closely interlinked and require a brief explanation.

.UL "Relational databases"
are built around the idea to describe data to a database engine so it can optimize operations and storage.
Data is put into
.I tables ,
with each column being an attribute of the stored data and each line being a new entry.
A database is a list of tables with relations between them.

As an example, let's imagine a database of a movie theater.
The database will have a
.I table
for the list of movies they have
.PRIMARY_KEY idmovie , (
title, duration, synopsis),
a table for the scheduling
.PRIMARY_KEY idschedule , (
.FOREIGN_KEY idmovie ,
.FOREIGN_KEY idroom ,
time slot),
a table for the rooms
.PRIMARY_KEY idroom , (
name), etc.
Tables have relations, for example the table "scheduling" has a column
.I idmovie
which points to entries in the "movie" table.

.UL "The SQL language"
enables CRUD operations on databases: creation, retrieval, update and deletion of entries.
SQL also enables administrative operations on the databases themselves: creating databases and tables, managing users with fine-grained authorizations, etc.
SQL is used between the application and the database, to perform operations and to provide results when due.
SQL is also used
.UL outside
the application, by admins for managing databases and potentially by
.I non-developer
users to retrieve data without a dedicated interface\*[*].
.FOOTNOTE1
One of the first objectives of SQL was to enable a class of
.I non-developer
users to talk directly to the database so they can access the data without bothering the developers.
This has value for many companies and organizations.
.FOOTNOTE2

Many tools were used or even developed over the years specifically to aleviate the inherent complexity and limitations of traditional SQL databases.
For example, designing databases becomes difficult when the list of tables grows;
Unified Modeling Language (UML) is then used to provide a graphical overview of the relations between tables.
SQL databases may be fast to retrieve data despite complicated operations, but when multiple sequential operations are required they become slow because of all the back-and-forths with the application;
thus, SQL databases can be scripted to automate operations and to provide a massive speed up
.I "stored procedures" , (
see
.I "PL/SQL" ).
Moreover, the latency between the database and the application makes internet-facing applications require parallelism to handle a high number of clients (or even moderate by today's standards), via multiple threads or concurrent applications.
Furthermore, writing SQL requests requires a lot of boilerplate since there is no integration in the programming languages, leading to multiple function calls for any operation on the database;
thus, object-relational mapping (ORM) libraries were created to reduce the massive code duplication.
And so on.

For many reasons, SQL is not a silver bullet to
.I solve
the database problem.
The encountered difficulties mentioned above and the original objectives of SQL not being universal\*[*], other database designs were created\*[*].
.FOOTNOTE1
To say the least!
Not everyone needs to let users access the database without going through the application.
For instance, writing a \f[I]blog\f[] for a small event or to share small stories about your life doesn't require manual operations on the database, fortunately.
.FOOTNOTE2
.FOOTNOTE1
A lot of designs won't be mentioned here.
The actual history of databases is often quite unclear since the categories of databases are sometimes vague, underspecified.
As mentioned, SQL is not a silver bullet and a lot of developers shifted towards other solutions, that's the important part.
.FOOTNOTE2
The NoSQL movement started because the stated goals of many actors from the early Web boom were different from SQL.
The need for very fast operations far exceeded what was practical at the moment with SQL.
This led to the use of more basic methods to manage data such as
.I "key-value stores" ,
which simply associate a value with an
.I index
for fast retrieval.
In this case, there is no need for the database to have
.I tables ,
data may be untyped, the entries may even have different attributes.
Since homogeneity is not necessary anymore, databases have fewer (or different) constraints.
Document-oriented databases are a sub-class of key-value stores, where metadata can be extracted from the entries for further optimizations.
And that's exactly what is being done in Document Oriented DataBase (DODB).

.UL "The stated goal of DODB"
is to provide a simple and easy-to-use
.UL library
for developers to perform CRUD operations on documents (undescribed data structures).
DODB aims basic to medium-sized projects, up to a few million entries\*[*].
.FOOTNOTE1
See the section
.dq "Limits of DODB" .
.FOOTNOTE2
Code simplicity implies hackability.
Traditional SQL relational databases have a snowballing effect on code complexity, including for applications with basic requirements.
However, DODB may be a great starting point to implement more sophisticated features for creative minds.

.UL "The non-goals of DODB"
are:
.STARTBULLET
.BULLET to provide a generic library w
.ENDBULLET

.UL "Contrary to SQL" ,
DODB has a very narrow scope: to provide a library enabling to store, to retrieve, to modify and to delete data.
In this way, DODB transforms any application in a database manager.
DODB doesn't provide an interactive shell, there is no request language to perform arbitrary operations on the database, no statistical optimizations of the requests based on query frequencies, etc.
Instead, DODB reduces the complexity of the infrastructure, stores data in plain files and enables simple manual scripting with widespread unix tools.
Simplicity is key.

.UL "Contrary to other NoSQL databases" ,
DODB doesn't provide an application but a library, nothing else.
The idea is to help developers to store their data themselves, not depending on
. I yet-another-all-in-one
massive tool.
The library writes (and removes) data on a storage device, has a few retrieval and update mechanisms and that's it\*[*].
.FOOTNOTE1
The lack of features
.I is
the feature.
Even with that motto, the tool still is expected to be convenient for most applications.
.FOOTNOTE2

Section 2 provides an extensive documentation on how DODB works and how to use it.
This section also presents the concept of "triggers" (automatic actions on database modification).
Section 3 introduces caches in both the database and triggers.
Section 4 presents the Common database, an implementation of DODB that should be relevant for most applications.
Section 5 presents the RAM-only database, for short-lived (temporary) data.
Section 6 is about memory-constrained environments.
Section 7 presents a few experiments to provide an overview of the performance you can expect from this approach.
Section 8 describes the limitations of DODB and its current implementation.
Section 9 presents the related work, alternative approaches and implementations.
Section 10 lays out future work on this project.
Section 11 presents a real-world usage of DODB.
Finally, section 12 provides a conclusion.
.
.SECTION How DODB works and basic usage
DODB is a hash table.
The key of the hash is an auto-incremented number and the value is the stored data.
The following section will explain how to use DODB for basic cases including the few added mechanisms to speed-up searches.
Also, the filesystem representation of the data will be presented since it enables easy off-application searches.

The presented code is in Crystal such as the DODB library.
Keep in mind that this document is all about the method more than the current implementation.
Anyone could implement the exact same library in almost every other language.
.
.SS Before starting: the example database
First things first, the following code is the structure used in the rest of the document to present the different aspects of DODB.
This is a simple object
.I Car ,
with a name, a color and a list of associated keywords (fast, elegant, etc.).
.SOURCE Ruby ps=9 vs=10
class Car
	property name     : String        # ex: Corvet
	property color    : String        # ex: red
	property keywords : Array(String) # ex: [fast, impressive]
end
.SOURCE
.
.SS DODB basic usage
Let's create a DODB database for our cars.
.SOURCE Ruby ps=9 vs=10
# Database creation
database = DODB::Storage::Uncached(Car).new "path/to/db-cars"

# Add an element to the db
database << Car.new "Corvet", "red", ["elegant", "fast"]

# Access all objects in the database
database.each do |car|
	pp! car
end
.SOURCE
Inserted entries are serialized\*[*] and written in a dedicated file.
.FOOTNOTE1
Serialization is currently in JSON.
.[
JSON
.]
CBOR
.[
CBOR
.]
is a work-in-progress.
Nothing binds DODB to a particular format.
.FOOTNOTE2
The key of the hash is a number, auto-incremented, used as the name of the stored file.
The following example shows the content of the file system after adding the first car.
.TREE1
$ tree db-cars/
db-cars/
|-- data
|   `-- 0000000000   <- the first car in the database
`-- last-index
.TREE2
In this example, the directory
.I db-cars/data
contains the serialized value, with a formated number as file name.
The file "0000000000" contains the following:
.QP
.SOURCE JSON ps=9 vs=10
{
  "name": "Corvet",
  "color": "red",
  "keywords": [
    "elegant",
    "fast"
  ]
}
.SOURCE
The car is serialized as expected in the file
.I 0000000000 .
.QE
.
.
The key of the entry (its number) is required to retrieve, to modify or to delete it.
.
.QP
.SOURCE Ruby ps=9 vs=10
# Get a value based on its key.
database[key]

# Update a value based on its key.
database[key] = new_value

# Delete a value based on its key.
database.delete 0
.SOURCE
.QE
.
The function
.FUNCTION_CALL each_with_key
lists the entries with their keys.
.
.QP
.SOURCE Ruby ps=9 vs=10
database.each_with_key do |value, key|
	puts "#{key}: #{value}"
end
.SOURCE
.QE
Of course, browsing the entire database to find a value (or its key) is a waste of resources and isn't practical for any non-trivial database.
That is when indexes come into play.
.
.
.SS Triggers
A simple way to quickly retrieve a piece of data is to create
.I indexes
based on its attributes.
When a value is inserted, modified or deleted from the database, an action can be performed automatically thanks to a user recorded callback.
Callbacks are named
.I triggers
in DODB and are used mainly to index data.
There are currently three main indexing triggers in
.CLASS DODB :
basic indexes, partitions and tags.
.
.SSS Basic indexes (1 to 1 relations)
Basic indexes
.CLASS DODB::Trigger::Index ) (
represent one-to-one relations, such as an index in SQL.
In the Car database, each car has a dedicated (unique) name.
This
.I name
attribute can be used to speed-up searches.
.QP
.SOURCE Ruby ps=9 vs=10
# Create an index based on the "name" attribute of the cars.
cars_by_name = cars.new_index "name", do |car|
	car.name
end

# Two other ways to say the same thing, thanks to the Crystal syntax:
cars_by_name = cars.new_index "name", { |car| car.name }
cars_by_name = cars.new_index "name", &.name
.SOURCE
The above code presents the creation of a new index on the database
.I cars .
An index requires a name (a simple string), which simply is "name" in this case since the indexed value is the name of the cars.
An index also requires a callback, a procedure to extract the value used for indexing.
In this case, the procedure takes a car as a parameter and returns its "name" attribute\*[*].
.FOOTNOTE1
This procedure can be arbitrarily complex and include any necessary data transformation.
For example, the netlibre project (discussed later in the papper) indexes their users' email, but emails are first encoded in base64 to avoid messing around with the filesystem.
.FOOTNOTE2

Once the index has been created, every inserted or modified entry in the database will be indexed.
Adding a trigger provides an
.I object
used to manipulate the database based on the related attribute.
Let's call it an
.I "index object" .
In the code above, the
.I "index object"
is named
.I "cars_by_name" .
.QE

The
.I "index object"
has several useful functions.
.QP
.SOURCE Ruby ps=9 vs=10
# Retrieve the car named "Corvet".
corvet = cars_by_name.get? "Corvet"

# Modify the car named "Corvet".
new_car = Car.new "Corvet", "green", ["eco-friendly"]
cars_by_name.update "Corvet", new_car

# In case the index hasn't changed (the name attribute in this example),
# the update can be even simpler.
cars_by_name.update new_car

# Delete the car named "Corvet".
cars_by_name.delete "Corvet"
.SOURCE
A car can now be searched, modified or deleted based on its name.
.QE
.
.
On the filesystem, indexes are represented as symbolic links.
.TREE1
storage
+-- data
|    `-- 0000000000   <- the car named "Corvet"
`-- indexes
      `-- by_name
          `-- Corvet -> ../../data/0000000000
.TREE2
.QP
As shown, the file "Corvet" is a symbolic link to a data file.
The name of the symlink file has been extracted from the value itself, enabling to list all the cars and their names with a simple
.COMMAND ls
in the
.DIRECTORY storage/indexes/by_name/
directory.
.QE
.
The basic indexes as shown in this section already give a taste of what is possible to do with DODB.
The following triggers will cover some other usual cases.
.
.
.SSS Partitions (1 to n relations)
An attribute can have a value that is shared by other entries in the database, such as the
.I color
attribute of our cars.

.QP
.SOURCE Ruby ps=9 vs=10
# Create a partition based on the "color" attribute of the cars.
cars_by_color = database.new_partition "color", do |car|
	car.color
end

# Shortcut:
cars_by_color = database.new_partition "color", &.color
.SOURCE
As with basic indexes, once the partition is asked to the database, every new or modified entry will be indexed.
.QE

.KS
Let's imagine having 3 cars, one is blue and the other two are red.
.TREE1
$ tree db-cars/
db-cars
+-- data
|    +-- 0000000000   <- this car is blue
|    +-- 0000000001   <- this car is red
|    `-- 0000000002   <- this car is red, too
|   ...
`-- partitions
      `-- by_color
        +-- blue
             `-- 0000000000 -> 0000000000
        `-- red
             +-- 0000000001 -> 0000000001
             `-- 0000000002 -> 0000000002
.TREE2
.QP
Listing all the blue cars is simple as running
.COMMAND ls
in the
.DIRECTORY db-cars/partitions/by_color/blue
directory!
.QE
.KE
.
.
.
.SSS Tags (n to n relations)
Tags are basically partitions but the indexed attribute can have multiple values.
.
.QP
.SOURCE Ruby ps=9 vs=10
# Create a tag based on the "keywords" attribute of the cars.
cars_by_keywords = database.new_tags "keywords", do |car|
	car.keywords
end

# Shortcut:
cars_by_keywords = database.new_tags "keywords", &.keywords
.SOURCE
As with other indexes, once the tag is requested to the database, every new or modified entry will be indexed.
.QE
.
.
.KS
Let's imagine having two cars with different associated keywords.
.TREE1
$ tree db-cars/
db-cars
+-- data
|    +-- 0000000000   <- this car is fast and cheap
|    `-- 0000000001   <- this car is fast and elegant
`-- tags
      `-- by_keywords
        +-- cheap
            `-- 0000000000 -> 0000000000
        +-- elegant
            `-- 0000000001 -> 0000000001
        `-- fast
            +-- 0000000000 -> 0000000000
            `-- 0000000001 -> 0000000001
.TREE2
.QP
Listing all the fast cars is simple as running
.COMMAND ls
in the
.DIRECTORY db-cars/tags/by_keywords/fast
directory!
And only a simple intersection of the files that are several directories is necessary to get cars which are both fast and cheap for example\*[*].
.QE
.KE
.FOOTNOTE1
A simple command is necessary to get cars that are both fast and cheap:
.br
.COMMAND "ls fast/* cheap/* | cut -f 2 -d '/' | sort | uniq -c | awk '{ print $1, $2 }' | grep ^2"
.br
This may appear barbaric for the unprepared mind, but this one-liner only includes basic commands most sysadmin understand and perhaps use daily.
Also, this can be as easily hidden in a very nice user-friendly command.
.FOOTNOTE2
.
.
.
.SSS Side note about triggers
DODB presents a few possible triggers (basic indexes, partitions and tags) which respond to an obvious need for fast searches and retrevial.
Though, the implementation involving an heavy use of the filesystem via the creation of symlinks comes from a certain vision about how a database could behave to provide a practical way for users to query the database
.UL "outside the application" .

Other kinds of triggers could
.B easily
be implemented in addition of those presented.
These new triggers may have completely different objectives\*[*], methods and performance\*[*].
.FOOTNOTE1
Providing a filesystem representation of the data is a fun experiment;
sysadmins can have a playful relation with the database thanks to an unconventional representation of the data.
.FOOTNOTE2
.FOOTNOTE1
New triggers could seek to improve performance by any means necessary including the gazillion ways which already exist.
.FOOTNOTE2
For example, a new kind of triggers could provide a way to accelerate searches based on an attribute, replicate data, send notifications to an external tool, etc.
The following sections will present indexing triggers with improved performance.
.
.
.SECTION DODB, slow? Nope. Let's talk about caches
The filesystem representation (of data and indexes) is convenient for the administrator, but input-output operations on a filesystem are slow.
Storing the data on a storage device is required to protect it from crashes and application restarts.
But data can be kept in memory for faster processing of requests.

The DODB library has an API close to a hash table.
Having a data cache is as simple as keeping a hash table in memory besides providing a filesystem storage, the retrieval becomes incredibly fast\*[*].
.FOOTNOTE1
Several hundred times faster, see the experiment section.
.FOOTNOTE2
Same thing for cached indexes.
Indexes can easily be cached, thanks to simple hash tables.

.B "Cached database" .
A cached database has the same API as the other DODB databases and keeps a copy of the entire database in memory for fast retrieval.
.QP
.SOURCE Ruby ps=9 vs=10
# Create a cached database
database = DODB::Storage::Cached(Car).new "path/to/db-cars"
.SOURCE
All the operations of the
.CLASS Storage::Uncached
class are available for
.CLASS Storage::Cached .
.QE
.
.B "Cached indexes" .
Since indexes do not require nearly as much memory as caching the entire database, they are cached by default.
.
.
.
.SECTION Common database
Storing the entire data-set in memory is an effective way to make the requests fast, as does
the
.I "cached database"
presented in the previous section.
Not all data-sets are compatible with this approach, for obvious reasons.
Thus, a tradeoff could be found to enable fast retrieval of data without requiring much memory.
Caching only a part of the data-set could already enable a massive speed-up even in memory-constrained environments.
The most effective strategy could differ from an application to another\*[*].
.FOOTNOTE1
Providing a generic algorithm that should work for all possible constraints is an hazardous endeavor.
.FOOTNOTE2
However, caching only the most recently requested values is a simple policy which may be efficient in many cases.
This strategy is implemented in the
.CLASS DODB::Storage::Common
database and this section will explain how it works.

Common database implements a
.I "Least Recently Used"
(LRU) cache eviction policy.
The strategy is simple, keeping only the most
.I "recently used"
values in memory.
Added, requested or modified values are considered
.I recent .
In case a new value is added to the cache and that the number of entries exceeds the cache size, the least recently used value is evicted, along with its related data from the cache.

.B "The Least Recently Used algorithm" .
Each time a value is added in the database, its key is put as the first element of a
.I set
structure (each value is unique).
This set is ordered, the first element being the most recently used.
Adding a value that is already present in the set is considered as
.I "using the value" ,
thus it is moved at the start of the set.
In case the number of entries exceeds what is allowed, the least recently used value is therefore the last element of the set.

.B "Implementation details" .
The LRU strategy is both simple and can be easily implemented efficiently with a double-linked list and a hash table.
The implementation is time-efficient;
the time spent adding a value is almost constant, it doesn't change much with the number of entries.
This efficiency is a memory tradeoff.
All the entries are added to a
.B "double-linked list"
(to keep track of the order of the added keys)
.UL and
to a
.B "hash table"
to perform efficient searches of the keys in the list.
Thus, all the nodes are added twice, once in the list, once in the hash.
This way, adding, removing and searching for an entry in the list is fast,
no matter the size of the list.

Moreover,
.I "common database"
enables to adjust the number of stored entries.
.
.QP
.SOURCE Ruby ps=9 vs=10
# Create a database with a data cache limited to 100.000 entries
database = DODB::Storage::Common(Car).new "path/to/db-cars", 100000
.SOURCE
The
.CLASS Storage::Common
class has the same API as the other database classes.
.QE
.
.SECTION RAM-only database for short-lived data
Databases are built around the objective to actually
.I store
data.
But sometimes the data has only the same lifetime as the application.
Stop the application and the data becomes irrelevant.
This happens in several occasions, for example when the application keeps track of the connected users.
This case is not covered by traditional databases; this is out-of-scope, short-lived data only is handled
.UL within
the application.

Since DODB is a library and not a separate application, providing a way to handle this usage of the database can be relevant.
Having the same API to handle both long and short-lived data can be useful.
Moreover, the previously mentioned triggers (basic indexes, partitions and tags) would also work the same way for these short-lived data.
Of course, in this case, the filesystem representation may be completely irrelevant.
Therefore, the
.I RAM-only
database and the
.I RAM-only
triggers were created.

Let's recap the advantages of the RAM-only database.
The DODB API is the same for short-lived (read: temporary) and long-lived data.
This includes the same triggers too, so a filesystem representation of the current state of the application is possible.
.I RAM-only
also means incredible performances since DODB only is a
.I very
small layer over a hash table.
.
.
.SS RAM-only database
To create a RAM-only database is as simple as the other options since the API is identical to other DODB databases.
Thus, changing from one to another is painless.
.QP
.SOURCE Ruby ps=9 vs=10
# RAM-only database creation
database = DODB::Storage::RAMOnly(Car).new "path/to/db-cars"
.SOURCE
Yes, the path still is required which may be seen as a quirk but the rationale\*[*] is sound.
.QE
.FOOTNOTE1
A path is still required despite the database being only in memory because indexes can still be instanciated for the database, and those indexes will require this directory.
Also, I worked enough already, leave me alone.
.FOOTNOTE2
.
.
.SS RAM-only triggers
Triggers have their RAM-only version.
.QP
.SOURCE Ruby ps=9 vs=10
# RAM-only basic indexes.
cars_by_name = cars.new_RAM_index "name", &.name

# RAM-only partitions.
cars_by_color = cars.new_RAM_partition "color", &.color

# RAM-only tags.
cars_by_keywords = cars.new_RAM_tags "keywords", &.keywords
.SOURCE
The API of the
.I "RAM-only index objects"
is exactly the same as the others.
.QE
As for the database API itself, changing from a version of an index to another is painless.
This way, one can opt for a cached index and, after some time not using the filesystem representation, decide to change for its RAM-only version; a 4-character modification and nothing else.
.
.
.
.SECTION DODB and memory constraint
Some environments may have very peculiar constraints, where caching data would cause problems or would be inefficient anyway\*[*].
.FOOTNOTE1
Caching would be inefficient for databases where the distribution of requests is homogeneous between the different entries, for example.
If the requests are random, without a small portion of the data receiving most requests (such as a Pareto distribution), caching becomes mostly irrelevant.
.FOOTNOTE2
In these cases, the
.CLASS "DODB::Storage::Uncached"
can be used\*[*].
.FOOTNOTE1
However, the
.CLASS DODB::Storage::Common
should be considered instead for most applications, even if the configured number of entries is low due to low RAM.
.FOOTNOTE2
.
.QP
.SOURCE Ruby ps=9 vs=10
# Uncached database creation
database = DODB::Storage::Uncached(Car).new "path/to/db-cars"
.SOURCE
.QE

.B "Uncached triggers" .
Caching an index shouldn't require a large amount of memory since the only stored data is an integer (the
.I key
of the data) and a string, which is also arguably true for partitions and tags (setting aside exceptions).
For that reason, these triggers are cached by default.
But for highly memory-constrained environments, the cache can be removed.
.QP
.SOURCE Ruby ps=9 vs=10
# Uncached basic indexes.
cars_by_name = cars.new_uncached_index "name", &.name

# Uncached partitions.
cars_by_color = cars.new_uncached_partition "color", &.color

# Uncached tags.
cars_by_keywords = cars.new_uncached_tags "keywords", &.keywords
.SOURCE
The API of the
.I "uncached index objects"
is exactly the same as the others.
.QE
.
.
.SECTION Experimental scenario
.LP
The following experiment shows the performance of DODB based on querying durations.
Data can be searched via
.I indexes ,
as for SQL databases.
As a reminder, three possible indexes exist in DODB:
(a) basic indexes for 1-to-1 relations, the document's attribute is related to a value and each value of this attribute is unique,
(b) partitions for 1-to-n relations, the attribute has a value and this value can be shared by other documents,
(c) tags for n-to-n relations, enabling the attribute to have multiple values whose are shared by other documents.

The scenario is simple: adding values to a database with indexes (basic, partitions and tags) then query 100 times a value based on the different indexes.
Loop and repeat.

Five instances of DODB are tested:
.STARTBULLET
.BULLET \fIuncached database\f[] shows the achievable performance with a strong memory constraint (nothing can be kept in-memory);
.BULLET \fIuncached database but cached index\f[] shows the improvement to expect with an index cache alone;
.BULLET \fIcommon database\f[] shows the most basic use of DODB, with a limited cache (100k entries)\*[*];
.BULLET \fIcached database\f[] represents a database will all the entries in cache (no eviction mechanism);
.BULLET \fIRAM only\f[], the database doesn't have a representation on disk (no data is written on it).
.ENDBULLET
.FOOTNOTE1
The data cache can be fine-tuned with the "common database", enabling the use of DODB in environments with low memory.
.FOOTNOTE2

The computer on which this test is performed\*[*] is a AMD PRO A10-8770E R7 (4 cores), 2.8 GHz.When mentioned, the
.I disk
is actually a
.I "temporary filesystem (tmpfs)"
to enable maximum efficiency.
.FOOTNOTE1
A very simple $50 PC, buyed online.
Nothing fancy.
.FOOTNOTE2

The library is written in Crystal and so is the benchmark (\f[CW]spec/benchmark-cars.cr\f[]).
Nonetheless, despite a few technicalities, the objective of this document is to provide an insight on the approach used in DODB more than this particular implementation.

The manipulated data type can be found in \f[CW]spec/db-cars.cr\f[].
.SOURCE Ruby ps=9 vs=9p vs=10
class Car
	property name     : String        # 1-1 relation
	property color    : String        # 1-n relation
	property keywords : Array(String) # n-n relation
end
.SOURCE
.
.
.SS Basic indexes (1 to 1 relations)
.LP
An index enables to match a single value based on a small string.
In our example, each \f[CW]car\f[] has an unique \fIname\f[] which is used as an index.

The following graph represents the result of 100 queries of a car based on its name.
The experiment starts with a database containing 1,000 cars and goes up to 250,000 cars.

.ps -2
.so graphs/query_index.grap
.ps \n[PS]
.QP
This figure shows the request durations to retrieve data based on a basic index with a database containing up to 250k entries, both with linear and logarithmic scales.
.QE

Since there is only one value to retrieve, the request is quick and time is almost constant.
When the value and the index are kept in memory (see \f[CW]RAM only\f[], \f[CW]Cached db\f[] and \f[CW]Common db\f[]), the retrieval is almost instantaneous\*[*].
.FOOTNOTE1
About 110 to 120 ns for RAM-only and cached database.
This is slightly more (about 200 ns) for Common database since there is a few more steps due to the inner structure to maintain.
.FOOTNOTE2
In case the value is on the disk, deserialization takes about 15 µs (see \f[CW]Uncached db\f[]).
The request is a little longer when the index isn't cached (see \f[CW]Uncached db and index\f[]); in this case DODB walks the filesystem to find the right symlink to follow, thus slowing the process even more, up to 20%.

The logarithmic scale version of this figure shows that \fIRAM-only\f[] and \fIcached\f[] databases have exactly the same performance.
The \fIcommon\f[] database spends 80 ns for its LRU caching eviction policy\*[*], making this database about 67% slower than the previous ones to retrieve a value.
.FOOTNOTE1
The LRU policy in DODB is implemented with a double-linked list and a hash table.
When a value is retrieved or modified, its key is put at the start of a list so the list order represents values from the most to the least recently used.
Also, a hash table is maintained to quickly jump to the right list entry.
Both these operations take time.
.FOOTNOTE2
Uncached databases are far away from these results, as shown by the logarithmically scaled figure.
The data cache makes requests at least 170 times faster.

.B "As a side note" :
the results depend on the data size.
The bigger the data, the slower the serialization (and deserialization).
In this example, the database entries are almost empty; they have very few attributes and not much content (a few dozen characters max).
Thus, performance of non-cached databases will be even more severely impacted with real-world data.
Alternative encodings, such as CBOR,
.[
CBOR
.]
should be considered for databases with non-trivial documents.
.
.
.SS Partitions (1 to n relations)
The previous example shown the retrieval of a single value from the database.
The following will show what happens when thousands of entries are retrieved.

A partition index enables to match a list of entries based on an attribute.
In the experiment, a database of cars is created along with a partition on their color.
Performance is analyzed based the partition size (the number of red cars) and the duration to retrieve all the entries.

.ps -2
.so graphs/query_partition.grap
.ps \n[PS]
.QP
This figure shows the retrieval of cars based on a partition (their color), with both a linear and a logarithmic scale.
The number of cars retrieved scales from 2000 to 10000.
.QE
In this example, both the linear and the logarithmic scales are represented to better grasp the difference between all databases.
The linear scale shows the linearity of the request time for uncached databases.
Respectively, the logarithmically scaled figure does the same for cached databases,
which are flattened in the linear scale since they are between one to five hundred times quicker than the uncached ones.

The duration of a retrieval grows linearly with the number of matched entries.
On both figures, a dashed line is drawn representing a linear growth based on the quickest retrieval observed from basic indexes for each database.
This dashed line and the observed results differ slightly; observed results grow more than what has been calculated.
This difference comes, at least partially, from the additional process of putting all the results in an array (which may also include some memory management) and the accumulated random delays for the retrieval of each value (due to the cache policy processing, to the processus scheduling on the machine, etc.).

Further analysis of the results may be interesting but this is far beyond the scope of this document.
The objective of this experiment is to give an idea of the performance that can be expected from DODB.
Basically, uncached databases are between 70 to 600 times slower than cached ones.
The eviction policy in
.I common
database slows down the retrievals, which makes it 70% to 6 times slower than
.I cached
and
.I RAM-only
databases, and the more data there is to retrieve, the worst it gets.
However, retrieving thousands and thousands of entries in a single request may not be a typical usage of databases, anyway.
.
.
.SS Tags (n to n relations)
A tag index enables to match a list of entries based on an attribute with potentially multiple values (such as an array).
In the experiment, a database of cars is created along with a tag index on a list of
.I keywords
associated with the cars, such as "elegant", "fast" and so on.
Performance is analyzed based the number of entries retrieved (the number of elegant cars) and the request duration.
.
.ps -2
.so graphs/query_tag.grap
.ps \n[PS]
.QP
This figure shows the retrieval of cars based on a tag (all cars tagged as
.I elegant ),
with both a linear and a logarithmic scale.
The number of cars retrieved scales from 1000 to 5000.
.QE
.
.
Tag and partition indexes request durations are similar because both are fundamentally the same thing:
.ENUM both tag and partition indexes enable to retrieve a list of entries;
.ENUM the keys of the database entries come from listing the content of a directory (uncached indexes) or are directly available from a hash (cached indexes);
.ENUM data is retrieved irrespective of the index, it is either read from the storage device or retrieved from a data cache, which depends on the type of database.
.ENDENUM

Contrary to partitions, the tag index enables multiple values for the same attribute.
For example, a car can be both
.I elegant
and
.I fast .
The DODB API enables to retrieve data matching several tags\*[*].
.FOOTNOTE1
Current DODB implementation performs the request of both tags then produces a list intersection.
The duration of the request would then be the addition of both tag requests and the duration of the intersection operation (plus an additional time span for memory management, depending on the list sizes).
.FOOTNOTE2
.
.
.SS Summary of the different databases and their use
.LP
.B "RAM-only database"
is the fastest database but dedicated to short-lived data (data is not saved on disk).

.B "Cached database"
enables the same performance on data retrieval than RAM-only while actually storing data on a storage device.
This database is to be considered to achieve maximum speed for data-sets fitting in memory.

.B "Common database"
enables to lower the memory requirements as much as desired.
The eviction policy implies some operations leading to poorer performances, however still widely acceptable in most cases.

.B "Uncached database"
is essentially a debug mode and is not expected to run in most real-life scenarii.
The purpose is to produce a control sample (involving only raw IO operations) to compare it to other (more realistic) implementations.

Cached indexes should be considered for most applications, and even more their RAM-only version in case the filesystem representation isn't necessary.
.
.\" .ps -2
.\" .TS
.\" allbox tab(:);
.\" c | lw(3.6i) | cew(1.4i).
.\" DODB instance:Comment and database usage:T{
.\" compared to RAM-only
.\" T}
.\" RAM only:T{
.\" Worst memory footprint, best performance.
.\" T}:-
.\" Cached db and index:T{
.\" Performance for retrieving a value is the same as RAM only while
.\" enabling the admin to manually search for data on-disk.
.\" T}:about the same perfs
.\" Common db, cached index:T{
.\" Performance is still excellent while requiring a
.\" .UL configurable
.\" amount of RAM.
.\" Should be used by default.
.\" T}:T{
.\" 67% slower (about 200 ns) which still is great
.\" T}
.\" Uncached db, cached index:Very slow. Common database should be considered instead.:170 to 180x slower
.\" Uncached db and index:T{
.\" Best memory footprint, worst performance.
.\" T}:200 to 210x slower
.\" .TE
.\" .ps \n[PS]
.
.SS Conclusion on performance
As expected, retrieving a single value is fast and the size of the database doesn't matter much.
Each deserialization and, more importantly, each disk access is a pain point.
Caching the value enables a massive performance gain, data can be retrieved several hundred times quicker.
The more entries requested, the slower it gets; but more importantly, the poorer performances it gets
.UL "per entry" .

The eviction policy implies poorer performances since it requires a few list and hash table operations, even if the current implementation (based on the LRU algorithm) is fairly simple and efficient.

To put things into perspective, requesting several thousand entries in DODB based on an index (partition or tag) is as slow as getting
.B "a single entry"
with a traditional SQL database\*[*].
.FOOTNOTE1
With Postgres, the request duration of a single value varies from 0.1 to 2 ms on my machine without a search, just the first available entry.
.FOOTNOTE2
.
.
.
.SECTION Limits of DODB
DODB provides basic database operations such as storing, retrieving, modifying and removing data.
However, DODB doesn't fully handle ACID properties\*[*]: atomicity, consistency, isolation and durability.
This section presents the limits of DODB, whether the current implementation or the approach, and presents some suggestions to fill the gaps.
.FOOTNOTE1
Traditional SQL databases handle ACID properties and may have created some "expectations" towards databases from a general public standpoint.
.FOOTNOTE2

.SS "Current state of DODB regarding ACID properties"
.STARTBULLET
.BULLET
.B Atomicity
isn't handled in DODB.
Multiple instructions cannot be chained and taken into account at the same time.
However, this a limitation of the current implementation, not the approach.

This issue could be resolved for example by introducing a
.I "global lock"
to prevent any modification while processing multiple instructions in one go.
This lock would eventually be shared accross multiple DODB instances, in case the chained instructions would modify several databases\*[*].
.FOOTNOTE1
Modifying several DODB instances at the same time is a rather common occurrence.
.FOOTNOTE2

.BULLET
.B Consistency
isn't handled in DODB.
No mechanism prevents invalid values to be added.
Once again, this only is a shortcoming of the implementation.

For the moment,
.I triggers
are used to create indexes, but the idea could fit another purpose: to create predicates to avoid invalid actions.
These new triggers could record user-defined procedures to perform database verifications and revert (or prevent) changes in case of a unvalid insertion, modification or deletion.

.BULLET
.B Isolation
is partially taken into account with a locking mechanism preventing race conditions when modifying a value.
This may be seen as simplistic but
.SHINE "good enough"
for most applications.

.BULLET
.B Durability
is taken into account.
Data is written on disk each time it changes.
Again, this is basic but
.SHINE "good enough"
for most applications.
.ENDBULLET
A future improvement could be to write a checksum for every written data, to easily remove corrupt data from a database.

.SS "Discussion on ACID properties"
First and foremost, both atomicity and isolation properties are inherently related to parallelism, whether through concurrent threads or applications.
Traditional SQL databases require both atomicity and isolation properties because they cannot afford not to have parallelism.
Since DODB is a library (and not a separate application) and is kept simple (no intermediary language to interpret, no complicated algorithm), it doesn't suffer from any communication latency or long processing delaying requests.
As the experimentation shown, retrieving a value in DODB only takes about 20 µs, 200 ns with a data cache.
Therefore, DODB could theoretically serve millions of requests per second from a single thread\*[*].
.FOOTNOTE1
FYI, the service
.I netlib.re
uses DODB and since the database is fast enough, parallelism isn't required despite enabling several thousand requests per second.
.FOOTNOTE2
Considering this swiftness, parallelism may seem as optional.

The consistency property is a safety net for potentially defective software.
Always nice to have, but not entirely necessary, especially for document-oriented databases.
Contrary to a traditional SQL database which often requires several modifications to different tables in one go to be kept consistent, a document-oriented database stores an entire document which already is internally consistent.
When several documents are involved (which happens from time to time), consistency needs to be checked, but this may not require much code\*[*].
Not checking systematically for consistency upon any database modification is a tradeoff between simplicity of the code plus speed, and security.
.FOOTNOTE1
As a side note, consistency is already taken care of within the application anyway.
Database verifications are just the last bastion against inserting junk data.
.FOOTNOTE2

Moreover, the consistency property in traditional SQL databases is often used for simple tasks but quickly becomes difficult to deal with.
Some companies and organizations (such as Doctors Without Borders for instance) cannot afford to implement all the preventive measures in their DBMSs due to the sheer complexity of it.
Instead, these organizations adopt curative measures that they may call "data-fix".
Thus, having some verifications in the database is not a silver bullet, it is complementary to other measures.

DODB may provide some form of atomicity and consistency at some point, but nothing fancy nor too advanced.
The whole point of the DODB project is to keep the code simple, hackable, enjoyable even.
Not handling these properties isn't a limitation of the DODB approach but a choice for this project\*[*].
.FOOTNOTE1
Which also results from a lack of time.
.FOOTNOTE2

.SS "Beyond ACID properties \[en] modern databases' features"
Most current databases (traditional relational databases, some key-value databases and so on) provide additional features.
These features may include for example high availability toolsets (replication, clustering, etc.), some forms of modularity (several storage backends, specific interfaces with other tools, etc.), interactive command lines or shells, user and authorization management, administration of databases, and so on.

Because DODB is a library and doesn't support an intermediary language for generic requests,
.TBD
.
.SS "The state of file systems, their limitations and useful features for DODB instances"
A
.dq filesystem
is the code responsible for the way data and meta-data will be written on a storage device, which basically is some sort of low-level CRUD operations.
This code links the user interface (files and directories) with the device drivers, which finally write bytes on a hard drive for example.
The next paragraphs will give an idea of how filesystems work, the implied limitations regarding DODB\*[*] as it uses filesystems in an overtly naive way and the filesystems' features DODB instances could use for better data management.
.FOOTNOTE1
Explaining the way filesystem work and their design is out of the scope of this document, so this part will be kept short for readability reasons.
.FOOTNOTE2

Beside filesystems designed for specific constraints, such as writing data on a compact disk\*[*] or providing a network filesystem, most
.dq generic
filesystems share a (loosely) common set of objectives.
.FOOTNOTE1
A compact disk has specific constraints since the device will then only provide read-only access to the data, obviating the need for most of the complexity revolving around fragmentation, inode management and so on.
All storage devices have their own particularities, but regular hard drives and solid-state drives are the important ones for this discussion since filesystems have mostly been designed for them.
.FOOTNOTE2
These features could be summarized in a few points.

.STARTBULLET
.KS
.BULLET
.B "CRUD operations" .
Above all, as already established, filesystems enable CRUD operations on a storage device through the concepts of directories and files; this is how users have been directly interacting with their computer to store data for decades.
.KE

.BULLET
.KS
.B "Reliability and safety" .
.TBD
Since computers do not run in a vacuum, many problems can occur during operation including the loss of the energy supply.
Filesystems try to mitigate damage by keeping a journal of operations (journalized filesystems).
Advanced filesystems may also detect file corruption with automated checksums.
.KE

.BULLET
.KS
.B "Security" .
.KE
File access should be limited in a number of cases.
For example, several applications with networking features might run on a computer.
If one of these applications is successfully attacked, the attacker shouldn't be able to access other services data or user data.
Same thing for shared computers, one user shouldn't be able to see other users' data.
Therefore, the most widespread form of security comes from filesystem permissions, enabling a user (or a group of users) to access (or to be denied from accessing) specific data (files and directories).
Those permissions include the right to read, to modify or to execute a file, to list or to remove files from a directory, to create or to remove directories and a few other permissions.
Extended permissions and attributes exist but are out-of-scope.

Beside permissions, encryption also brings some kind of security.
In this case, the point is to prevent attackers from accessing protected data despite retrieving files.
Some advanced filesystems can encrypt files individually, others provide the encryption of a whole partition, both methods having their pros and cons.

.BULLET
.KS
.B "Performance and capacity" .
Many file systems were developed over the years to circumvent contemporary limitations on file or partition sizes, the number of possible files, the limitation on path name lengths, etc.
While storage devices mostly impose physical limitations, a filesystem may be wasting resources because of a simplistic or inadequate design.
.KE

Depending on the scenario, the filesystem might become wasteful or slow.
Some filesystems cannot handle a huge number of small files (from hundreds of millions to billions) without wasting a lot of space, such as ext4 which doesn't have block suballocation: once a file and has at least one byte in it, it takes a 4kB block size and 4kB-1 bytes are wasted.
So, worst case scenario, data rate is
.FRAC 1 4000
(huge waste) meaning that a 1GB of data would require an entire 4TB hard drive\*[*] (without even taking the inodes' size into account).
.FOOTNOTE1
Ext4 can integrate up to 60 bytes of data into an extended inode.
.FOOTNOTE2

.BULLET
.KS
.B "Miscealeneous and advanced features" .
A few other features need to be mentionned, such as block suballocation, file content included in the inode, etc.
Some filesystems added more than a decade ago then under-explored features such as snapshots, compression and transactions.
.KE
.ENDBULLET

In conclusion, no current filesystem has been designed to be used the way DODB use them.
However, having a few millions entries is fine on most filesystems.
.
.
.SECTION Alternatives
Other approaches have been used to store data over the years, including but not limited to SQL and key-value stores.
This section briefly presents some of them and their difference from DODB.

.STARTBULLET
.BULLET
.B "Traditional DBMS" .
This category includes all SQL database management systems with a dedicated application handling databases and the operations upon them.
Most known DBMSs are MSSQL, Postgres, Oracle and MariaDB.

These applications are inherently complex for different reasons.
.STARTBULLET
.BULLET They require a description of the data;
.BULLET They require queries written in a dedicated language (SQL);
.BULLET They implement many sophisticated algorithms for performance reasons;
.BULLET Data is written in unconventional formats that may change (slightly or completely) at any moment;
.BULLET Their codebase is gargantuan\*[*], between one and several million lines of code.
.ENDBULLET
.FOOTNOTE1
MadiaDB has 2.3 million lines of code (MLOC) and 1.7 MLOC for Postgres.
Other mentioned DBMSs aren't open-source software, but it seems reasonable to consider their number of LOC to be in the same ballpark.
.br
Just to put things into perspective, DODB is less than 1300 lines of code.
Sure, DODB doesn't have the same features, but are they worth multiplying the codebase by 1700?
.FOOTNOTE2

.BULLET
.B "Key-value stores."
.B "Memcached"

.B "Redis"

.B "MongoDB"

.B "duckdb"
.ENDBULLET

.TBD
.
.SECTION Future work
This section presents all the features I want to see in a future version of the DODB library.
.
.SS Pagination via the indexes: offset and limit
Right now, browsing the entire database by requesting a limited list at a time is possible, thanks to some functions accepting an
.I offset
and a
.I size .
However, this is not possible with the indexes, thus when querying for example a partition the API provides the entire list of matching values.
This is not acceptable for databases with large partitions and tags: memory will be over-used and requests will be slow.
.
.SS DODB and security
Right now, security isn't managed in DODB, at all.
DODB isn't vulnerable to SQL injections, but an internet-facing application may encounter a few other problems including, but not limited to, buffer overflows and code injection.
However, a few security mechanisms exist to prevent data leak or data modification from an outsider and the DODB library may implement some of them in the future.

.B "Preventing data leak with in-app memory management" .
Since DODB is a library, any attack on the application using it can lead to a data leak.
For the moment, any part of the application can access data stored in memory.
Operating systems provide system calls to protect parts of the allocated memory.
For instance,
.FUNCTION_CALL mlock
prevents a region of memory from being put in the swap because it could lead to a data leak.
The
.FUNCTION_CALL madvise
syscall can prevent parts of the application's memory to be put in a code-dump\*[*], which is a debug file created when an application crashes containing (part of) the process's memory when it crashed.
.FOOTNOTE1
.FUNCTION_CALL madvice
has the interesting option
.I MADV_DONTDUMP
to prevent a data leak through a core-dump, but it is linux-specific.
.FOOTNOTE2

In a running process,
.FUNCTION_CALL mprotect
prevents the application itself to access part of its own memory;
the idea is to read (or write) memory only once you ask for it via a syscall.
Thus, you cannot access data from anywhere (by mistake or after an attack).

These mechanisms could be used internally by DODB to prevent a data leak since memory is handled by the library.
However, the Crystal language doesn't provide a way to manage memory manually and this may be a problem for mlock and mprotect.
Depending on the plateform (the operating system), these syscalls may require the memory to be aligned with the memory pages.
Thus, the implementation won't be easy in the current state of affairs\*[*] (read: with the current Crystal's standard library).
.FOOTNOTE1
But again, the Crystal language is an implementation detail.
The working principle of DODB can be translated in basically any other language, including some enabling access to low-level memory operations.
.FOOTNOTE2

.B "Other methods to prevent data leak" .
By default, a running application can call any syscall and open sockets or any file owned by the user.
Thus, in case a user runs an internet-facing application that is hacked, the content of any of their file can be leaked, including sensible files (in the
.DIRECTORY ~/.ssh/
directory for example).

To prevent applications from performing unwanted operations (calling syscalls or accessing some files unrelated to the proper functioning of the application), operating systems implemented specific ways to enforce security.
All of these methods have their own pros and cons.
This obviously goes beyond the scope of this document, but let's mention a few widespread mechanisms\*[*].
.FOOTNOTE1
Actual statistics on the use of the different security mechanisms is rather hard to obtain.
Both presented mechanisms are supported by a wealthy company or by the operating system itself.
.FOOTNOTE2
.STARTBULLET
.BULLET "\fBAppArmor\f[] (linux):"
Linux has many mechanisms to handle software security, one of the most known is AppArmor (supported since 2009 by Canonical).
AppArmor can limit the use of syscalls and the access to files and directories.
A configuration file is used to describe what an application (defined by the path to its executable) can or cannot do; the source code of the application is left untouched.

.BULLET "\fBpledge and unveil\f[] (openbsd):"
the OpenBSD operating system includes two complementary syscalls to handle permissions: pledge (syscalls) and unveil (directories and files).
The design of these functions is simple: applications often have an
.I initialization
phase during which the connections are made or files are opened (including configuration files),
then comes the
.I running
phase, during which the application needs less priviledges.
Therefore, an application can access whatever it needs for its initialization phase, which is less prone to attacks, then restricts its own rights over syscalls and files before accepting connections from the internet.
For example, a web server can read its configuration file to learn the path to the files to serve, then prevents itself from accessing any other file (including its own configuration file) before serving the files.

In-app mechanisms such as these greatly simplifies the configuration.
Security parameters related to the filesystem don't require to be sync with the configuration of the application.
Also, any syscall that is irrelevent for the
.I running
phase can be disallowed without fuss, which makes pledge+unveil inherently safer than AppArmor and the like.

.ENDBULLET

Common to all the above mechanisms (AppArmor and pledge+unveil), by default, without taking a deep dive into software architecture, none of these prevents a user from accessing the entirety of the database.
A malicious user who successfully took control of the application can now open files (at least in the DODB directory) and read the application's memory (including cached data).
.
.
.SECTION Real-world usage: netlibre
DODB instances have been deployed in a real-world setting by the netlibre service.
This section presents this service and its use of DODB, showing how this method of handling data can be used in conventional online services.

.B Presentation .
Netlibre
.[
netlibre
.]
is a service providing free domain names and a website to manage DNS zones.
Domains can be shared and transfered between users, so organizations do not have to rely on a single person.
Users are helped as much as possible through dedicated user interfaces for complex resource records, many automatic zone verifications, documentation and so on.
Resource records can be automatically updated via
.I tokens ,
enabling users to host services on the internet despite having a dynamic IP address; those tokens are used with a trivial command:
.SOURCE Ruby ps=9 vs=10
wget "https://www.netlib.re/token-update/<token>"
.SOURCE
Thus, netlibre is a real-life service providing domains to more than 7500 users to this day.

.B "The technical parts" .
The service is split into three components: the user interface (the website), an authentication daemon\*[*] (\fIauthd\f[]) and a daemon handling all the server operations related to the actual service (\fIdnsmanagerd\f[]).
.FOOTNOTE1
The authentication daemon is separated from the service-specific code in order to factor authentication for future applications.
Thus, besides factoring code, this enables users to register only once for multiple services.
.FOOTNOTE2
Several "databases" are maintained:
.ENUM \fIusers\f[] (authd), authentication and user preference data.
.ENUM \fIdomains\f[] (dnsmanagerd), ownership management.
.ENUM \fIzones\f[] (dnsmanagerd), actual zone content management.
.ENUM \fItokens\f[] (dnsmanagerd), enabling automatic updates of records, each token being related to a single record of a single domain.
.ENUM \fIhub\f[] (dnsmanagerd), for data synchronization between users in case of collaborative work (several owners working on a domain at the same time).
.ENDENUM

Besides
.I "hub"
which is an instance of
.I RAM-only
database since it contains inherently volatile data, the others are instances of the
.I Common
database.
This way, application start-up phase is practically instanteneous while caching data for busy entries.

Performance-wise, netlibre handles between 2 to 3k req/s with a single core, without any optimization.
Code is written in one of the most naive way possible\*[*] and still is performing fine\*[*].
.FOOTNOTE1
Keep in mind that netlibre (through
.B libipc )
uses poll(2), a very old syscall to handle its event loop (from the 80's!); not newest and way faster event facilities such as epoll(2) and the like.
Also, JSON is used for data storage and queries between the service and its users instead of a more efficient format such as CBOR.
Furthermore, each entry in the logs is written by opening a file, appending the string to the end of the file then closing the file.
Finally, any modification to the content of a zone triggers the on-disk write of its representation in the Bind9 zone file format.
.br
It's almost as the application intentionally avoids any possible optimization.
.FOOTNOTE2
.FOOTNOTE1
Especially given that the number of actual requests is expected to be around 10 requests per second on busy days.
.FOOTNOTE2
Indexes with filesystem representation enables quick debugging sessions and to perform a few basic tasks (such as listing all the domains of a user) which, in practice, is great to have at our fingertips with simple unix tools.

.SECTION Conclusion
The
.I common
database should be an acceptable choice for most applications.

.STARTBULLET
.BULLET it is possible to write other triggers to replace the way index, partition and tags are working, and store their data differently, possibly in a flat file for example.
.BULLET talk about netlib.re
.BULLET triggers are available and can be adapted to do anything, indexes are a simple use of triggers
.BULLET common db is great for most applications
.BULLET indexes can provide a view of the current state of the database
.BULLET ramdb is a great tool, same API than the rest so you can attach indexes to it
.ENDBULLET

DODB won't power the next
.I "AI thing" ,
it will never handle databases with a petabyte of data nor revolutionize cryptocurrency.
However, DODB may be a better fit than traditional databases for your next blog, your gaming forum, the software forge of your dreams and maybe your future MMORPG.

.TBD

.APPENDIX LRU vs Efficient LRU
.ps -2
.so graphs/addition_lru.grap
.ps \n[PS]

.APPENDIX Common database performance
The
.I Common
database enables to configure the number of allowed entries in the cache.
The following figures show the performance of the common database depending on the cache size.

.ps -2
.so graphs/lru_query_index.grap
.ps \n[PS]
.QP
This figure shows the request durations to retrieve data based on a basic index with a database containing up to 250k entries.
.QE
.EQ
delim $$
.EN
This figure shows a value being requested and since there is only a single value being requested in the test, it is immediately put in the cache and is never evicted.
For that reason, the result is stable amongst all
.I common
database instances:
.vp 2p
$+-$ 170 ns.
.EQ
delim off
.EN
.ps -2
.so graphs/lru_query_partition.grap
.ps \n[PS]
.QP
This figure shows the request durations to retrieve data based on a partition containing up to 10k entries.
.QE
As we see in the figure, the duration for data retrieval grows almost linearly for databases with a sufficient cache size (starting with 10k entries).
When the cache size is not sufficient, the requests are hundred times slower, which explain why the database with a cache size of one thousand entries isn't even represented in the graph, and why the 5k database has great results up to 5k partitions.
.ps -2
.so graphs/lru_query_tag.grap
.ps \n[PS]
.QP
This figure shows the request durations to retrieve data based on a tag containing up to 5k entries.
.QE
As for partitions, the response time depends on the number of entries to retrieve and the duration increases linearly with the number of elements.
.
.
.APPENDIX Recap of the DODB API
This section provides a quick shorthand manual for the most important parts of the DODB API.
For an exhaustive API documentation, please generate the development documentation for the library.
The command
.COMMAND "make doc"
generates the documentation, then the
.COMMAND "make serve-doc"
command enables to browse the full documentation with a web browser\*[*].
.FOOTNOTE1
The
.COMMAND "make serve-doc"
requires darkhttpd
.[
darkhttpd
.]
but this can be adapted to any other web server.
.FOOTNOTE2
.
.SS Database creation
.QP
.SOURCE Ruby ps=9 vs=10
# Uncached, cached, common and RAM-only database creation.
database = DODB::Storage::Uncached(Car).new "path/to/db"
database = DODB::Storage::Cached(Car).new "path/to/db"
database = DODB::Storage::Common(Car).new "path/to/db", 50000 # nb cache entries
database = DODB::Storage::RAMOnly(Car).new "path/to/db"
.SOURCE
.QE
.
.SS Browsing the database
.QP
.SOURCE Ruby ps=9 vs=10
# List all the values in the database
database.each do |value|
	# ...
end
.SOURCE
.QE

.QP
.SOURCE Ruby ps=9 vs=10
# List all the values in the database with their key
database.each_with_key do |value, key|
	# ...
end
.SOURCE
.QE
.
.SS Database search, update and deletion with the key (integer associated to the value)
.KS
.QP
.SOURCE Ruby ps=9 vs=10
value = database[key]  # May throw a MissingEntry exception
value = database[key]? # Returns nil if the value doesn't exist
database[key] = value
database.delete key
.SOURCE
Side note for the
.I []
function: in case the value isn't in the database, the function throws an exception named
.CLASS DODB::MissingEntry .
To avoid this exception and get a
.I nil
value instead, use the
.I []?
function.
.QE
.KE
.
.
.SS Trigger creation
.QP
.SOURCE Ruby ps=9 vs=10
# Uncached, cached and RAM-only basic indexes.
cars_by_name = cars.new_uncached_index "name", &.name
cars_by_name = cars.new_index          "name", &.name
cars_by_name = cars.new_RAM_index      "name", &.name

# Uncached, cached and RAM-only partitions.
cars_by_color = cars.new_uncached_partition "color", &.color
cars_by_color = cars.new_partition          "color", &.color
cars_by_color = cars.new_RAM_partition      "color", &.color

# Uncached, cached and RAM-only tags.
cars_by_keywords = cars.new_uncached_tags "keywords", &.keywords
cars_by_keywords = cars.new_tags          "keywords", &.keywords
cars_by_keywords = cars.new_RAM_tags      "keywords", &.keywords
.SOURCE
.QE
.
.
.SS Database retrieval, update and deletion with an index
.
.QP
.SOURCE Ruby ps=9 vs=10
# Get a value from a 1-1 index.
car = cars_by_name.get  "Corvet" # May throw a MissingEntry exception
car = cars_by_name.get? "Corvet" # Returns nil if the value doesn't exist
.SOURCE
.QE
.
.QP
.SOURCE Ruby ps=9 vs=10
# Get a value from a partition (1-n relations) or a tag (n-n relations) index.
red_cars = cars_by_color.get "red"      # empty array if no such cars exist
fast_cars = cars_by_keywords.get "fast" # empty array if no such cars exist

# Several tags can be selected at the same time, to narrow the search.
cars_both_fast_and_expensive = cars_by_keywords.get ["fast", "expensive"]
.SOURCE
.QE
.
The basic 1-1
.I "index object"
can update a value by selecting an unique entry in the database.
.QP
.SOURCE Ruby ps=9 vs=10
car = cars_by_name.update updated_car            # If the `name` hasn't changed.
car = cars_by_name.update "Corvet", updated_car  # If the `name` has changed.

car = cars_by_name.update_or_create updated_car  # Updates or creates the value.
car = cars_by_name.update_or_create "Corvet", updated_car # Same.
.SOURCE
.QE
For deletion, database entries can be selected based on any index.
Partitions and tags can take a block of code to narrow the selection.
.QP
.SOURCE Ruby ps=9 vs=10
cars_by_name.delete "Corvet"    # Deletes the car named "Corvet".
cars_by_color.delete "red"      # Deletes all red cars.

# Deletes cars that are both slow and expensive.
cars_by_keywords.delete ["slow", "expensive"]

# Deletes all cars that are both blue and slow.
cars_by_color.delete "blue", do |car|
	car.keywords.includes? "slow"
end

# Same.
cars_by_keywords.delete "slow", do |car|
	car.color == "blue"
end
.SOURCE
.QE
.
.
.SSS Tags: search on multiple keys
The Tag index enables to search for a value based on multiple keys.
For example, searching for all cars that are both fast and elegant can be written this way:
.QP
.SOURCE Ruby ps=9 vs=10
fast_elegant_cars = cars_by_keywords.get ["fast", "elegant"]
.SOURCE
Used with a list of keys, the
.FUNCTION_CALL get
function returns an empty list in case the search failed.
.br
The implementation was designed to be simple (7 lines of code), not efficient.
However, with data and index caches, the search is expected to meet about everyone's requirements, speed-wise, given that the tags are small enough (a few thousand entries).
.QE
.
.