Longer explanation of the experimental scenario.

toying-with-ramdb
Philippe PITTOLI 2024-05-13 02:24:59 +02:00
parent 7428ba757c
commit 71d8fa53d5
2 changed files with 696 additions and 8 deletions

View File

@ -1,7 +1,76 @@
.NH
Indexes
.LP
.so macros.roff
.TITLE Brief performance analysis of Document Oriented DataBase (DODB)
.AUTHOR Philippe P.
.ABSTRACT1
DODB is a database-as-library, enabling a very simple way to store applications' data: storing serialized
.I documents
(basically any data type) in plain files.
To speed-up searches, attributes of these documents can be used as indexes which leads to create a few symbolic links
.I symlinks ) (
on the disk.
.br
See the \f[CW]README\f[] for a longer explanation.
This document briefly presents an experiment to understand the performances we can get with this approach.
.br
.UL Status :
WIP
.ABSTRACT2
.SECTION Experimental scenario
.LP
The following experiment shows the performance of DODB based on quering durations.
Data can be searched via
.I indexes ,
as for SQL databases.
Three possible indexes exist in DODB:
(a) basic indexes, representing 1 to 1 relations, the document's attribute is related to a value and each value of this attribute is unique,
(b) partitions, representing 1 to n relations, the attribute has a value and this value can be shared by other documents,
(c) tags, representing n to n relations, enabling the attribute to have multiple values whose are shared by other documents.
The scenario is simple: adding values to a database with indexes (basic, partitions and tags) then query 100 times a value based on the different indexes.
Loop and repeat.
Four instances of DODB are tested:
.BULLET \fIuncached database\f[] shows the achievable performance with a strong memory constraint (nothing can be kept in-memory) ;
.BULLET \fIuncached data but cached index\f[] shows the improvement you can expect by having a cache on indexes ;
.BULLET \fIcached database\f[] shows the most basic use of DODB\*[*] ;
.BULLET \fIRAM only\f[], the database doesn't have a representation on disk (no data is written on it).
The \fIRAM only\f[] instance shows a possible way to use DODB: to keep a consistent API to store data, including in-memory data with a lifetime related to the application's.
.ENDBULLET
.FOOTNOTE1
Having a cached database will probably be the most widespread use of DODB.
When memory isn't scarce, there is no point not using it to achieve better performance.
.FOOTNOTE2
The computer on which this test is performed\*[*] is a AMD PRO A10-8770E R7 (4 cores), 2.8 GHz.When mentioned, the
.I disk
is actually a
.I "temporary file-system (tmpfs)"
to enable maximum efficiency.
.FOOTNOTE1
A very simple $50 PC, buyed online.
Nothing fancy.
.FOOTNOTE2
The library is written in Crystal and so is the benchmark (\f[CW]spec/benchmark-cars.cr\f[]).
Nonetheless, despite a few technicalities, the objective of this document is to provide an insight on the approach used in DODB more than this particular implementation.
The manipulated data type can be found in \f[CW]spec/db-cars.cr\f[].
.SOURCE Ruby ps=9 vs=9p
class Car
property name : String # 1-1 relation
property color : String # 1-n relation
property keywords : Array(String) # n-n relation
end
.SOURCE
.
.SECTION Basic indexes (1 to 1 relations)
.LP
An index enables to match a single value based on a small string.
Since there is only one value to retrieve, the request is quick and time is almost constant.
When the value and the index are kept in memory (see \f[CW]RAM only\f[] and \f[CW]Cached db\f[]), the retrieval is almost instantaneous (about 50 to 120 ns).
In case the value is on the disk, deserialization takes about 15 µs (see \f[CW]Uncached db, cached index\f[]).
The request is a little longer when the index isn't cached, in this case DODB walks the file-system to find the right symlink to follow, thus slowing the process even more, by up to 20%.
.G1
copy "legend.grap"
frame invis ht 3 wid 4 left solid bot solid
@ -71,9 +140,8 @@ copy "../data/index.d" thru X
.gcolor
X
.G2
.NH
Partitions
.bp
.SECTION Partitions (1 to n relations)
.LP
.G1
copy "legend.grap"
@ -142,8 +210,7 @@ copy "../data/partitions.d" thru X
X
.G2
.bp
.NH
Tags
.SECTION Tags (n to n relations)
.LP
.G1
copy "legend.grap"

621
graphs/macros.roff Normal file
View File

@ -0,0 +1,621 @@
.\" .RP = report document
.nr PO 0.5i \" page offset default 1i
.nr LL 7.5i \" line length default 6i
.nr FM 0.8i \" page foot margin default 1i
.nr DI 0
.nr FF 3 \" footnotes' type: numbered, with point, indented
.
.nr LIST_NUMBER 0 +1
.
.R1
no-label-in-reference
accumulate
.R2
.
. \" COLORS
.defcolor darkgreen rgb 0.1 0.5 0.2
.defcolor darkblue rgb 0.3 0.3 0.7
.defcolor darkred rgb 0.7 0.3 0.3
.defcolor black rgb 0 0 0
.defcolor color_box rgb 1 1 .6
.
. \" with semantic
.defcolor citation rgb 0.4 0.4 0.4
.defcolor citationbar rgb 0.3 0.3 0.7
.defcolor explanation rgb 0.7 0.4 0.4
.defcolor explanationbar rgb 0.8 0.3 0.3
.
.defcolor specialcolor_command rgb 0.7 0.3 0.3
.defcolor specialcolor_type rgb 0.6 0.3 0.5
.defcolor specialcolor_constructor rgb 0.1 0.5 0.2
.defcolor specialcolor_module rgb 0.1 0.5 0.2
.defcolor specialcolor_function rgb 0.4 0.4 0.7
.defcolor specialcolor_question rgb 0.0 0.0 0.7
.defcolor specialcolor_operator rgb 0.3 0.8 0.3
.defcolor specialcolor_shine rgb 0.3 0.3 0.7
.
. \" SIZES
.nr specialsize_command 10
.nr specialsize_type 8
.nr specialsize_constructor 8
.nr specialsize_module 8
.nr specialsize_function 8
.nr specialsize_operator 9
.nr specialsize_question 10 \" Current point size, no change.
.nr specialsize_shine 11
.
. \" FONTS
.ds specialfont_command CW
.ds specialfont_type CW
.ds specialfont_constructor CW
.ds specialfont_module CW
.ds specialfont_function CW
.ds specialfont_operator CW
.ds specialfont_question I
.ds specialfont_shine B
.
.
.de BELLOWEXPLANATION1
.sp 0.5
.ps 7 \" point size (~= font size)
.vs 8p \" vertical spacing between lines
..
.de BELLOWEXPLANATION2
.br
.ps 9
.vs 11p
..
.
.\" BULLET and ENUM => do not add space when no parameter are provided
.de BULLET \" Bullet points
.IP \(bu 2
.ie '\\$1'' \
.
.el \\$*
..
.de ENDBULLET
.in -2 \" indent
..
.
.de ENUM \" Numbered list
.nr LIST_NUMBER +1
.IP \\n[LIST_NUMBER] 2
.ie '\\$1'' \
.
.el \\$*
..
.de ENDENUM
.nr LIST_NUMBER 0
.in -2 \" indent
..
.
.de b1 \" Begin code box
.B1
.sp 0.2
.ft CW
..
.de b2 \" End code box
.sp 0.5
.B2
.ft
..
.
.de CITATION1
.KS \" start a keep
.ft I \" citation in italics
.mk C \" set a marker for line drawing
.in +1 \" indent a bit
.gcolor citation
..
.ig
The CITATION2 macro closes the quote then draws a line
from current line to the start of the quote.
..
.de CITATION2
.mk D \" set second marker to come back here
.ft \" back to previous font
.in -1 \" remove indent
.gcolor \" remove previous color
.gcolor citationbar
.\" r = move upward
.\" Z D t = drawing thickness
.\" L = draw the line
\r\
\Z'\D't 1p''\
\L'|\\nCu' \" draw line
.gcolor black \" remove previous color
.sp -2 \" get two lines back
\Z'\D't 1'' \" get the previous drawing thickness back
.KE \" end of the keep
..
.
.de NAMECITATION
.QP
.vs -\\n[legendps]p
.ps -\\n[legendps]
.in -1.2
.ll +1.2
\h'|-2'\(em\h'|-0.4'
\\$*
.br
.LP
..
.
.de EXPLANATION1
.KS \" start a keep
.ft B \" citation in italics
.mk C \" set a marker for line drawing
.in +1 \" indent a bit
.gcolor explanation
..
.de EXPLANATION2
.ft \" back to previous font
.in -1 \" remove indent
.gcolor \" remove previous color
.gcolor explanationbar
\r\L'|\\nCu' \" draw line (\r moves upward, \L draw the line, ...)
.gcolor \" remove previous color
.sp -1 \" get two lines back
.KE \" end of the keep
..
.
.de METAINFO1
.ft CW \" constant width font
.ps 8 \" small font
.vs 9p \" smaller vertical spacing between lines
..
.de METAINFO2
.sp 1
.vs \" come back to the previous vertical spacing
.ps \" come back to the previous point size
.ft \" come back to the previous font
.sp -1 \" return one line above
..
.
.
.de FRAC
.ie '\\$3'' \{\
\v'-.7m\s[\\n(.s*6u/10u]+.7m'\\$1\v'-.7m\s0+.7m'\
\(f/\s[\\n(.s*6u/10u]\\$2\s0
\}
.el \{\
\v'-.7m\s[\\n(.s*6u/10u]+.7m'\\$1\v'-.7m\s0+.7m'\
\(f/\s[\\n(.s*6u/10u]\\$2\s0\\$3
\}
..
.de FOOTNOTE_TO_COLUMN_WIDTH
.nr pg@fn-colw \\n[pg@colw] \" footnotes' column width
..
.de SINGLE_COLUMN
.1C
.FOOTNOTE_TO_COLUMN_WIDTH
..
.de TWO_COLUMNS
.2C
.FOOTNOTE_TO_COLUMN_WIDTH
..
.de HORIZONTALLINE
\l'15'
.FOOTNOTE_TO_COLUMN_WIDTH
..
.
. \" Fonts and colors.
.
.de SPECIAL_WORDS
.ie !'\\$3'' \\$3\c
.nr current_size \\n[.s] \" Current point size.
.gcolor specialcolor_\\*[semantictoken]
.
.if !((\\n[current_size] == \\n[specialsize_\\*[semantictoken]]) \
.ps \\n[specialsize_\\*[semantictoken]]
.
.ie '\\$2'' \{\
\f[\\*[specialfont_\\*[semantictoken]]]\\$1\f[]
. ps \\n[current_size]
. gcolor black \" FIXME: should be the previous color
\}
.el \{\
\f[\\*[specialfont_\\*[semantictoken]]]\\$1\f[]\c
. ps \\n[current_size]
. gcolor black \" FIXME: should be the previous color
\\$2
\}
..
.de SMALLFONT
.ps 8
.vs 9p
..
.de NORMALFONT
.vs
.ps
..
.de COMMAND1
.b1
..
.de COMMAND2
.b2
..
.de COMMANDNAME
.ds semantictoken command
.SPECIAL_WORDS \\$@
..
.de FUNCTION
.ds semantictoken function
.SPECIAL_WORDS \\$@
..
.de TYPE
.ds semantictoken type
.SPECIAL_WORDS \\$@
..
.de TYPECLASS
.I "\\$1" "\\$2"
..
.de OPERATOR
.ds semantictoken operator
.SPECIAL_WORDS \\$@
..
.de QUESTION
.ds semantictoken question
.SPECIAL_WORDS \\$@
\h'5p'
..
.de CONSTRUCTOR
.ds semantictoken constructor
.SPECIAL_WORDS \\$@
..
.de MODULE
.ds semantictoken module
.SPECIAL_WORDS \\$@
..
.de SHINE
.ds semantictoken shine
.SPECIAL_WORDS \\$@
..
.de MODULEX
.MODULE \\$1 ,
..
.de TBD
.ft B
To be defined or to finish.
.ft R
..
.de ARROW
.br
\(->\h'5p' \\$*
..
.af dy 00
.af mo 00
.ds CURRENT_DATE \\n(dy/\\n(mo/\\n[year]
.ds WEBSITE https://t.karchnu.fr/doc
.ds EMAIL karchnu@karchnu.fr
.de INFORMATIONS
Check out for newer versions:
.ft CW
.ps 8
\h'2p' \\$1
.ps
.ft
.br
And if you have questions:
.ft CW
\h'13p' \\$2
.ft
.\" .DE
.LP
Lastly compiled the
.SHINE \*[CURRENT_DATE]
(day/month/year, you know, like in any sane civilization).
..
.de INFORMATIONS_FR
.LP
Nouvelles versions :
.ft CW
.ps 8
\h'2p' \\$1
.ps
.ft
.br
Questions :
.ft CW
\h'36p' \\$2
.ft
.\" .DE
.LP
Compilé pour la dernière fois le
.SHINE \*[CURRENT_DATE]
..
.
.\" RENAMING REQUESTS
.
.de SECTION
.NH
.ps +3
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.ps
.PARAGRAPH_INDENTED
..
.de SUBSECTION
.NH 2
.ps +1
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.ps
.PARAGRAPH_INDENTED
..
.de SUBSUBSECTION
.NH 3
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.ps
.PARAGRAPH_INDENTED
..
.de SUBSUBSUBSECTION
.NH 4
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.PARAGRAPH_INDENTED
..
.de SECTION_NO_NUMBER
.SH
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.PARAGRAPH_INDENTED
..
.de SUBSECTION_NO_NUMBER
.SH 2
.fam H \" helvetica family
\\$*
.fam \" back to previous font family
.PARAGRAPH_INDENTED
..
.de PARAGRAPH_INDENTED
.PP
..
.de PARAGRAPH_UNINDENTED
.LP
..
.de NO_ABSTRACT
.AB no
..
.de ABSTRACT1
.AB
..
.de ABSTRACT2
.AE
..
.ds CH Page %
.de TITLE
.TL
\\$*
.ds LH \\$*
.de HD .XX
.sp -2.8
\l'7.5i'
.sp +1.5
.br
..XX
..
.de AUTHOR
. AU
. ie !'\\$1'' \\$*
..
.de FOOTNOTE1
. FS
..
.de FOOTNOTE2
. FE
..
.de VOCABULARY1
. KS
. BULLET
. UL "\\$*" :
..
.de VOCABULARY2
. KE
..
.
.
.de HIGHLIGHT
.
. nr @wd \w'\\$1'
. nr x1 0
. nr y1 (\\n[rst]u - \\n[rsb]u + .4m)
. nr x2 (\\n[@wd]u + .4m)
. nr y2 0
. nr x3 0
. nr y3 (\\n[rst]u - \\n[rsb]u + .4m)
. nr x4 (\\n[@wd]u + .4m)
. nr y4 0
.
\h'.2m'\
\h'-.2m'\v'(.2m - \\n[rsb]u)'\
\M[color_box]\
\D'P \\n[x1] -\\n[y1]u \\n[x2]u \\n[y2]u \\n[x3]u \\n[y3]u -\\n[x4]u \\n[y4]u '\
\h'.2m'\v'-(.2m - \\n[rsb]u)'\
\M[]\
\\$1\
\h'.2m'
..
.
.
.
.ds SPACE_SS_NUMBER_TITLE 0.5\" not a number register because of leading 0
.nr CURRENT_SECTION 0 +1
.nr CURRENT_APPENDIX 0
.af CURRENT_APPENDIX I
.nr CURRENT_SUBSECTION 0 +1
.nr CURRENT_SSSECTION 0 +1
.rm SECTION
.de SECTION
. nr CURRENT_SUBSECTION 0 \" reset current subsection numbering
. nr CURRENT_SSSECTION 0 \" reset current subsubsection numbering
. ie !(\\n[CURRENT_SECTION]=0) .sp +1
. br
. ie (\\n[APPENDIX_TIME]=0) \
. ds RH \\n+[CURRENT_SECTION].\h'\\*[SPACE_SS_NUMBER_TITLE]' \\$*
. el \{
. ds RH \\n[CURRENT_APPENDIX].\h'\\*[SPACE_SS_NUMBER_TITLE]' \\$*
. bp \}
. ps +2
. fam H \" helvetica family
. ft B
. ne 4 \" should be at least a few lines left at the bottom of the page
\\*[RH]
. ft
. fam \" back to previous font family
. ps -2
. PARAGRAPH_INDENTED
..
.nr APPENDIX_TIME 0
.de APPENDIX
. nr CURRENT_APPENDIX +1
. nr APPENDIX_TIME 1
. SECTION \\$*
..
.de SS
. nr CURRENT_SSSECTION 0
. ie (\\n[APPENDIX_TIME]=0) \
. SUBSECTION_NO_NUMBER \\n[CURRENT_SECTION].\
\\n+[CURRENT_SUBSECTION]\h'\\*[SPACE_SS_NUMBER_TITLE]' \\$*
.el \
. SUBSECTION_NO_NUMBER \\n[CURRENT_APPENDIX].\
\\n+[CURRENT_SUBSECTION]\h'\\*[SPACE_SS_NUMBER_TITLE]' \\$*
..
.de SSS
. br
. ps -2
. fam H \" helvetica family
. ft B
. ie (\\n[APPENDIX_TIME]=0) \
. SUBSECTION_NO_NUMBER \\n[CURRENT_SECTION].\
\\n[CURRENT_SUBSECTION].\\n+[CURRENT_SSSECTION]\h'\
\\*[SPACE_SS_NUMBER_TITLE]' \\$*
. el \
\\n[CURRENT_APPENDIX].\
\\n[CURRENT_SUBSECTION].\\n+[CURRENT_SSSECTION]\h'\
\\*[SPACE_SS_NUMBER_TITLE]' \\$*
. ft
. fam \" back to previous font family
. ps +2
. PARAGRAPH_INDENTED
..
.de INNERBULLET
. in +1
. br
\(bu
. in +1
. sp -1
\\$*
. in -2
..
.de EENUM \" Numbered list
. nr ENUM_INDENTATION 2
. ie !(\\n[LIST_NUMBER]=0) .in -\\n[ENUM_INDENTATION]
. br
\\n+[LIST_NUMBER].
. in +\\n[ENUM_INDENTATION]
. sp -1
\\$*
..
.de EENDENUM
. nr LIST_NUMBER 0
. in -\\n[ENUM_INDENTATION]
..
.nr legendps 2
.de LEGEND1
. QP
. vs -\\n[legendps]p
. ps -\\n[legendps]
. in -1.2
. ll +1.2
. br
..
.de LEGEND2
. br
. vs +\\n[legendps]p
. ps +\\n[legendps]
. br
. LP
..
.de IEME
\\$1\u\s-4\\$2\s+4\d
..
.de CENTERED
. ce
\\$*
. br
..
.de GIVEEXAMPLE1
. in +1
. ll -1
. KS \" start a keep
. \" .ft I \" citation in italics
. mk C \" set a marker for line drawing
. in +1 \" indent a bit
. gcolor citation
..
.de GIVEEXAMPLE2
. mk D \" set second marker to come back here
. \" .ft \" back to previous font
. in -1 \" remove indent
. gcolor black\" remove previous color
. gcolor citationbar
. \" r = move upward
. \" Z D t = drawing thickness
. \" L = draw the line
\r\
\Z'\D't 1p''\
\L'|\\nCu' \" draw line
. gcolor black \" remove previous color
. sp -2 \" get two lines back
\Z'\D't 0.5p'' \" get the previous drawing thickness back
. KE \" end of the keep
. ll +1
. in -1
..
.de ST
.nr ww \w'\\$1'
\Z@\v'-.25m'\l'\\n[ww]u'@\\$1
..
.de INCREMENT
.br
.in \\*[PINCREMENT]
.br
\h'-\\*[DECALAGE]'\\*[CHARACTER]\h'|0'\\$*
..
.de D
.ds DECALAGE 1.0
.ds PINCREMENT 2
.ds CHARACTER \(bu
.INCREMENT \\$*
..
.de DD
.ds DECALAGE 1.0
.ds PINCREMENT 3
.ds CHARACTER \(bu
.INCREMENT \\$*
..
.de AA
.ds DECALAGE 1.5
.ds PINCREMENT 3
.ds CHARACTER \(->
.INCREMENT \\$*
..
.de AAA
.ds DECALAGE 1.5
.ds PINCREMENT 4
.ds CHARACTER \(->
.INCREMENT \\$*
..
.de ED
.br
.in 0
..