diff options
Diffstat (limited to 'nixos/modules/services/databases/foundationdb.xml')
-rw-r--r-- | nixos/modules/services/databases/foundationdb.xml | 621 |
1 files changed, 355 insertions, 266 deletions
diff --git a/nixos/modules/services/databases/foundationdb.xml b/nixos/modules/services/databases/foundationdb.xml index 7883680d46c..bf4b644c9b8 100644 --- a/nixos/modules/services/databases/foundationdb.xml +++ b/nixos/modules/services/databases/foundationdb.xml @@ -3,42 +3,50 @@ xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0" xml:id="module-services-foundationdb"> - -<title>FoundationDB</title> - -<para><emphasis>Source:</emphasis> <filename>modules/services/databases/foundationdb.nix</filename></para> - -<para><emphasis>Upstream documentation:</emphasis> <link xlink:href="https://apple.github.io/foundationdb/"/></para> - -<para><emphasis>Maintainer:</emphasis> Austin Seipp</para> - -<para><emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x</para> - -<para>FoundationDB (or "FDB") is an open source, distributed, transactional -key-value store.</para> - -<section xml:id="module-services-foundationdb-configuring"><title>Configuring and basic setup</title> - -<para>To enable FoundationDB, add the following to your -<filename>configuration.nix</filename>: - + <title>FoundationDB</title> + <para> + <emphasis>Source:</emphasis> + <filename>modules/services/databases/foundationdb.nix</filename> + </para> + <para> + <emphasis>Upstream documentation:</emphasis> + <link xlink:href="https://apple.github.io/foundationdb/"/> + </para> + <para> + <emphasis>Maintainer:</emphasis> Austin Seipp + </para> + <para> + <emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x + </para> + <para> + FoundationDB (or "FDB") is an open source, distributed, transactional + key-value store. + </para> + <section xml:id="module-services-foundationdb-configuring"> + <title>Configuring and basic setup</title> + + <para> + To enable FoundationDB, add the following to your + <filename>configuration.nix</filename>: <programlisting> services.foundationdb.enable = true; services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x </programlisting> -</para> - -<para>The <option>services.foundationdb.package</option> option is required, -and must always be specified. Due to the fact FoundationDB network protocols and -on-disk storage formats may change between (major) versions, and upgrades must -be explicitly handled by the user, you must always manually specify this -yourself so that the NixOS module will use the proper version. Note that minor, -bugfix releases are always compatible.</para> - -<para>After running <command>nixos-rebuild</command>, you can verify whether -FoundationDB is running by executing <command>fdbcli</command> (which is added -to <option>environment.systemPackages</option>): - + </para> + + <para> + The <option>services.foundationdb.package</option> option is required, and + must always be specified. Due to the fact FoundationDB network protocols and + on-disk storage formats may change between (major) versions, and upgrades + must be explicitly handled by the user, you must always manually specify + this yourself so that the NixOS module will use the proper version. Note + that minor, bugfix releases are always compatible. + </para> + + <para> + After running <command>nixos-rebuild</command>, you can verify whether + FoundationDB is running by executing <command>fdbcli</command> (which is + added to <option>environment.systemPackages</option>): <programlisting> $ sudo -u foundationdb fdbcli Using cluster file `/etc/foundationdb/fdb.cluster'. @@ -66,14 +74,14 @@ Cluster: fdb> </programlisting> -</para> - -<para>You can also write programs using the available client libraries. -For example, the following Python program can be run in order to grab the -cluster status, as a quick example. (This example uses -<command>nix-shell</command> shebang support to automatically supply the -necessary Python modules). - + </para> + + <para> + You can also write programs using the available client libraries. For + example, the following Python program can be run in order to grab the + cluster status, as a quick example. (This example uses + <command>nix-shell</command> shebang support to automatically supply the + necessary Python modules). <programlisting> a@link> cat fdb-status.py #! /usr/bin/env nix-shell @@ -100,255 +108,336 @@ a@link> ./fdb-status.py FoundationDB available: True a@link> </programlisting> -</para> - -<para>FoundationDB is run under the <command>foundationdb</command> user and -group by default, but this may be changed in the NixOS configuration. The -systemd unit <command>foundationdb.service</command> controls the -<command>fdbmonitor</command> process.</para> - -<para>By default, the NixOS module for FoundationDB creates a single -SSD-storage based database for development and basic usage. This storage engine -is designed for SSDs and will perform poorly on HDDs; however it can handle far -more data than the alternative "memory" engine and is a better default choice -for most deployments. (Note that you can change the storage backend on-the-fly -for a given FoundationDB cluster using <command>fdbcli</command>.)</para> - -<para>Furthermore, only 1 server process and 1 backup agent are started in the -default configuration. See below for more on scaling to increase this.</para> - -<para>FoundationDB stores all data for all server processes under -<filename>/var/lib/foundationdb</filename>. You can override this using -<option>services.foundationdb.dataDir</option>, e.g. - + </para> + + <para> + FoundationDB is run under the <command>foundationdb</command> user and group + by default, but this may be changed in the NixOS configuration. The systemd + unit <command>foundationdb.service</command> controls the + <command>fdbmonitor</command> process. + </para> + + <para> + By default, the NixOS module for FoundationDB creates a single SSD-storage + based database for development and basic usage. This storage engine is + designed for SSDs and will perform poorly on HDDs; however it can handle far + more data than the alternative "memory" engine and is a better default + choice for most deployments. (Note that you can change the storage backend + on-the-fly for a given FoundationDB cluster using + <command>fdbcli</command>.) + </para> + + <para> + Furthermore, only 1 server process and 1 backup agent are started in the + default configuration. See below for more on scaling to increase this. + </para> + + <para> + FoundationDB stores all data for all server processes under + <filename>/var/lib/foundationdb</filename>. You can override this using + <option>services.foundationdb.dataDir</option>, e.g. <programlisting> services.foundationdb.dataDir = "/data/fdb"; </programlisting> - -</para> - -<para>Similarly, logs are stored under -<filename>/var/log/foundationdb</filename> by default, and there is a -corresponding <option>services.foundationdb.logDir</option> as well.</para> - -</section> - -<section xml:id="module-services-foundationdb-scaling"><title>Scaling processes and backup agents</title> - -<para>Scaling the number of server processes is quite easy; simply specify -<option>services.foundationdb.serverProcesses</option> to be the number of -FoundationDB worker processes that should be started on the machine.</para> - -<para>FoundationDB worker processes typically require 4GB of RAM per-process at -minimum for good performance, so this option is set to 1 by default since the -maximum amount of RAM is unknown. You're advised to abide by this restriction, -so pick a number of processes so that each has 4GB or more.</para> - -<para>A similar option exists in order to scale backup agent processes, -<option>services.foundationdb.backupProcesses</option>. Backup agents are not -as performance/RAM sensitive, so feel free to experiment with the number of -available backup processes.</para> - -</section> - -<section xml:id="module-services-foundationdb-clustering"><title>Clustering</title> - -<para>FoundationDB on NixOS works similarly to other Linux systems, so this -section will be brief. Please refer to the full FoundationDB documentation for -more on clustering.</para> - -<para>FoundationDB organizes clusters using a set of -<emphasis>coordinators</emphasis>, which are just specially-designated worker -processes. By default, every installation of FoundationDB on NixOS will start -as its own individual cluster, with a single coordinator: the first worker -process on <command>localhost</command>.</para> - -<para>Coordinators are specified globally using the -<command>/etc/foundationdb/fdb.cluster</command> file, which all servers and -client applications will use to find and join coordinators. Note that this file -<emphasis>can not</emphasis> be managed by NixOS so easily: FoundationDB is -designed so that it will rewrite the file at runtime for all clients and nodes -when cluster coordinators change, with clients transparently handling this -without intervention. It is fundamentally a mutable file, and you should not -try to manage it in any way in NixOS.</para> - -<para>When dealing with a cluster, there are two main things you want to -do:</para> - -<itemizedlist> - <listitem><para>Add a node to the cluster for storage/compute.</para></listitem> - <listitem><para>Promote an ordinary worker to a coordinator.</para></listitem> -</itemizedlist> - -<para>A node must already be a member of the cluster in order to properly be -promoted to a coordinator, so you must always add it first if you wish to -promote it.</para> - -<para>To add a machine to a FoundationDB cluster:</para> - -<itemizedlist> - <listitem><para>Choose one of the servers to start as the initial coordinator. - </para></listitem> - <listitem><para>Copy the <command>/etc/foundationdb/fdb.cluster</command> file - from this server to all the other servers. Restart FoundationDB on all of - these other servers, so they join the cluster.</para></listitem> - <listitem><para>All of these servers are now connected and working together - in the cluster, under the chosen coordinator.</para></listitem> -</itemizedlist> - -<para>At this point, you can add as many nodes as you want by just repeating -the above steps. By default there will still be a single coordinator: you can -use <command>fdbcli</command> to change this and add new coordinators.</para> - -<para>As a convenience, FoundationDB can automatically assign coordinators -based on the redundancy mode you wish to achieve for the cluster. Once all the -nodes have been joined, simply set the replication policy, and then issue the -<command>coordinators auto</command> command</para> - -<para>For example, assuming we have 3 nodes available, we can enable double -redundancy mode, then auto-select coordinators. For double redundancy, 3 -coordinators is ideal: therefore FoundationDB will make -<emphasis>every</emphasis> node a coordinator automatically:</para> + </para> + + <para> + Similarly, logs are stored under <filename>/var/log/foundationdb</filename> + by default, and there is a corresponding + <option>services.foundationdb.logDir</option> as well. + </para> + </section> + <section xml:id="module-services-foundationdb-scaling"> + <title>Scaling processes and backup agents</title> + + <para> + Scaling the number of server processes is quite easy; simply specify + <option>services.foundationdb.serverProcesses</option> to be the number of + FoundationDB worker processes that should be started on the machine. + </para> + + <para> + FoundationDB worker processes typically require 4GB of RAM per-process at + minimum for good performance, so this option is set to 1 by default since + the maximum amount of RAM is unknown. You're advised to abide by this + restriction, so pick a number of processes so that each has 4GB or more. + </para> + + <para> + A similar option exists in order to scale backup agent processes, + <option>services.foundationdb.backupProcesses</option>. Backup agents are + not as performance/RAM sensitive, so feel free to experiment with the number + of available backup processes. + </para> + </section> + <section xml:id="module-services-foundationdb-clustering"> + <title>Clustering</title> + + <para> + FoundationDB on NixOS works similarly to other Linux systems, so this + section will be brief. Please refer to the full FoundationDB documentation + for more on clustering. + </para> + + <para> + FoundationDB organizes clusters using a set of + <emphasis>coordinators</emphasis>, which are just specially-designated + worker processes. By default, every installation of FoundationDB on NixOS + will start as its own individual cluster, with a single coordinator: the + first worker process on <command>localhost</command>. + </para> + + <para> + Coordinators are specified globally using the + <command>/etc/foundationdb/fdb.cluster</command> file, which all servers and + client applications will use to find and join coordinators. Note that this + file <emphasis>can not</emphasis> be managed by NixOS so easily: + FoundationDB is designed so that it will rewrite the file at runtime for all + clients and nodes when cluster coordinators change, with clients + transparently handling this without intervention. It is fundamentally a + mutable file, and you should not try to manage it in any way in NixOS. + </para> + + <para> + When dealing with a cluster, there are two main things you want to do: + </para> + + <itemizedlist> + <listitem> + <para> + Add a node to the cluster for storage/compute. + </para> + </listitem> + <listitem> + <para> + Promote an ordinary worker to a coordinator. + </para> + </listitem> + </itemizedlist> + + <para> + A node must already be a member of the cluster in order to properly be + promoted to a coordinator, so you must always add it first if you wish to + promote it. + </para> + + <para> + To add a machine to a FoundationDB cluster: + </para> + + <itemizedlist> + <listitem> + <para> + Choose one of the servers to start as the initial coordinator. + </para> + </listitem> + <listitem> + <para> + Copy the <command>/etc/foundationdb/fdb.cluster</command> file from this + server to all the other servers. Restart FoundationDB on all of these + other servers, so they join the cluster. + </para> + </listitem> + <listitem> + <para> + All of these servers are now connected and working together in the + cluster, under the chosen coordinator. + </para> + </listitem> + </itemizedlist> + + <para> + At this point, you can add as many nodes as you want by just repeating the + above steps. By default there will still be a single coordinator: you can + use <command>fdbcli</command> to change this and add new coordinators. + </para> + + <para> + As a convenience, FoundationDB can automatically assign coordinators based + on the redundancy mode you wish to achieve for the cluster. Once all the + nodes have been joined, simply set the replication policy, and then issue + the <command>coordinators auto</command> command + </para> + + <para> + For example, assuming we have 3 nodes available, we can enable double + redundancy mode, then auto-select coordinators. For double redundancy, 3 + coordinators is ideal: therefore FoundationDB will make + <emphasis>every</emphasis> node a coordinator automatically: + </para> <programlisting> fdbcli> configure double ssd fdbcli> coordinators auto </programlisting> -<para>This will transparently update all the servers within seconds, and -appropriately rewrite the <command>fdb.cluster</command> file, as well as -informing all client processes to do the same.</para> - -</section> - -<section xml:id="module-services-foundationdb-connectivity"><title>Client connectivity</title> - -<para>By default, all clients must use the current -<command>fdb.cluster</command> file to access a given FoundationDB cluster. -This file is located by default in -<command>/etc/foundationdb/fdb.cluster</command> on all machines with the -FoundationDB service enabled, so you may copy the active one from your cluster -to a new node in order to connect, if it is not part of the cluster.</para> - -</section> - -<section xml:id="module-services-foundationdb-authorization"><title>Client authorization and TLS</title> - -<para>By default, any user who can connect to a FoundationDB process with the -correct cluster configuration can access anything. FoundationDB uses a -pluggable design to transport security, and out of the box it supports a -LibreSSL-based plugin for TLS support. This plugin not only does in-flight -encryption, but also performs client authorization based on the given -endpoint's certificate chain. For example, a FoundationDB server may be -configured to only accept client connections over TLS, where the client TLS -certificate is from organization <emphasis>Acme Co</emphasis> in the -<emphasis>Research and Development</emphasis> unit.</para> - -<para>Configuring TLS with FoundationDB is done using the -<option>services.foundationdb.tls</option> options in order to control the peer -verification string, as well as the certificate and its private key.</para> - -<para>Note that the certificate and its private key must be accessible to the -FoundationDB user account that the server runs under. These files are also NOT -managed by NixOS, as putting them into the store may reveal private -information.</para> - -<para>After you have a key and certificate file in place, it is not enough to -simply set the NixOS module options -- you must also configure the -<command>fdb.cluster</command> file to specify that a given set of coordinators -use TLS. This is as simple as adding the suffix <command>:tls</command> to your -cluster coordinator configuration, after the port number. For example, assuming -you have a coordinator on localhost with the default configuration, simply -specifying:</para> + <para> + This will transparently update all the servers within seconds, and + appropriately rewrite the <command>fdb.cluster</command> file, as well as + informing all client processes to do the same. + </para> + </section> + <section xml:id="module-services-foundationdb-connectivity"> + <title>Client connectivity</title> + + <para> + By default, all clients must use the current <command>fdb.cluster</command> + file to access a given FoundationDB cluster. This file is located by default + in <command>/etc/foundationdb/fdb.cluster</command> on all machines with the + FoundationDB service enabled, so you may copy the active one from your + cluster to a new node in order to connect, if it is not part of the cluster. + </para> + </section> + <section xml:id="module-services-foundationdb-authorization"> + <title>Client authorization and TLS</title> + + <para> + By default, any user who can connect to a FoundationDB process with the + correct cluster configuration can access anything. FoundationDB uses a + pluggable design to transport security, and out of the box it supports a + LibreSSL-based plugin for TLS support. This plugin not only does in-flight + encryption, but also performs client authorization based on the given + endpoint's certificate chain. For example, a FoundationDB server may be + configured to only accept client connections over TLS, where the client TLS + certificate is from organization <emphasis>Acme Co</emphasis> in the + <emphasis>Research and Development</emphasis> unit. + </para> + + <para> + Configuring TLS with FoundationDB is done using the + <option>services.foundationdb.tls</option> options in order to control the + peer verification string, as well as the certificate and its private key. + </para> + + <para> + Note that the certificate and its private key must be accessible to the + FoundationDB user account that the server runs under. These files are also + NOT managed by NixOS, as putting them into the store may reveal private + information. + </para> + + <para> + After you have a key and certificate file in place, it is not enough to + simply set the NixOS module options -- you must also configure the + <command>fdb.cluster</command> file to specify that a given set of + coordinators use TLS. This is as simple as adding the suffix + <command>:tls</command> to your cluster coordinator configuration, after the + port number. For example, assuming you have a coordinator on localhost with + the default configuration, simply specifying: + </para> <programlisting> XXXXXX:XXXXXX@127.0.0.1:4500:tls </programlisting> -<para>will configure all clients and server processes to use TLS from now -on.</para> - -</section> - -<section xml:id="module-services-foundationdb-disaster-recovery"><title>Backups and Disaster Recovery</title> - -<para>The usual rules for doing FoundationDB backups apply on NixOS as written -in the FoundationDB manual. However, one important difference is the security -profile for NixOS: by default, the <command>foundationdb</command> systemd unit -uses <emphasis>Linux namespaces</emphasis> to restrict write access to the -system, except for the log directory, data directory, and the -<command>/etc/foundationdb/</command> directory. This is enforced by default -and cannot be disabled.</para> - -<para>However, a side effect of this is that the <command>fdbbackup</command> -command doesn't work properly for local filesystem backups: FoundationDB uses a -server process alongside the database processes to perform backups and copy the -backups to the filesystem. As a result, this process is put under the -restricted namespaces above: the backup process can only write to a limited -number of paths.</para> - -<para>In order to allow flexible backup locations on local disks, the -FoundationDB NixOS module supports a -<option>services.foundationdb.extraReadWritePaths</option> option. This option -takes a list of paths, and adds them to the systemd unit, allowing the -processes inside the service to write (and read) the specified -directories.</para> - -<para>For example, to create backups in <command>/opt/fdb-backups</command>, -first set up the paths in the module options:</para> + <para> + will configure all clients and server processes to use TLS from now on. + </para> + </section> + <section xml:id="module-services-foundationdb-disaster-recovery"> + <title>Backups and Disaster Recovery</title> + + <para> + The usual rules for doing FoundationDB backups apply on NixOS as written in + the FoundationDB manual. However, one important difference is the security + profile for NixOS: by default, the <command>foundationdb</command> systemd + unit uses <emphasis>Linux namespaces</emphasis> to restrict write access to + the system, except for the log directory, data directory, and the + <command>/etc/foundationdb/</command> directory. This is enforced by default + and cannot be disabled. + </para> + + <para> + However, a side effect of this is that the <command>fdbbackup</command> + command doesn't work properly for local filesystem backups: FoundationDB + uses a server process alongside the database processes to perform backups + and copy the backups to the filesystem. As a result, this process is put + under the restricted namespaces above: the backup process can only write to + a limited number of paths. + </para> + + <para> + In order to allow flexible backup locations on local disks, the FoundationDB + NixOS module supports a + <option>services.foundationdb.extraReadWritePaths</option> option. This + option takes a list of paths, and adds them to the systemd unit, allowing + the processes inside the service to write (and read) the specified + directories. + </para> + + <para> + For example, to create backups in <command>/opt/fdb-backups</command>, first + set up the paths in the module options: + </para> <programlisting> services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ]; </programlisting> -<para>Restart the FoundationDB service, and it will now be able to write to -this directory (even if it does not yet exist.) Note: this path -<emphasis>must</emphasis> exist before restarting the unit. Otherwise, systemd -will not include it in the private FoundationDB namespace (and it will not add -it dynamically at runtime).</para> + <para> + Restart the FoundationDB service, and it will now be able to write to this + directory (even if it does not yet exist.) Note: this path + <emphasis>must</emphasis> exist before restarting the unit. Otherwise, + systemd will not include it in the private FoundationDB namespace (and it + will not add it dynamically at runtime). + </para> -<para>You can now perform a backup:</para> + <para> + You can now perform a backup: + </para> <programlisting> $ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups $ sudo -u foundationdb fdbbackup status -t default </programlisting> - -</section> - -<section xml:id="module-services-foundationdb-limitations"><title>Known limitations</title> - -<para>The FoundationDB setup for NixOS should currently be considered beta. -FoundationDB is not new software, but the NixOS compilation and integration has -only undergone fairly basic testing of all the available functionality.</para> - -<itemizedlist> - <listitem><para>There is no way to specify individual parameters for - individual <command>fdbserver</command> processes. Currently, all server - processes inherit all the global <command>fdbmonitor</command> settings. - </para></listitem> - <listitem><para>Ruby bindings are not currently installed.</para></listitem> - <listitem><para>Go bindings are not currently installed.</para></listitem> -</itemizedlist> - -</section> - -<section xml:id="module-services-foundationdb-options"><title>Options</title> - -<para>NixOS's FoundationDB module allows you to configure all of the most -relevant configuration options for <command>fdbmonitor</command>, matching it -quite closely. A complete list of options for the FoundationDB module may be -found <link linkend="opt-services.foundationdb.enable">here</link>. You should -also read the FoundationDB documentation as well.</para> - -</section> - -<section xml:id="module-services-foundationdb-full-docs"><title>Full documentation</title> - -<para>FoundationDB is a complex piece of software, and requires careful -administration to properly use. Full documentation for administration can be -found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.</para> - -</section> - + </section> + <section xml:id="module-services-foundationdb-limitations"> + <title>Known limitations</title> + + <para> + The FoundationDB setup for NixOS should currently be considered beta. + FoundationDB is not new software, but the NixOS compilation and integration + has only undergone fairly basic testing of all the available functionality. + </para> + + <itemizedlist> + <listitem> + <para> + There is no way to specify individual parameters for individual + <command>fdbserver</command> processes. Currently, all server processes + inherit all the global <command>fdbmonitor</command> settings. + </para> + </listitem> + <listitem> + <para> + Ruby bindings are not currently installed. + </para> + </listitem> + <listitem> + <para> + Go bindings are not currently installed. + </para> + </listitem> + </itemizedlist> + </section> + <section xml:id="module-services-foundationdb-options"> + <title>Options</title> + + <para> + NixOS's FoundationDB module allows you to configure all of the most relevant + configuration options for <command>fdbmonitor</command>, matching it quite + closely. A complete list of options for the FoundationDB module may be found + <link linkend="opt-services.foundationdb.enable">here</link>. You should + also read the FoundationDB documentation as well. + </para> + </section> + <section xml:id="module-services-foundationdb-full-docs"> + <title>Full documentation</title> + + <para> + FoundationDB is a complex piece of software, and requires careful + administration to properly use. Full documentation for administration can be + found here: <link xlink:href="https://apple.github.io/foundationdb/"/>. + </para> + </section> </chapter> |