nixos: add foundationdb module, documentation

Signed-off-by: Austin Seipp <aseipp@pobox.com>

1 files changed, 279 insertions, 0 deletions

diff --git a/nixos/modules/services/databases/foundationdb.xml b/nixos/modules/services/databases/foundationdb.xml
new file mode 100644
index 00000000000..d10a5cfe836
--- /dev/null
+++ b/nixos/modules/services/databases/foundationdb.xml
@@ -0,0 +1,279 @@
+<chapter xmlns="http://docbook.org/ns/docbook"
+         xmlns:xlink="http://www.w3.org/1999/xlink"
+         xmlns:xi="http://www.w3.org/2001/XInclude"
+         version="5.0"
+         xml:id="module-foundationdb">
+
+<title>FoundationDB</title>
+
+<para><emphasis>Source:</emphasis> <filename>modules/services/databases/foundationdb.nix</filename></para>
+
+<para><emphasis>Upstream documentation:</emphasis> <link xlink:href="https://apple.github.io/foundationdb/"/></para>
+
+<para><emphasis>Maintainer:</emphasis> Austin Seipp</para>
+
+<para><emphasis>Default version:</emphasis> 5.1.x</para>
+
+<para>FoundationDB (or "FDB") is a distributed, open source, high performance,
+transactional key-value store. It can store petabytes of data and deliver
+exceptional performance while maintaining consistency and ACID semantics over a
+large cluster.</para>
+
+<section><title>Configuring and basic setup</title>
+
+<para>To enable FoundationDB, add the following to your
+<filename>configuration.nix</filename>:
+
+<programlisting>
+services.foundationdb.enable = true;
+</programlisting>
+</para>
+
+<para>After running <command>nixos-rebuild</command>, you can verify whether
+FoundationDB is running by executing <command>fdbcli</command> (which is added
+to <option>environment.systemPackages</option>):
+
+<programlisting>
+$ sudo -u foundationdb fdbcli
+Using cluster file `/etc/foundationdb/fdb.cluster'.
+
+The database is available.
+
+Welcome to the fdbcli. For help, type `help'.
+fdb> status
+
+Using cluster file `/etc/foundationdb/fdb.cluster'.
+
+Configuration:
+  Redundancy mode        - single
+  Storage engine         - memory
+  Coordinators           - 1
+
+Cluster:
+  FoundationDB processes - 1
+  Machines               - 1
+  Memory availability    - 5.4 GB per process on machine with least available
+  Fault Tolerance        - 0 machines
+  Server time            - 04/20/18 15:21:14
+
+...
+
+fdb>
+</programlisting>
+</para>
+
+<para>FoundationDB is run under the <command>foundationdb</command> user and
+group by default, but this may be changed in the NixOS configuration. The
+systemd unit <command>foundationdb.service</command> controls the
+<command>fdbmonitor</command> process.</para>
+
+<para>By default, the NixOS module for FoundationDB creates a single
+SSD-storage based database for development and basic usage. This storage engine
+is designed for SSDs and will perform poorly on HDDs; however it can handle far
+more data than the alternative "memory" engine and is a better default choice
+for most deployments. (Note that you can change the storage backend on-the-fly
+for a given FoundationDB cluster using <command>fdbcli</command>.)</para>
+
+<para>Furthermore, only 1 server process and 1 backup agent are started in the
+default configuration. See below for more on scaling to increase this.</para>
+
+<para>FoundationDB stores all data for all server processes under
+<filename>/var/lib/foundationdb</filename>. You can override this using
+<option>services.foundationdb.dataDir</option>, e.g.
+
+<programlisting>
+services.foundationdb.dataDir = "/data/fdb";
+</programlisting>
+
+</para>
+
+<para>Similarly, logs are stored under
+<filename>/var/log/foundationdb</filename> by default, and there is a
+corresponding <option>services.foundationdb.logDir</option> as well.</para>
+
+</section>
+
+<section><title>Scaling processes and backup agents</title>
+
+<para>Scaling the number of server processes is quite easy; simply specify
+<option>services.foundationdb.serverProcesses</option> to be the number of
+FoundationDB worker processes that should be started on the machine.</para>
+
+<para>FoundationDB worker processes typically require 4GB of RAM per-process at
+minimum for good performance, so this option is set to 1 by default since the
+maximum aount of RAM is unknown. You're advised to abide by this restriction,
+so pick a number of processes so that each has 4GB or more.</para>
+
+<para>A similar option exists in order to scale backup agent processes,
+<option>services.foundationdb.backupProcesses</option>. Backup agents are not
+as performance/RAM sensitive, so feel free to experiment with the number of
+available backup processes.</para>
+
+</section>
+
+<section><title>Clustering</title>
+
+<para>FoundationDB on NixOS works similarly to other Linux systems, so this
+section will be brief. Please refer to the full FoundationDB documentation for
+more on clustering.</para>
+
+<para>FoundationDB organizes clusters using a set of
+<emphasis>coordinators</emphasis>, which are just specially-designated worker
+processes. By default, every installation of FoundationDB on NixOS will start
+as its own individual cluster, with a single coordinator: the first worker
+process on <command>localhost</command>.</para>
+
+<para>Coordinators are specified globally using the
+<command>/etc/foundationdb/fdb.cluster</command> file, which all servers and
+client applications will use to find and join coordinators. Note that this file
+<emphasis>can not</emphasis> be managed by NixOS so easily: FoundationDB is
+designed so that it will rewrite the file at runtime for all clients and nodes
+when cluster coordinators change, with clients transparently handling this
+without intervention.</para>
+
+<para>When dealing with a cluster, there are two main things you want to
+do:</para>
+
+<itemizedlist>
+  <listitem><para>Add a node to the cluster for storage/compute.</para></listitem>
+  <listitem><para>Promote an ordinary worker to a coordinator.</para></listitem>
+</itemizedlist>
+
+<para>A node must already be a member of the cluster in order to properly be
+promoted to a coordinator, so you must always add it first if you wish to
+promote it.</para>
+
+<para>To add a machine to a FoundationDB cluster:</para>
+
+<itemizedlist>
+  <listitem><para>Choose one of the servers to start as the initial coordinator.
+      </para></listitem>
+  <listitem><para>Copy the <command>/etc/foundationdb/fdb.cluster</command> file
+      from this server to all the other servers. Restart FoundationDB on all of
+      these other servers, so they join the cluster.</para></listitem>
+  <listitem><para>All of these servers are now connected and working together
+      in the cluster, under the chosen coordinator.</para></listitem>
+</itemizedlist>
+
+<para>At this point, you can add as many nodes as you want by just repeating
+the above steps. By default there will still be a single coordinator: you can
+use <command>fdbcli</command> to change this and add new coordinators.</para>
+
+<para>As a convenience, FoundationDB can automatically assign coordinators
+based on the redundancy mode you wish to achieve for the cluster. Once all the
+nodes have been joined, simply set the replication policy, and then issue the
+<command>coordinators auto</command> command</para>
+
+<para>For example, assuming we have 3 nodes available, we can enable double
+redundancy mode, then auto-select coordinators. For double redundancy, 3
+coordinators is ideal: therefore FoundationDB will make
+<emphasis>every</emphasis> node a coordinator automatically:</para>
+
+<programlisting>
+fdbcli> configure double ssd
+fdbcli> coordinators auto
+</programlisting>
+
+<para>This will transparently update all the servers within seconds, and
+appropriately rewrite the <command>fdb.cluster</command> file, as well as
+informing all client processes to do the same.</para>
+
+</section>
+
+<section><title>Client connectivity</title>
+
+<para>By default, all clients must use the current
+<command>fdb.cluster</command> file to access a given FoundationDB cluster.
+This file is located by default in
+<command>/etc/foundationdb/fdb.cluster</command> on all machines with the
+FoundationDB service enabled, so you may copy the active one from your cluster
+to a new node in order to connect, if it is not part of the cluster.</para>
+
+</section>
+
+<section><title>Backups and Disaster Recovery</title>
+
+<para>The usual rules for doing FoundationDB backups apply on NixOS as written
+in the FoundationDB manual. However, one important difference is the security
+profile for NixOS: by default, the <command>foundationdb</command> systemd unit
+uses <emphasis>Linux namespaces</emphasis> to restrict write access to the
+system, except for the log directory, data directory, and the
+<command>/etc/foundationdb/</command> directory. This is enforced by default
+and cannot be disabled.</para>
+
+<para>However, a side effect of this is that the <command>fdbbackup</command>
+command doesn't work properly for local filesystem backups: FoundationDB uses a
+server process alongside the database processes to perform backups and copy the
+backups to the filesystem. As a result, this process is put under the
+restricted namespaces above: the backup process can only write to a limited
+number of paths.</para>
+
+<para>In order to allow flexible backup locations on local disks, the
+FoundationDB NixOS module supports a
+<option>services.foundationdb.extraReadWritePaths</option> option. This option
+takes a list of paths, and adds them to the systemd unit, allowing the
+processes inside the service to write (and read) the specified
+directories.</para>
+
+<para>For example, to create backups in <command>/opt/fdb-backups</command>,
+first set up the paths in the module options:</para>
+
+<programlisting>
+services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
+</programlisting>
+
+<para>Restart the FoundationDB service, and it will now be able to write to
+this directory (even if it does not yet exist.) Note: this path
+<emphasis>must</emphasis> exist before restarting the unit. Otherwise, systemd
+will not include it in the private FoundationDB namespace (and it will not add
+it dynamically at runtime).</para>
+
+<para>You can now perform a backup:</para>
+
+<programlisting>
+$ sudo -u foundationdb fdbbackup start  -t default -d file:///opt/fdb-backups
+$ sudo -u foundationdb fdbbackup status -t default
+</programlisting>
+
+</section>
+
+<section><title>Known limitations</title>
+
+<para>The FoundationDB setup for NixOS should currently be considered beta.
+FoundationDB is not new software, but the NixOS compilation and integration has
+only undergone fairly basic testing of all the available functionality.</para>
+
+<itemizedlist>
+  <listitem><para>TLS plugin support is compiled in, but it's currently not
+      possible to specify the set of TLS certificate options in
+      <command>services.foundationdb</command></para></listitem>
+  <listitem><para>There is no way to specify individual parameters for
+      individual <command>fdbserver</command> processes. Currently, all server
+      processes inherit all the global <command>fdbmonitor</command> settings.
+      </para></listitem>
+  <listitem><para>Python bindings are not currently installed.</para></listitem>
+  <listitem><para>Ruby bindings are not currently installed.</para></listitem>
+  <listitem><para>Java bindings are not currently installed.</para></listitem>
+  <listitem><para>Go bindings are not currently installed.</para></listitem>
+</itemizedlist>
+
+</section>
+
+<section><title>Options</title>
+
+<para>NixOS's FoundationDB module allows you to configure all of the most
+relevant configuration options for <command>fdbmonitor</command>, matching it
+quite closely. For a complete list of all options, check <command>man
+configuration.nix</command>.</para>
+
+</section>
+
+<section><title>Full documentation</title>
+
+<para>FoundationDB is a complex piece of software, and requires careful
+administration to properly use. Full documentation for administration can be
+found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.</para>
+
+</section>
+
+</chapter>