path: root/nixos/modules/services/databases/foundationdb.xml

<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb">
  <title>FoundationDB</title>
  <para>
    <emphasis>Source:</emphasis>
    <filename>modules/services/databases/foundationdb.nix</filename>
  </para>
  <para>
    <emphasis>Upstream documentation:</emphasis>
    <link xlink:href="https://apple.github.io/foundationdb/">https://apple.github.io/foundationdb/</link>
  </para>
  <para>
    <emphasis>Maintainer:</emphasis> Austin Seipp
  </para>
  <para>
    <emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
  </para>
  <para>
    FoundationDB (or <quote>FDB</quote>) is an open source, distributed,
    transactional key-value store.
  </para>
  <section xml:id="module-services-foundationdb-configuring">
    <title>Configuring and basic setup</title>
    <para>
      To enable FoundationDB, add the following to your
      <filename>configuration.nix</filename>:
    </para>
    <programlisting>
services.foundationdb.enable = true;
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
</programlisting>
    <para>
      The <option>services.foundationdb.package</option> option is
      required, and must always be specified. Due to the fact
      FoundationDB network protocols and on-disk storage formats may
      change between (major) versions, and upgrades must be explicitly
      handled by the user, you must always manually specify this
      yourself so that the NixOS module will use the proper version.
      Note that minor, bugfix releases are always compatible.
    </para>
    <para>
      After running <command>nixos-rebuild</command>, you can verify
      whether FoundationDB is running by executing
      <command>fdbcli</command> (which is added to
      <option>environment.systemPackages</option>):
    </para>
    <programlisting>
$ sudo -u foundationdb fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb&gt; status

Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory
  Coordinators           - 1

Cluster:
  FoundationDB processes - 1
  Machines               - 1
  Memory availability    - 5.4 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 04/20/18 15:21:14

...

fdb&gt;
</programlisting>
    <para>
      You can also write programs using the available client libraries.
      For example, the following Python program can be run in order to
      grab the cluster status, as a quick example. (This example uses
      <command>nix-shell</command> shebang support to automatically
      supply the necessary Python modules).
    </para>
    <programlisting>
a@link&gt; cat fdb-status.py
#! /usr/bin/env nix-shell
#! nix-shell -i python -p python pythonPackages.foundationdb52

import fdb
import json

def main():
    fdb.api_version(520)
    db = fdb.open()

    @fdb.transactional
    def get_status(tr):
        return str(tr['\xff\xff/status/json'])

    obj = json.loads(get_status(db))
    print('FoundationDB available: %s' % obj['client']['database_status']['available'])

if __name__ == &quot;__main__&quot;:
    main()
a@link&gt; chmod +x fdb-status.py
a@link&gt; ./fdb-status.py
FoundationDB available: True
a@link&gt;
</programlisting>
    <para>
      FoundationDB is run under the <command>foundationdb</command> user
      and group by default, but this may be changed in the NixOS
      configuration. The systemd unit
      <command>foundationdb.service</command> controls the
      <command>fdbmonitor</command> process.
    </para>
    <para>
      By default, the NixOS module for FoundationDB creates a single
      SSD-storage based database for development and basic usage. This
      storage engine is designed for SSDs and will perform poorly on
      HDDs; however it can handle far more data than the alternative
      <quote>memory</quote> engine and is a better default choice for
      most deployments. (Note that you can change the storage backend
      on-the-fly for a given FoundationDB cluster using
      <command>fdbcli</command>.)
    </para>
    <para>
      Furthermore, only 1 server process and 1 backup agent are started
      in the default configuration. See below for more on scaling to
      increase this.
    </para>
    <para>
      FoundationDB stores all data for all server processes under
      <filename>/var/lib/foundationdb</filename>. You can override this
      using <option>services.foundationdb.dataDir</option>, e.g.
    </para>
    <programlisting>
services.foundationdb.dataDir = &quot;/data/fdb&quot;;
</programlisting>
    <para>
      Similarly, logs are stored under
      <filename>/var/log/foundationdb</filename> by default, and there
      is a corresponding <option>services.foundationdb.logDir</option>
      as well.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-scaling">
    <title>Scaling processes and backup agents</title>
    <para>
      Scaling the number of server processes is quite easy; simply
      specify <option>services.foundationdb.serverProcesses</option> to
      be the number of FoundationDB worker processes that should be
      started on the machine.
    </para>
    <para>
      FoundationDB worker processes typically require 4GB of RAM
      per-process at minimum for good performance, so this option is set
      to 1 by default since the maximum amount of RAM is unknown. You’re
      advised to abide by this restriction, so pick a number of
      processes so that each has 4GB or more.
    </para>
    <para>
      A similar option exists in order to scale backup agent processes,
      <option>services.foundationdb.backupProcesses</option>. Backup
      agents are not as performance/RAM sensitive, so feel free to
      experiment with the number of available backup processes.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-clustering">
    <title>Clustering</title>
    <para>
      FoundationDB on NixOS works similarly to other Linux systems, so
      this section will be brief. Please refer to the full FoundationDB
      documentation for more on clustering.
    </para>
    <para>
      FoundationDB organizes clusters using a set of
      <emphasis>coordinators</emphasis>, which are just
      specially-designated worker processes. By default, every
      installation of FoundationDB on NixOS will start as its own
      individual cluster, with a single coordinator: the first worker
      process on <command>localhost</command>.
    </para>
    <para>
      Coordinators are specified globally using the
      <command>/etc/foundationdb/fdb.cluster</command> file, which all
      servers and client applications will use to find and join
      coordinators. Note that this file <emphasis>can not</emphasis> be
      managed by NixOS so easily: FoundationDB is designed so that it
      will rewrite the file at runtime for all clients and nodes when
      cluster coordinators change, with clients transparently handling
      this without intervention. It is fundamentally a mutable file, and
      you should not try to manage it in any way in NixOS.
    </para>
    <para>
      When dealing with a cluster, there are two main things you want to
      do:
    </para>
    <itemizedlist spacing="compact">
      <listitem>
        <para>
          Add a node to the cluster for storage/compute.
        </para>
      </listitem>
      <listitem>
        <para>
          Promote an ordinary worker to a coordinator.
        </para>
      </listitem>
    </itemizedlist>
    <para>
      A node must already be a member of the cluster in order to
      properly be promoted to a coordinator, so you must always add it
      first if you wish to promote it.
    </para>
    <para>
      To add a machine to a FoundationDB cluster:
    </para>
    <itemizedlist spacing="compact">
      <listitem>
        <para>
          Choose one of the servers to start as the initial coordinator.
        </para>
      </listitem>
      <listitem>
        <para>
          Copy the <command>/etc/foundationdb/fdb.cluster</command> file
          from this server to all the other servers. Restart
          FoundationDB on all of these other servers, so they join the
          cluster.
        </para>
      </listitem>
      <listitem>
        <para>
          All of these servers are now connected and working together in
          the cluster, under the chosen coordinator.
        </para>
      </listitem>
    </itemizedlist>
    <para>
      At this point, you can add as many nodes as you want by just
      repeating the above steps. By default there will still be a single
      coordinator: you can use <command>fdbcli</command> to change this
      and add new coordinators.
    </para>
    <para>
      As a convenience, FoundationDB can automatically assign
      coordinators based on the redundancy mode you wish to achieve for
      the cluster. Once all the nodes have been joined, simply set the
      replication policy, and then issue the
      <command>coordinators auto</command> command
    </para>
    <para>
      For example, assuming we have 3 nodes available, we can enable
      double redundancy mode, then auto-select coordinators. For double
      redundancy, 3 coordinators is ideal: therefore FoundationDB will
      make <emphasis>every</emphasis> node a coordinator automatically:
    </para>
    <programlisting>
fdbcli&gt; configure double ssd
fdbcli&gt; coordinators auto
</programlisting>
    <para>
      This will transparently update all the servers within seconds, and
      appropriately rewrite the <command>fdb.cluster</command> file, as
      well as informing all client processes to do the same.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-connectivity">
    <title>Client connectivity</title>
    <para>
      By default, all clients must use the current
      <command>fdb.cluster</command> file to access a given FoundationDB
      cluster. This file is located by default in
      <command>/etc/foundationdb/fdb.cluster</command> on all machines
      with the FoundationDB service enabled, so you may copy the active
      one from your cluster to a new node in order to connect, if it is
      not part of the cluster.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-authorization">
    <title>Client authorization and TLS</title>
    <para>
      By default, any user who can connect to a FoundationDB process
      with the correct cluster configuration can access anything.
      FoundationDB uses a pluggable design to transport security, and
      out of the box it supports a LibreSSL-based plugin for TLS
      support. This plugin not only does in-flight encryption, but also
      performs client authorization based on the given endpoint’s
      certificate chain. For example, a FoundationDB server may be
      configured to only accept client connections over TLS, where the
      client TLS certificate is from organization <emphasis>Acme
      Co</emphasis> in the <emphasis>Research and Development</emphasis>
      unit.
    </para>
    <para>
      Configuring TLS with FoundationDB is done using the
      <option>services.foundationdb.tls</option> options in order to
      control the peer verification string, as well as the certificate
      and its private key.
    </para>
    <para>
      Note that the certificate and its private key must be accessible
      to the FoundationDB user account that the server runs under. These
      files are also NOT managed by NixOS, as putting them into the
      store may reveal private information.
    </para>
    <para>
      After you have a key and certificate file in place, it is not
      enough to simply set the NixOS module options – you must also
      configure the <command>fdb.cluster</command> file to specify that
      a given set of coordinators use TLS. This is as simple as adding
      the suffix <command>:tls</command> to your cluster coordinator
      configuration, after the port number. For example, assuming you
      have a coordinator on localhost with the default configuration,
      simply specifying:
    </para>
    <programlisting>
XXXXXX:XXXXXX@127.0.0.1:4500:tls
</programlisting>
    <para>
      will configure all clients and server processes to use TLS from
      now on.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-disaster-recovery">
    <title>Backups and Disaster Recovery</title>
    <para>
      The usual rules for doing FoundationDB backups apply on NixOS as
      written in the FoundationDB manual. However, one important
      difference is the security profile for NixOS: by default, the
      <command>foundationdb</command> systemd unit uses <emphasis>Linux
      namespaces</emphasis> to restrict write access to the system,
      except for the log directory, data directory, and the
      <command>/etc/foundationdb/</command> directory. This is enforced
      by default and cannot be disabled.
    </para>
    <para>
      However, a side effect of this is that the
      <command>fdbbackup</command> command doesn’t work properly for
      local filesystem backups: FoundationDB uses a server process
      alongside the database processes to perform backups and copy the
      backups to the filesystem. As a result, this process is put under
      the restricted namespaces above: the backup process can only write
      to a limited number of paths.
    </para>
    <para>
      In order to allow flexible backup locations on local disks, the
      FoundationDB NixOS module supports a
      <option>services.foundationdb.extraReadWritePaths</option> option.
      This option takes a list of paths, and adds them to the systemd
      unit, allowing the processes inside the service to write (and
      read) the specified directories.
    </para>
    <para>
      For example, to create backups in
      <command>/opt/fdb-backups</command>, first set up the paths in the
      module options:
    </para>
    <programlisting>
services.foundationdb.extraReadWritePaths = [ &quot;/opt/fdb-backups&quot; ];
</programlisting>
    <para>
      Restart the FoundationDB service, and it will now be able to write
      to this directory (even if it does not yet exist.) Note: this path
      <emphasis>must</emphasis> exist before restarting the unit.
      Otherwise, systemd will not include it in the private FoundationDB
      namespace (and it will not add it dynamically at runtime).
    </para>
    <para>
      You can now perform a backup:
    </para>
    <programlisting>
$ sudo -u foundationdb fdbbackup start  -t default -d file:///opt/fdb-backups
$ sudo -u foundationdb fdbbackup status -t default
</programlisting>
  </section>
  <section xml:id="module-services-foundationdb-limitations">
    <title>Known limitations</title>
    <para>
      The FoundationDB setup for NixOS should currently be considered
      beta. FoundationDB is not new software, but the NixOS compilation
      and integration has only undergone fairly basic testing of all the
      available functionality.
    </para>
    <itemizedlist spacing="compact">
      <listitem>
        <para>
          There is no way to specify individual parameters for
          individual <command>fdbserver</command> processes. Currently,
          all server processes inherit all the global
          <command>fdbmonitor</command> settings.
        </para>
      </listitem>
      <listitem>
        <para>
          Ruby bindings are not currently installed.
        </para>
      </listitem>
      <listitem>
        <para>
          Go bindings are not currently installed.
        </para>
      </listitem>
    </itemizedlist>
  </section>
  <section xml:id="module-services-foundationdb-options">
    <title>Options</title>
    <para>
      NixOS’s FoundationDB module allows you to configure all of the
      most relevant configuration options for
      <command>fdbmonitor</command>, matching it quite closely. A
      complete list of options for the FoundationDB module may be found
      <link linkend="opt-services.foundationdb.enable">here</link>. You
      should also read the FoundationDB documentation as well.
    </para>
  </section>
  <section xml:id="module-services-foundationdb-full-docs">
    <title>Full documentation</title>
    <para>
      FoundationDB is a complex piece of software, and requires careful
      administration to properly use. Full documentation for
      administration can be found here:
      <link xlink:href="https://apple.github.io/foundationdb/">https://apple.github.io/foundationdb/</link>.
    </para>
  </section>
</chapter>