FoundationDB (FDB) is an ACID-compliant, multi-model, distributed database. The software started out life almost ten years ago. In March of 2015, Apple acquired the company behind FDB and in 2018, they open sourced the software under an Apache 2.0 license. VMWare's WaveFront runs an FDB cluster with at least a petabyte of capacity, Snowflake runs FDB for their metadata storage for their Cloud database service and Apple uses FDB for their CloudKit backend.
FDB uses the concept of Layers to add functionality. There are layers for MongoDB API-compatible document storage, record-oriented storage and SQL support among others.
FDB is optimised for SSDs to the point that you need to decide between HDD and SSD-specific configurations when setting up a database. The clustering support allows for scaling both up and down with data being automatically rebalanced. FDB utilises SQLite for its underlying storage engine.
FDB itself is written in Flow, a programming language the engineers behind FDB developed. The language adds actor-based concurrency as well as new keywords and control-flow primitives to C++11. As of this writing FDB is comprised of 100K lines of Flow / C++11 code and a further 83K lines of C code.
In this post I'll take a look at setting up a FoundationDB cluster and running a simple leaderboard example using Python. The leaderboard code used in this post originated in this forum post.
A FoundationDB Cluster, Up & Running
I've put together a cluster of three m5d.xlarge on AWS EC2. These instance types come with 4 vCPUs, 15 GB of RAM, 150 GB of NVMe SSD storage and up to 10 GBit/s of networking connectivity. The three instances cost of $0.75 / hour to run.
On all three instances I'll first format the NVMe partition using the XFS file system. This file system was first created by Silicon Graphics in 1993 and has excellent performance characteristics when run on SSDs.
$ sudo mkfs -t xfs /dev/nvme1n1
$ sudo mkdir -p /var/lib/foundationdb/data
$ sudo mount /dev/nvme1n1 /var/lib/foundationdb/data
I'll then install some prerequisites for the Python code in this post.
$ sudo apt update
$ sudo apt install \
python-dev \
python-pip \
virtualenv
On the first server I'll create a virtual environment and install the FoundationDB and Pandas python packages.
$ virtualenv ~/.fdb
$ source ~/.fdb/bin/activate
$ pip install foundationdb pandas
FoundationDB's server package depends on the client package being installed beforehand so I'll download and install that first. The following was run on all three instances.
$ wget -c https://www.foundationdb.org/downloads/6.0.18/ubuntu/installers/foundationdb-clients_6.0.18-1_amd64.deb
$ sudo dpkg -i foundationdb-clients_6.0.18-1_amd64.deb
The following will install the server package and was run on all three instances as well.
$ wget -c https://www.foundationdb.org/downloads/6.0.18/ubuntu/installers/foundationdb-server_6.0.18-1_amd64.deb
$ sudo dpkg -i foundationdb-server_6.0.18-1_amd64.deb
I'll run a command to configure the first server to switch binding from the local network interface to the private network instead. This way it'll be reachable by the other two servers without being available to the wider internet.
$ sudo /usr/lib/foundationdb/make_public.py
/etc/foundationdb/fdb.cluster is now using address 172.30.2.218
I'll then take the contents from /etc/foundationdb/fdb.cluster on the first server and place them in the same file on the other two servers.
With the cluster configuration synced between all three machines I'll restart FDB on each of the systems.
$ sudo service foundationdb restart
I'll then configure FDB for SSD storage, triple replication and set all three instances up as coordinators.
$ fdbcli
configure triple ssd
coordinators auto
This is the resulting status after those changes.
status details
Using cluster file `/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 3
Cluster:
FoundationDB processes - 3
Machines - 3
Memory availability - 15.1 GB per process on machine with least available
Fault Tolerance - 0 machines (1 without data loss)
Server time - 02/18/19 08:53:13
Data:
Replication health - Healthy (Rebalancing)
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 0 MB
Operating space:
Storage server - 142.4 GB free on most full server
Log server - 142.4 GB free on most full server
Workload:
Read rate - 14 Hz
Write rate - 0 Hz
Transactions started - 9 Hz
Transactions committed - 1 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
172.30.2.4:4500 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 15.1 GB RAM )
172.30.2.137:4500 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.4 GB / 15.2 GB RAM )
172.30.2.218:4500 ( 1% cpu; 0% machine; 0.026 Gbps; 0% disk IO; 0.4 GB / 15.1 GB RAM )
Coordination servers:
172.30.2.4:4500 (reachable)
172.30.2.137:4500 (reachable)
172.30.2.218:4500 (reachable)
A FoundationDB & Python Leaderboard
I'll use Pandas to read CSV data in chunks out of a GZIP-compressed CSV file. This file contains 20 million NYC taxi trips conducted in 2009. I'll feed the originating neighbourhood(s) and final total taxi fare into FDB.
$ python
from datetime import datetime
import fdb
from fdb.tuple import pack, unpack
import pandas as pd
fdb.api_version(510)
@fdb.transactional
def get_score(tr, user):
user_key = pack(('scores', user))
score = tr.get(user_key)
if score == None:
score = 0
tr.set(user_key, pack((score,)))
tr.set(pack(('leaderboard', score, user)), b'')
else:
score = unpack(score)[0]
return score
@fdb.transactional
def add(tr, user, increment=1):
score = get_score(tr, user)
total = score + increment
user_key = pack(('scores', user))
tr.set(user_key, pack((total,)))
tr.clear(pack(('leaderboard', score, user)))
tr.set(pack(('leaderboard', total, user)), b'')
return total
cols = ['trip_id',
'vendor_id',
'pickup_datetime',
'dropoff_datetime',
'store_and_fwd_flag',
'rate_code_id',
'pickup_longitude',
'pickup_latitude',
'dropoff_longitude',
'dropoff_latitude',
'passenger_count',
'trip_distance',
'fare_amount',
'extra',
'mta_tax',
'tip_amount',
'tolls_amount',
'ehail_fee',
'improvement_surcharge',
'total_amount',
'payment_type',
'trip_type',
'pickup',
'dropoff',
'cab_type',
'precipitation',
'snow_depth',
'snowfall',
'max_temperature',
'min_temperature',
'average_wind_speed',
'pickup_nyct2010_gid',
'pickup_ctlabel',
'pickup_borocode',
'pickup_boroname',
'pickup_ct2010',
'pickup_boroct2010',
'pickup_cdeligibil',
'pickup_ntacode',
'pickup_ntaname',
'pickup_puma',
'dropoff_nyct2010_gid',
'dropoff_ctlabel',
'dropoff_borocode',
'dropoff_boroname',
'dropoff_ct2010',
'dropoff_boroct2010',
'dropoff_cdeligibil',
'dropoff_ntacode',
'dropoff_ntaname',
'dropoff_puma']
db = fdb.open()
counter, start = 0, datetime.utcnow()
for chunk in pd.read_csv('trips_xaa.csv.gz',
header=None,
chunksize=10000,
names=cols,
usecols=['total_amount',
'pickup_ntaname']):
for x in range(0, len(chunk)):
add(db,
chunk.iloc[x].pickup_ntaname,
chunk.iloc[x].total_amount)
counter = counter + 1
print (counter * 10000) / (datetime.utcnow() - start).total_seconds()
The above imported at a rate of 495 records per second. While the import was taking place I was able to begin querying the leaderboard.
from operator import itemgetter
import fdb
from fdb.tuple import pack, unpack
fdb.api_version(510)
@fdb.transactional
def top(tr, count=3):
out = dict()
iterator = tr.get_range_startswith(pack(('leaderboard',)), reverse=True)
for key, _ in iterator:
_, score, user = unpack(key)
if score in out.keys():
out[score].append(user)
elif len(out.keys()) == count:
break
else:
out[score] = [user]
return dict(sorted(out.items(), key=itemgetter(0), reverse=True))
top(db)
This is the top three pick up points by total cab fare after a few minutes of importing the CSV file.
{75159.25000000016: ['Hudson Yards-Chelsea-Flatiron-Union Square'],
47637.469999999936: ['SoHo-TriBeCa-Civic Center-Little Italy'],
134147.24000000008: ['Midtown-Midtown South']}