Enter Tigris

Tigris is an S3 Compatible file storage built on Fly.io, for Fly.io users, with interesting properties:

  • Objects are initially stored in the region closest to the uploader, say Chicago, where they are most likely to be served.
  • If that object is requested from Singapore, Tigris will transparently move, then cache the object there. Like a CDN you don’t have to configure.
  • Every file under 128kb is cached globally.

How they do that is beyond the scope of this post but we recommend reading through their architecture documentation, it's very good!

I am very interested in that last bullet point; if every object under 128kb is cached and synced globally, that means we have a globally spanning Key Value store that should be crazy fast to read and write to! So let’s build a simple KV Store against Tigris.

Experiment

Building off my previous post where I broke down and built our own AWS client, I’ll be using a similar extension to Req I built as shown in this gist. Our entire API will be something like this:

KV.get("/path/to/key") # get key contents KV.put("/path/to/key", value) # update/create KV.put_transaction("/path/to/key", value) # update/create with global lock on key KV.delete("/path/to/key") # delete KV.get("/path/to/") # list keys

I will spare you the details here as this is literally just reading and writing to a file and encoding the value using :erlang.term_to_binary.

Keen observers might realize we have one glaring issue and that’s data races. If we have a globally spanning cluster and two regions write to the same value at the same time, we’ll have a conflict.

To solve this, there is one function in our API that’s not like the others: put_transaction. The BEAM comes with a function that helps us out here called global.trans/3. It will create a “global lock on an id” and execute your callback ONLY on a single node. Our put_transaction code literally looks like:

def put_transaction(key, value) do :global.trans(key, fn -> put(key, value) end, 0) end

Now we have globally distributed transactions on our KV store. Normally the trans function will retry forever until it gets a global lock, in our code we tell it to not retry; we’ll fail here with :aborted and let our users figure out what to do.

And that is that! We’re using the key based “File System” of Tigris as a KV store and so long as our values are under 128kb we’re syncing as fast as the internet allows!

Setup Tigris

Assuming you are using Fly.io already, adding Tigris to an existing app is as easy as:

$ fly storage create ? Choose a name, use the default, or leave blank to generate one: demo-bucket Your project (demo-bucket) is ready. See details and next steps with:

Setting the following secrets on app:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • BUCKET_NAME
  • AWS_ENDPOINT_URL_S3

Secrets are staged for the first deployment

And that is it! We have full access to the Tigris API and the common configuration variables already set. As simple as that we have a functional KV store that's durable, globally cached and for cheap.

Brainstorming

Let’s brainstorm what else we could achieve with this, noting this is a brainstorming session; no bad ideas, and not production promises.

Durable Processes

Since the read/write latency is low, we can simply write our process state out to Tigris every few seconds and/or on terminate. The next time the process or LiveView is loaded, we have our last known good state.

Tables and Indexes Oh My!

The API is just a file system; we could use key paths as a sort of table spaces Req.get("s3://kv-bucket/users/") or create indexes of data we need to query often Req.get("s3://kv-bucket/users/idaho"), keeping those indexes up to date via a Worker is an exercise for the reader.

Single Tenancy

Maybe your users need exclusive single tenancy; you could put_transaction a SQLite DB into a folder with a .lock key pointing to the Fly machine-id that owns it. When the first user opens up the website, download the DB and put_transaction the .lock file. When the next user opens up the page in a new region, check the lock and redirect them to the machine that owns the DB with fly-replay. Syncing everything periodically or when the last user closes the page, you’ve got a globally distributed SQLite. Obviously, this would fall over if you have too much concurrency, racing to read/write the lock file, but for smaller users, this could work well!

Append-only Tables

Every machine can create new files with their data in a common directory. Another machine could query that list of files and collate them into a common thread.

Wrap-up

Tigris has given a rare gift, a new way to look at an old tool, S3. Globally distributed, close to our users, and fast global sync for files smaller than 128kb. Not only can we simply replace S3 in our existing stacks, but we can also use it in new and fresh ways!

I can’t wait to see what you build with Tigris!