Subscribing To A Database

NOTE: THIS PAGE IS CURRENTLY UNDER DEVELOPMENT

This article describes a simple pattern for wiring database queries into a re-frame app. It describes how you can "subscribe" to data in a remote database in a clean way.

The One True Source

In the reference implementation of re-frame, apps have a single source of data called app-db.

The re-frame README asks you to imagine app-db as something of an in-memory database. You query it (via subscriptions) and transactionally update it (via event handlers). Sometimes the updates are simple and CRUD-like and sometimes the queries are complicated, involving the equivalent of select with where clauses and sorting and limits.

Components Don't Know, Don't Care

Components/Views never know of app-db's structure, much less its existence.

Instead, they 'subscribe' somewhat declaratively to data, perhaps like this (subscribe [:something "blah"]), which allows Components to obtain a stream of updates to "something", while knowing nothing about the source of the data.

A 2nd Source

SPAs are seldom completely self contained from a data point of view. And, obviously, there is a continuum between apps which are truly standalone data-wise, and those which are so remote-data-centric that it is utterly central to the app's function. In this article, we're exploring the remote-data-centric end of things.

And just to be clear, when I'm talking about remote-data, I'm thinking of remote databases like firebase, rethinkdb, Postgress, Datomic, etc. Data sources that we want to query and mutate.

So, the question is: how would we integrate this kind of remote data into an app when re-frame requires only one source of data: app-db?

By way of explanation, let's deal with a more concrete version of that question: how could we wire up a Component which
displays a collection of items, when those items come from a remote database?

In your minds eye, imagine that these items should be obtained via a select id, price, description from items where type="see through" across on that remote database.

Always Via A Subscription

In re-frame, Components always obtain data via a subscription. Always. So our items-showing Component is going to (subscribe [:items "see through"]). It is the subscription handler which will deliver the items.

So, somewhere a subscription handler will be defined:

(re-frame/register-sub
  :items
  (fn [db [_ type]
    ...))

Which is fine ... except we haven't solved this problem at all, have we? We've just transferred the problem away from the Component and into the subscription handler?

Well, yes, we have and isn't that a fine thing!! That's precisely what we want from our subscription handlers ... to manage how the data is sourced. To hide that from the Component. Well done us!

Our Subscription Handlers Job

I'll give code in a minute, but first, let's describe the how the subscription handler will work:

1. Upon being required to provide item data, it has to issue a query to the remote database. See the select statement above. Or perhaps it will be done Restfully. Or via a firebase connection. Something. But it is the subscription handlers job to know how to do this.

2. The laws of physics demand that this query be async. The query results will arrive sometime "later". So this handler must organise that the query results are eventually placed into app-db, at some known path. In the meantime, it will ensure that the absence of results is also communicated to the Component's, so perhaps it can display "Loading ...".

3. It will return, to the Component, a reference to that path within app-db, so that when the query results eventually arrive, they will flow through into the component for display.

4. It will detect when the Component no longer requires the subscription, and it will clean up, getting rid of those now-unneeded items, and sorting out any stateful database connection issues.

Notice what's happening here. In many respects, app-db is still acting as the single source of data. The subscription handler is organising for the right remote data to "flow" into app-db at a certain path, when it is required. And, equally, for this data to be cleaned up when its no longer required.

Also, notice that by putting ALL data into app-db, it is available to event handlers as well, should they need it when servicing events. Were this data held off in a separate place, other than app-db, it wouldn't be available in this useful way.

Some Code

Enough fluffing about, here's a code sketch:

(re-frame/register-sub
  :items
  (fn [db [_ type]
    (let  [query-token  (issue-items-query! 
                          type  
                          :on-success #(re-frame/dispatch [:write-to  [:some :path])])]
      (reagent/make-reaction
        (fn [] (get-in @db [:some :path] []))
        :on-dispose #(do (terminate-items-query! query-token)
                       (re-frame/dispatch [:items-cleanup [:some :path]))))

A few things to notice:

1. You have to write issue-items-query!. Are you making a Restful GET? Are you writing json packets down a websocket? The query has to be made.
2. We do not issue the query via a dispatch because, to me, it isn't an event. But we most certainly do handle the arrival of query results via a dispatch and associated event handler. That to me is an external event happening to the system. This handler can curate the arriving data in whatever way makes sense. Maybe it does nothing more than a assoc into this path, or maybe this is a rethinkdb changefeed subscription and your event handler will have to carefully collapse the newly arriving data with what has previously been returned. Do what needs to be done in that event handler.
3. we use the make-reaction function to create a reaction which will return that place where query results are to be placed.
4. We use the on-dispose callback on the reaction to do any cleanup work when we realize that the subscription is no longer needed.

What To Like

There's a lot to like about the flexibility of this approach. For example, if you are using rethinkdb, which supports queries which yield "change feeds", rather than one off query result, you have to actively close such queries, when they are no longer needed. That's easy to do in our cleanup code.

We can control how data "flows" out of the query into app-db.

With some extra work, we could keep a register of all current queries and reissue them, if ever we noticed that we'd lost internet connection, and then regained it.

Problems

Because we are storing remote data in app-db, this approach is currently incompatible with undo and redo. We a future release of re-frame, we'll fix that.

combining this with dynamic subscriptions

XXX