Subscribing To A Database

NOTE: THIS PAGE IS CURRENTLY UNDER DEVELOPMENT

This article describes a pattern for wiring database queries into a re-frame app. It describes how you can "subscribe" to data in a remote database in a simple, clean and flexible way.

There Can Be Only One?

re-frame apps have a single source of data called app-db.

The re-frame README asks you to imagine app-db as something of an in-memory database. You query it (via subscriptions) and transactionally update it (via event handlers).

Components Don't Know, Don't Care

Components/Views never know of app-db's structure, much less its existence.

Instead, they subscribe, somewhat declaratively, to data, a bit like this (subscribe [:something "blah"]), and that allows Components to obtain a stream of updates to "something", while knowing nothing about the source of these updates.

A 2nd Source

All good but ...

SPAs are seldom completely self contained from a data point of view.

There's a continuum between apps which are 100% standalone data-wise, and those where remote-data is utterly central to the app's function. In this article, we're exploring more the remote-data-centric end of this continuum.

And just to be clear, when I'm talking about remote-data, I'm thinking of remote databases like firebase, rethinkdb, Postgress, Datomic, etc - data sources that the app must query and mutate.

So, the question is: how would we integrate this kind of remote data into an app when re-frame has only one source of data: app-db? How do we introduce a second or even third source of data? How would we subscribe to that data, and how would we update that data?

By way of explanation, let's explore a concrete version of that question: how could we wire up a Component which displays a collection of items, when those items come from a remote database?

In your mind's eye, imagine the items would be obtained via a query against that remote database: select id, price, description from items where type="see through".

Always Via A Subscription

In re-frame, Components always obtain data via a subscription. Always.

Our items-showing Component is going to (subscribe [:items "see through"]) and the subscription handler will deliver the items.

So, somewhere there will be a subscription handler defined:

(re-frame/register-sub
  :items
  (fn [db [_ type]
    ...))

Which is fine ... except we haven't really solved this problem at all, have we? We've just transferred the problem away from the Component and into the subscription handler?

Well, yes, we have, and isn't that a fine thing!! That's precisely what we want from our subscription handlers ... to manage how the data is sourced. To hide that from the Component.

Our Subscription Handlers Job

I'll give code in a minute, but first, let's describe how the subscription handler will work:

1. Upon being required to provide items, it has to issue a query to the remote database. See the select statement above. Or perhaps it will be done Restfully. Or via a firebase connection. Something. But it is the subscription handlers job to know how to do this.

2. This query be async - the results will arrive sometime "later". So this handler must organise for the query results to be placed into app-db, at some known path, when they eventually arrive. In the meantime, the handler might want to ensure that the absence of results is also communicated to the Component, so perhaps it can display "Loading ...".

3. The subscription handler must return something to the Component. It should give back a reaction to that path within app-db, so that when the query results eventually arrive, they will flow through into the component for display.

4. The subscription handler will detect when the Component no longer requires the subscription - the Component ceases to exist - and it will clean up, getting rid of those now-unneeded items, and sorting out any stateful database connection issues.

Notice what's happening here. In many respects, app-db is still acting as the single source of data. The subscription handler is organising for the right remote data to "flow" into app-db at a certain path, when it is required. And, equally, for this data to be cleaned up when it is no longer required.

Also, notice that putting ALL interesting data into app-db has nice flow on effects. In particular, it means it is available to event handlers, should they need it when servicing events (event handlers get db as a parameter, right?).
If this item data was held in a separate place, other than app-db, it wouldn't be available in this useful way.

Some Code

Enough fluffing about with words, here's a code sketch for our subscription handler:

(re-frame/register-sub
  :items
  (fn [db [_ type]
    (let  [query-token (issue-items-query! 
                          type  
                          :on-success #(re-frame/dispatch [:write-to  [:some :path])])]
      (reagent/make-reaction
        (fn [] (get-in @db [:some :path] []))
        :on-dispose #(do (terminate-items-query! query-token)
                       (re-frame/dispatch [:items-cleanup [:some :path]))))

A few things to notice:

1. You have to write issue-items-query!. Are you making a Restful GET? Are you writing json packets down a websocket? The query has to be made.

2. We do not issue the query via a dispatch because, to me, it isn't an event. But we most certainly do handle the arrival of query results via a dispatch and associated event handler. That to me is an external event happening to the system. This handler can curate the arriving data in whatever way makes sense. Maybe it does nothing more than to assoc into a app-db path, or maybe this is a rethinkdb changefeed subscription and your event handler will have to collate the newly arriving data with what has previously been returned. Do what needs to be done in that event handler, so that the right data to be put into the right path.

3. We use Reagent's make-reaction function to create a reaction which will return that place within app-db where the query results are to be placed.

4. We use the on-dispose callback on this reaction to do any cleanup work when the subscription is no longer needed. Clean up app-db? Clean up the database connection?

Any Good?

It turns out that this is a surprisingly flexible and clean approach. There's a lot to like about it.

For example, if you are using rethinkdb, which supports queries which yield "change feeds", rather than a one-off query result, you have to actively close such queries, when they are no longer needed. That's easy to do in our cleanup code.

We can source some data from both Postgress and firebase in the one app, using the same pattern.

With some extra work, we could keep a register of all current queries (subscriptions). Then, if ever we noticed that the app had lost internet connection, and then regained it, we could organise for all this queries to be rerun, avoiding stale results.

Problems

Because we are storing remote data in app-db, this approach is currently incompatible with undo and redo. In the next version of re-frame, we'll fix that by allowing only part of app-db to be versioned in undo/redo operations. The part which doesn't hold remote data.

Query De-duplication

In cases where the same query is simultaneously issued from multiple places, you'd want to de-duplicate the queries.

One possibility is to do this duplication in issue-items-query! itself. You can count count the duplicate queries and only clear the data when that count goes to 0.

Dynamic subscriptions

XXX

Notes from @nidu

1. If I understand the subscription snippet correctly - you will call issue-items-query! multiple times if you subscribe one resource multiple times, which is what i tried to avoid with counters. However this logic can be incorporated inside issue-items-query!.
2. You say that initiating data fetching is not an event, but it probably should transition app-db into loading state which is an observable state change.

Thanks To

@nidu for his very valuable comments and insights on this material