Implementing live updates with Backbone.js

I don’t know about you, but backbone.js always left me crave for more. It’s the simplicity of it that makes me want to learn how others are using it to solve real world problems. When I’ve started work on Clevertim CRM, which heavily uses backbone.js, I wanted to read about some of the real world problems and their solutions, in order to reuse as much as possible, rather than spend time re-inventing the wheel. In the end, I had to come up with my own solutions for a lot of problems, like paginating and filtering backbone.js views/collections and got bitten by subtle bugs. While it was a good learning experience, sometimes, wise people learn from other people’s experiences and that’s always preferred. This article is part of my effort to share some of the lessons learnt and get feedback on the solutions I have come up with, which, no doubt, can be improved.

In this article, I’m going to cover the problem of live updates that I had to solve. It’s a problem that comes up a lot when you implement collaborative solutions. Having many users updating the same information, you need to ensure that everyone sees the updates seamlessly, in real-time. Translating that into technical requirements, that would be:

  1. Detect when collections and models change on the server side and be able to update all the clients
  2. Handle collection additions – a new model is added to the collection
  3. Handle collection updates – existing models get updated
  4. Handle collection deletes – existing models get deleted and therefore, removed from collections

As a first step, I started by looking at the infrastructure provided by backbone.js to update collections, hoping I can reuse that. Collections have a fetch method which (re)loads the collection from the server. It has support for reloading the entire collection (a “reset fetch”) or appending new models to your collection (an “add fetch”). Fetching is simple to use:


    coll.fetch();
    coll.fetch( { add : true } );

The former is doing a “reset fetch”, while the latter is an “add fetch”.
The “add” version, while doing an incremental update, as I wanted, it cannot cope with updates or deletes. In particular, if you send an existing model from the server, the “add fetch” will detect it as duplicate and ignore it. It will not update the existing model.

The “reset fetch” can get the job done but it’s slow and it’s not going to win you any beauty contests. In my case, implementing a CRM system, I had to deal with a lot of data, potentially thousands of models for each collection, so doing a “reset fetch” every time anything changes was out of the question. I think “reset fetch” is more suited for smaller collections that don’t change very often.

At this point, I was on my own. I had to implement my own way of getting incremental updates and updating the backbone.js models and the solution had to be generic enough so it covers all the models and collections that I had. To give you an idea, I had collections for contacts, companies, sales opportunities, cases, files, tasks, notes, comments, persisted filters, custom fields, users, latest activity stream and a few others.

The server side and the detection problem

The server side is not a backbone.js problem per se, but it’s a critical part of the solution to implementing live updates on the client side and it just cannot be ignored. On the server side you have to answer a few questions like:

  1. How can we detect when models change (add, update, delete operations) so we can stream those back to other clients
  2. How will those changes be streamed back to clients
  3. What type of data structure to use to send data back; something that’s generic enough to cover all your collections and make adding new collections as painless as possible

Detecting server side changes

The requirement is to detect whether a model has changed since the last client update. It’s not just a boolean whether something has changed or not as it needs to cater for multiple clients (users), each updating at slightly different times.
My idea here was for each record for each model to store the timestamp of the last update and then each client stores the timestamp of the last “sync” operation with the server.
Sync-ing becomes a question of “Give me all the records that have timestamp >= client timestamp”.

So, for each DB data model corresponding to my backbone.js models, I added 3 timestamps:


added_on         - timestamp when was the model added
deleted_on       - timestamp when the model was deleted (marked deleted)
last_modified_on - timestamp when any modifications (add, update, delete)
                   were made to this model

For clarity, I’ve also added a boolean “is_deleted” that marks the models that have been deleted. It’s worth noticing that, in this model, we can’t just delete models from the DB, as we would not be able to easily detect those changes, since the models will no longer be in the DB.

Relying that last_modified_on gets updated for all the operations (add, update, delete), detecting what has changed can be done by running queries against the DB data model, something like this:


select * from contacts where last_modified_on >= client_timestamp
select * from companies where last_modified_on >= client_timestamp
select * from sales where last_modified_on >= client_timestamp

and so on…

We can easily detect what has changed with a few conditions on the model, which we can process when we iterate over the results of the queries above.


if model.added_on >= client_timestamp => added
if model.is_deleted and model.deleted_on >= client_timestamp => deleted
else => updated (relies on the implicit condition already in the sql:
                    last_modified_on >= client_timestamp)

Client/Server data structure

Now that we have detected what has changed, it’s a matter of packing the info back into a data structure that the client can process in order to update the backbone.js models. For each model, we can send back what has changed in a JSON data structure like this:


	{
	  "add" : [ json_model1, json_model2, ... ],
	  "upd" : [ json_modelA, json_modelB, ... ],
	  "del" : [ json_modelX, json_modelY, ...]
	}

This contains a list of models that were added, a list with the updated models and a list with the models that were deleted from the last client timestamp. This is just formatting in a JSON structure the changes we detected earlier.

We need to do this for every collection that has changed.

We also need to pass back the new timestamp which represents an “as of” for the incremental data that we’re sending. Once the client updates the backbone.js models with the data we’re sending, it’ll also update its own client_timestamp to this new timestamp. This actually ensures we’re not going to get the same changes again and we’re only going to always get the new changes that we don’t yet have.
The final data structure can look something like this:


{
    data : {
        contacts : {
                      "add" : [ json_model1, json_model2, ... ],
                      "upd" : [ json_modelA, json_modelB, ... ],
                      "del" : [ json_modelX, json_modelY, ...]
                    },
        companies : {
                      "add" : [ json_model1, json_model2, ... ],
                      "upd" : [ json_modelA, json_modelB, ... ],
                      "del" : [ json_modelX, json_modelY, ...]
                    }
        ...
    },
    timestamp: the_new_timestamp
}

json_models above should be the json representation of your models. This is important, because then we can pass the whole array into add or remove calls on the relevant collections, rather than have to iterate in our code over the models.

Client/Server communication

The client needs to constantly keep in sync with the server, so that implies either a web socket, COMET “forever connection” or a simpler AJAX polling scheme. The advantage of the latter is that is easier to implement and understand, so we’ll go for that. Since this mechanism is well encapsulated, the implementation detail for this communication can be changed later on, without affecting the rest of the implementation.

In the AJAX polling solution, a client will setup a regular callback with setInterval and make an AJAX call to the server to a /sync/ endpoint. It will pass in its latest client_timestamp and the server will run the queries to detect what has changed and reply with the JSON structure to pass that information back to the client.

For each collection under data, the client can easily now update the backbone.js associated collection. It’s just a matter of doing a collection.add for new models, collection.remove for deleted models and iterating over updated models and calling set on each one. Here’s some code, as an example:


updateCollection : function(data, collectionTag, collection, addToFront){
    var colInfo = data[collectionTag],
        add, upd, del;
    if(colInfo) {
        //do incremental update of the collection
        add = colInfo.add;
        if(add && _.isArray(add) && add.length > 0) {
            if(addToFront){
                collection.add(add, { at : 0 });
            } else {
                collection.add(add);
            }
        }
        upd = colInfo.upd;
        if(upd && _.isArray(upd) && upd.length > 0) {
            _.each( upd, function(elem) {
                if(elem.id) {
                    var model = collection.get(elem.id);
                    if(model) {
                        model.set(elem);
                    } else {
                        collection.add(elem);
                    }
                }
            } );
        }
        del = colInfo.del;
        if(del && _.isArray(del) && del.length > 0) {
            collection.remove(del);
        }
    }
}

The nice thing about this function is that we can call it in turn for all our collections:


    updateCollection(data, 'contacts', contactsColl);
    updateCollection(data, 'companies', companiesColl);
    updateCollection(data, 'sales', salesColl);
    updateCollection(data, 'whatsNew', whatsNewColl, true);

For whatsNew, we add to the front of the collection, so that we can display more easily the new items at the top of our view.
Once you update the collections and nothing fails, you can safely update the timestamp associated with the incremental data you have just received.

app.timestamp = data.timestamp;

We will pass this new timestamp into the next sync AJAX call.
If something fails during the update of the models, we don’t update the timestamp, so next time we’ll get the same data again, just to make sure we’re not losing updates due to transient errors.

Conclusion

That’s pretty much it in terms of live updates.

Views will automatically update on the triggers from add, change or remove on collections or on the individual models.

For performance reasons, you’ll want to ensure you only update your views once. If you receive many updates via this sync mechanism, you don’t want to update the view for every element. Try it if you don’t believe me it’s going to kill the performance of your app.

Backbone.js will trigger an add/remove/change event for every element you add, which could in turn trigger a re-render of your views. To get around that, just debounce your trigger handlers like this:

    collection.on("add render change", _.debounce( reRenderView, 0 ));

Another significant problem to solve in relation to updating views like that is that you don’t want to do it if the user is in the middle of typing something, attaching something, playing with the view you’re rendering under him. You don’t want any typed text to be lost. This will add some complexity to your views, in particular the re-rendering part. Implementing that can be done with simple state machines in your views and deferring re-rendering in certain states.

A scheme like the one I have just described is currently working well for our one page CRM app. We do have a Quick Demo if you’re interested to have a look. If you do decide to have look, don’t hesitate to send me any feedback. I’m always looking for feedback on how I can improve it. The same goes for the code above. If you think it can be done better, let me know. After all, it was one of my goals when posting this article here.

Mike Speranza is a co-founder at Clevertim, a provider of a web based contact manager and simple CRM for small companies. Clevertim CRM is built with backbone.js on the client side and django/python on the server side. He maintains this CRM blog and this (rarely updated) technology blog. He lives in London, UK.

3 Comments on "Implementing live updates with Backbone.js"

  1. moby says:

    hi,
    i quickly hovered your article. as far as i glanced it covers live updates over rest-ful api. would it be any chance to hear on real-time data over sockets. how it will fit to backbone structure. thanks.

  2. Joe says:

    Thanks! I had something very similar in mind, but you took some things into consideration that I hadn’t thought of. I pretty much just used your method verbatim. Thanks for the insight.

  3. You really make it seem so easy with your presentation but I find this matter to
    be actually sometthing which I think I would never understand.
    It seems too complicated and very broad for me.
    I’m looking forwzrd for your next post, I will try to get the
    hang of it!

    my blog – stem cells diabetes bbc

Got something to say? Go for it!