ShareJS and ShareDB

by Joseph Gentle

This was posted on the sharejs & derby mailing lists and there's a great discussion there.

I first wrote ShareJS hoping for a simple library on top of which people could build their own collaborative applications. I showed off the first working version in April 2012, running on NodeJS 0.4. At the time the complete list of nodejs libraries fit in a github wiki page. There was also no websockets - not even a draft spec.

When I wrote ShareJS I wasn't thinking about JSON at all. It was a strange little idea a german researcher proposed over Italian food at the google wave summit. (Held right after wave was cancelled). I somehow convinced Jeremy (@nornagon) to implement the damn thing (much harder than it sounds!) and we put it in sharejs, because sharejs can support editing arbitrary types.

By the end of 2012 ShareJS was up to version 0.6, which was a beautifully crafted snowflake. I prided myself on its tests-to-code ratio of about 1.5:1. I took it personally when bugs survived in that code base. Those tests were good enough to find bugs in socket.io, bugs in uglifyjs and once a bug in the coffeescript compiler. I had sharejs working on browsers down to IE6.

Then at the start of 2013 I moved to the bay area and started working at Lever. The company was only a few months old at the time. Nate (our CEO) had written this little framework called Derby which he'd presented at the realtime.js conference. We geeked out over combining forces - what would Derby look like on top of ShareJS? It was a total conceit - probably a terrible idea for such a young company to invest resources into but we did it anyway. We did it - but we never had quite enough time to do everything well. It was as fast as it needed to be, with just enough features to make our app work just well enough. I'm super proud of what lever is growing into, but I want my code to never fall down. That was my standard with sharejs a few years ago, and I ruined it. I still get this guilty vertigo feeling when I look at the issue tracker.

This all hit home early this year when I was invited onto some

javascript podcast. One of the people on the call asked about using sharejs in his own project - he wanted a simple collaborative text editor. I told him about all of sharejs's great features and he said in that case it probably wasn't the right tool for him. I didn't know how to respond - it was like I had been slapped in the face. I made ShareJS exactly for his use case; and I failed. I took sharejs.org down the other day and I felt good about it - I felt like the site was lying. (And it was still running 0.5 if you can believe it.)

I've been struggling all year to answer the question of, what next?

In a sense I've made two versions of sharejs. The first version (0.6) is a small tool to embed collaborative text editing on your site.

The second version (livedb + sharejs 0.7) is an OT-based database for realtime websites.

My hope with the 0.7 rewrite was to have a unified library to cover all of the above use cases. I think I've failed in that goal.

So my plan going forward is to split the project in two.

What is currently ShareJS master (v0.7) will be combined with livedb and renamed sharedb. Nate & his team at Lever will run sharedb (as they've been doing for the past year). It will be updated to only support JSON documents at the root level - which simplifies a lot of code. (JSON documents support subdocuments with arbitrary OT types anyway, so there won't be any loss in functionality.) Nate is also talking about merging the web client code into the repo with livedb.

Meanwhile I'm going to revert sharejs back to the 0.6 codebase and modernise the whole thing. The API should stay (more or less) the same as it was at v0.6, although I'm going to merge the cursors branch in.

In practical terms this means:

  • Native (default) websocket support
  • Coffee -> JS
  • Mocha
  • Browserify
  • Out of the box quilljs support + examples
  • Resurrect the old DB bindings for sqlite, postgresql, couchdb, leveldb, etc.
  • ShareJS will not support having multiple backend servers (unless you shard). (Although it would be nice if it could also use livedb (sharedb) as a backend somehow).
  • It'll be called sharejs 1.0 when this is done. (Although I feel like at this point we deserve a higher number)

I'm not sure about whether to just use a document name (like 0.6 does now) or use collection name + document name (0.7 style). I really don't have a strong opinion.

I don't have a timeframe for this - I want to finish the new json type first. But I wanted to hear everyone's thoughts before that happens.

As always, I'd love to hear comments / feedback about all of the above. Sorry its such a bumpy ride!