JavaScript filesystem database

Tue, 21 Dec 2010 18:33:00 +0000

I couldn’t resist and, after many conversations with Mário Valente, I challenged myself to build a complete JavaScript filesystem based database from scratch. In short, something like CouchDB but written in JavaScript and using a filesystem approach for document storage.

Some characteristics of this new database system:

fully written in JavaScript: nodeJS is my choice for its architecture and portability;
every document must contain only JSON data;
every document can be manipulated on the filesystem: the idea is to allow someone to edit a document using vim, for instance, without breaking anything. It also lets you easily backup, move or replicate your data without creating a heavy load on the database;
the underlying filesystem can be changed: changing the underlying filesystem offers numerous possibilities, like easy distribution (using OpenAFS, XTREEMFS or others), replication (using finefs or others), and even attaching different backends (using FUSE, for example);
the system must be available through an HTTP REST interface: this makes it very easy to immediately integrate the system into any existing applications. Bonus points if the API is compatible with CouchDB;
queries must be performed using a MapReduce approach: this should make it very easy to perform queries on a large dataset.

So, right now I’m half way through it. I managed to write the backend and I’m on my way to the HTTP REST interface. I’ll create a github repo after I have something functional that can be easily built and tested.

To REST or not to REST

Mon, 20 Jul 2009 08:41:51 +0000

Yesterday @esjewett pointed out that tarpipe’s API 2.0 might not be considered RESTful. Quoting the previously written article “A RESTful ESME API”:

REST is often seen as an alternative to RPC APIs, or “Remote Procedure Call” APIs. At root, the difference as described by Wikipedia is that RPC is about telling an application to do something while REST is about changing the state of the resources of an application.

Paul Sadauskas, on another article about the same subject, adds some rules that your API should not follow:

Clients have to read documentation to know the locations of top-level resources.

Clients have to concatenate strings to get to the next resource.

You have an “API/Key/Token” in a header or a url.

You have a version string in a url.

All this made me think about possible changes to become RESTful. Things seem to be easy when you’re offering operations to resources you own or control. When you’re dealing with processes that’s a whole different story. As @esjewett best puts it:

Rest on Bruno Pedro

JavaScript filesystem database

To REST or not to REST