JavaScript filesystem database

I couldn’t resist and, after many conversations with Mário Valente, I challenged myself to build a complete JavaScript filesystem based database from scratch. In short, something like CouchDB but written in JavaScript and using a filesystem approach for document storage.

Some characteristics of this new database system:

  • fully written in JavaScript: nodeJS is my choice for its architecture and portability;
  • every document must contain only JSON data;
  • every document can be manipulated on the filesystem: the idea is to allow someone to edit a document using vim, for instance, without breaking anything. It also lets you easily backup, move or replicate your data without creating a heavy load on the database;
  • the underlying filesystem can be changed: changing the underlying filesystem offers numerous possibilities, like easy distribution (using OpenAFSXTREEMFS or others), replication (using finefs or others), and even attaching different backends (using FUSE, for example);
  • the system must be available through an HTTP REST interface: this makes it very easy to immediately integrate the system into any existing applications. Bonus points if the API is compatible with CouchDB;
  • queries must be performed using a MapReduce approach: this should make it very easy to perform queries on a large dataset.

So, right now I’m half way through it. I managed to write the backend and I’m on my way to the HTTP REST interface. I’ll create a github repo after I have something functional that can be easily built and tested.

What do you think of a system like this? Any features you’d like to add or change?

Advertisements

13 thoughts on “JavaScript filesystem database”

    1. Hi Pedro,

      I read your post! Great stuff, by the way.

      The goals are completely different, as you say, but some work can be merged, I guess. I'll update all my progress in this blog.

  1. It seems that my project is similar to yours. Check it out: https://github.com/finalclass/finaldb
    It is using a database as a storage. One document is one file.
    You can map the documents and you can store milions of docs with no big impact of performance.
    Now I’m thinking about creating the db in nodejs that would store everything in one file. This one file would be a one great javascript object (the concept similar to firebase). It will be easier to copy (backup/sync) this kind of database since copying millions of files is very slow.

    1. Hi Szymon,

      Interesting, but performance is very poor. I found that to be the biggest drawback of this approach: writing thousands of files to disk is not very good in terms of performance, is it?

      1. yes, unfortunately dealing with many files is slow. In final-db I store every document in 2 sub-directories so that getting a file is not that slow, but still file-system based databases aren’t fast.

  2. (imho) I think that fs-db have do with a simple ways, lightweights, got some simple api to fetch and store the json data.. for more biggest things, there were many options databases to use..

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s