MongoDB的nodejs 键值压缩driver

这是E叔同事tzlastom 所写的nodejs的一个mongodb driver,征得他同意后E叔转到咱中文站了。

原文地址:https://www.npmjs.com/package/mongodbx

A wrapper of Node.js mongoDB driver with document key compression

var mongodb = require(‘mongodbx’);
mongodb.mongodbx.initialize({
    ‘collections’: {
        ‘mongodbxTest’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: true,
        }
    }
});
 
var record = {‘name’: ‘name’, ‘expireTime’: new Date(), ‘deleteFlag’: true, ‘notConfiguredKey’: ‘ext’};
var db = new mongodb.Db(‘test’, new mongodb.Server(‘localhost’, 27017));
db.open(function(err, db){
    var collection = db.collection(“mongodbxTest”);
    collection.insert(record, {w:1}, function(){
        collection.find().toArray(console.log);
    });
});
[tong@localhost mongodbx]$ node test.js
null [{_id: 551b6c63ee8ff65407e59847, name: 'name', expireTime: Wed Apr 01 2015 11:56:19 GMT+0800 (SGT), deleteFlag: true, notConfiguredKey: 'ext'}]
 
rs0:PRIMARY> db.mongodbxTest.find()
{“_id”: ObjectId(“551b6c63ee8ff65407e59847″), “n”: “name”, “e”: ISODate(“2015-04-01T03:56:19.625Z”), “d”: true, “notConfiguredKey”: “ext”}

As a key-value database, one fault of mongodb is to store the key string for each document. This will cost a lot of IO resources. For example, a collection with column named ‘deleteFlag’, mongodb need 11 bytes for key string storage –The value itself(true/false) only need 1 byte!

There do have a ‘best pratice’ for this: shorten the key string. For ‘deleteFlag’, design/code with ‘fdel’, even ‘d’. However, this solution will cause some software nightmare. The code’s maintainability && readability will be compromised.

To optimise mongodb’s storage && be friendly to code/design, we can put a middleware between our application and mongodb driver. The middleware will translate document bi-direction. For example, {deleteFlag: true}(application model) <=> {d: true}(mongodb document).

  • Pass all test cases in mongodb driver(driver version: 2.0.25, server version: 3.0.1).
  • Performance: about 250ms to compress/decompress 10k documents(Core i7 3.40GHZ).
  • params.collections (Object)
    • params.collections[].enableCompact (Boolean, default: params.debug). collection level compress config.
    • params.collections[].columns (Array/Object).
      • Object: translate map;
      • Array: using base62(a-z, A-Z, 0-9) to compress, the compressed value is key’s index in columns array. Make sure this array is append only.
  • params.enableCompact (Boolean, default:false). Global compress config.
  • params.debug (Boolean, default:false).
  • params.mongodb mongodb driver object(require(‘mongodb’)). For those don’t want embedded mongodb driver.
mongodb.mongodbx.initialize({
    ‘collections’: {
        ‘mongodbxTest’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: true,
        },
        ‘mongodbxTestBase62Map’: {
            columns: ['name', 'expireTime', 'deleteFlag'],
            enableCompact: true,
        },
        ‘noCompactCol’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: false,
        },
        ‘dependOnBlockConfig’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
        },
    },
    enableCompact: true
});
 
#Specify you mongodb driver
var mongodb = require(‘mongodb’);
require(‘mongodbx’).initialize({
    collections: …
    mongodb: mongodb,
});
mongodb.mongodbx.[Api]
});
  • params. see mongodbx.initialize params.collections[]
console.log(mongodb.mongodbx.getColumnNameMap(‘mongodbxTest’));

Will print:

{ name: ‘n’, expireTime: ‘e’, deleteFlag: ‘d’ }
console.log(‘Map translate normal:’, mongodb.mongodbx.translateToCompact(‘name’, ‘mongodbxTest’));
console.log(‘Map translate notConfiguredKey:’, mongodb.mongodbx.translateToCompact(‘notConfiguredKey’, ‘mongodbxTest’));
 
console.log(‘Base62 translate 1st Key:’, mongodb.mongodbx.translateToCompact(‘name’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 2nd Key:’, mongodb.mongodbx.translateToCompact(‘expireTime’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 3rd Key:’, mongodb.mongodbx.translateToCompact(‘deleteFlag’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate notConfiguredKey:’, mongodb.mongodbx.translateToCompact(‘notConfiguredKey’, ‘mongodbxTestBase62Map’));

Will print:

Map translate normal: n
Map translate notConfiguredKey: notConfiguredKey
Base62 translate 1st Key: a
Base62 translate 2nd Key: b
Base62 translate 3rd Key: c
Base62 translate notConfiguredKey: notConfiguredKey
console.log(‘Map translate normal:’, mongodb.mongodbx.translateToOriginal(‘n’, ‘mongodbxTest’));
console.log(‘Map translate notConfiguredKey:’, mongodb.mongodbx.translateToOriginal(‘notConfiguredKey’, ‘mongodbxTest’));
 
console.log(‘Base62 translate 1st Key:’, mongodb.mongodbx.translateToOriginal(‘a’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 2nd Key:’, mongodb.mongodbx.translateToOriginal(‘b’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 3rd Key:’, mongodb.mongodbx.translateToOriginal(‘c’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate notConfiguredKey:’, mongodb.mongodbx.translateToOriginal(‘notConfiguredKey’, ‘mongodbxTestBase62Map’));

Will print

Map translate normal: name
Map translate notConfiguredKey: notConfiguredKey
Base62 translate 1st Key: name
Base62 translate 2nd Key: expireTime
Base62 translate 3rd Key: deleteFlag
Base62 translate notConfiguredKey: notConfiguredKey
  • mongodb.Collection.mapReduce not supported
  • mongodb.Collection.group not supported
  • mongodb.Collection.find $where not supported
  • embeded-document’s key compress not supported

With mongodbx, application can get tranlated record. But if we use mongodb shell, the output is still less-readability. Try to solve this…

发表评论