There’s a whole nosql movement these days looking to replace relational database management systems, with “something else,” where that something else can be anything from key-value stores to document databases to flat files and map-reduce frameworks, or column-oriented databases.
Here on the Geeknet engineering team, we’ve got a lot going on with our sites SourceForge.net, Slashdot, freshmeat, and fossfor.us. We’ve been using relational databases Postgres and MySQL for a variety of things for a long time. But we have various needs where the new non-relational datastores make a lot of sense, so we’ve been investigating a number of these nosql systems.
MongoDB at SourceForge
Earlier this year I was part of a rewrite of the project pages on SourceForge.net. We wanted a document store that would allow us to save all the data we needed for a project page render in a single record. Our tests lead us to MongoDB, which is fast and has simple, easy to use replication. Mongo has allowed us to serve all of the project pages on SourceForge.net without any page level caching.
I first posted about our use of MongoDB on my personal blog about six months ago. Now I’m happy to announce that we’ve re-factored out the bits of our MongoDB helper infrastructure that are generally reusable into a separate Python package called Ming. It’s the code that runs SourceForge.net’s most popular pages, and which we are using to build several interesting new projects.
Ming rules Mongo
If you’re developing Python applications where a document style database makes sense, Ming provides a clear way to interact with MongoDB. It allows you to attach to multiple MongoDB sessions, do work in each, which makes working with replicated databases easier. It helps with data migrations, and can help you to enforce some schema on your Mongo documents so that you can make assumptions about document structures in your code.
We’re very happy with it, and it’s pretty well tested, both in terms of unit testing, and in terms of use on high volume, high traffic sites.
Ming borrows a lot of ideas from SQLAlchemy , and we think that Ming and SQLAlchemy are two great tastes that taste great together.
Some data fits the document-oriented view of the world really well, and at other times it makes a lot of sense to store data in an RDBMS where you get all the data integrity and transaction processing help that an RDBMS provides.
More to come…
We’ve never stopped providing free tools to open source projects, promoting open source, or working with open source tools. Nearly all of the engineers here contribute to various open source projects. Our individual commitment to various communities has always been high. But we haven’t been very organized about it, and that’s something we’re trying to change. Among other things we’ve started actively looking for internal stuff that we can open up for the rest of the community. Ming is our first release.