For a bit of history of my dealings with MongoDB at Urban Airship, I gave a couple versions of a Scaling with MongoDB talk:

• at Open Source Bridge (latest & greatest)
• at Update Portland (original, less polished version)

My coworker Adam Lowry even gave a follow-up talk of sorts at Postgres Open 2011 (slides) about migrating one of our datasets off of MongoDB and (back) on to PostgreSQL.

After reading through those slides you’re probably wondering why we’re still dealing with MongoDB at all. We fully intended to migrate our data out of it by now, but priorities change, deadlines slip, and we never expected one of our last uses of MongoDB to experience a surge in growth.

The dataset in question seemed ideal for MongoDB:

• Ephemeral – if we lose it we experience service degradation for a short while, but nothing catastrophic
• Small – easily fits into memory (~15 GB)
• Secondary index – In a key/value store we would have had to manage a secondary index manually

So this dataset dodged a lot of the previous problems we had with MongoDB and seemed safe to migrate at our leisure.

Global Write Lock

MongoDB has a global write lock. This means that while applying an insert or update, a single mongod instance can’t respond to other queries.

Our dataset may be small but it has a heavy read and write load. When the service it backed experienced a surge in usage, MongoDB quickly became CPU bound. This was especially frustrating considering mongod was running in a simple master/slave setup on two servers: each with 16 cores and enough memory to hold all the data a few times over again.

Because of the global write lock and heavy write load, operations are effectively serialized and executed on a single core. Meaning our servers didn’t even look loaded, as just 1 core would be 100% utilized by mongod.

Let the Sharding Begin

So we need to utilize multiple cores…
To do that we need multiple write locks…
There’s 1 write lock per mongod. So…
…multiple mongods per server?

We’d been avoiding sharding after having no luck getting it working in the 1.5.x dev series, but it’s our only choice now to get multiple mongods. I ran some tests and it seemed like we could turn our master/slave setup into a 2 shard setup with 2 mongods and 1 arbiter per shard with downtime in the seconds or low minutes.

The operational complexity of configuring a MongoDB cluster is daunting with each component bringing its own caveats:

mongod config servers

• You need exactly 3 config mongods (1 is fine for testing, which makes things appear simpler than they really are).
• There are lots of caveats with the config servers, so read Changing Config Servers carefully before configuring your cluster.
• Otherwise these mongod instances are fairly blackboxish to me. Despite being mongod processes you administer them completely differently.

mongos routers

• 1 per app server. This wouldn’t be a big deal except that our mongoses often start failing and require flushRouterConfig to be run on them. 2.0.1 supposedly fixes this, but we haven’t tested that yet (and trading known problems for new unknown ones is always scary).
• mongos instances can use a lot of CPU and seem to have random spikes where they fully utilize every core very briefly. Keep this in mind if your application servers are already CPU bound.
• On the bright side mongos balanced our data rather quickly. Our shard key is a uuid, and it properly setup reasonable ranges in very short order without having to preconfigure them.
• “mongos” is a terribly confusing name. It sounds like multiple mongo instances. We’ve taken to calling them mongooses internally due to frequent typos and confusion.

arbiters

• You need at least 3 members in a replica set in order to complete an election if 1 member goes down.
• We haven’t had any issues with arbiters… not sure what we’d do if one broke somehow but since they have no persistent data they’re safe to restart at any time.

mongods

• Early on we ran into a problem where changing replica set member entries wasn’t propagated to the config servers’ shard configuration. Restarting every mongos fixed it.
• As far as I can tell a new replica set member will never leave the initial RECOVERING state until all operations to that set are stopped. Even 40 updates per second was enough of a trickle to prevent a new set member from leaving RECOVERING to becoming a SECONDARY. We had to shutdown mongoses to cut off all traffic to bring up a new member. (The replication log gave every indication of being caught up and our usual update load is thousands per second.)
• Setting rest in the config file doesn’t seem to work. Put –rest in your command line options.
• Sending an HTTP request to a mongod’s main port (instead of the HTTP endpoint) seems to be capable of crashing the mongod.

Client Drivers

While a single replica set member was in a RECOVERING state our Java services couldn’t complete any operations while our Python service was happily working away.

Summary

Right now we’re getting by with 2 shards on 2 dedicated servers and then mongoses and config servers spread throughout other servers. There appears to be some data loss occurring, though due to the ephemeral fast changing nature of this dataset it’s very difficult to determine definitively or reproduce independently.

So we’re trying to migrate off of MongoDB to a custom service better suited for this dataset ASAP.

1. Narrative talks based on experiences in production (not how-tos)
2. Database-agnostic backend systems focused groups (not just a NoSQL meetup)

So I started one: Update Portland.

And I gave my promised MongoDB talk: schmongodb.

And 10gen sent swag! (Thanks to Meghan! It was a big hit.)

And my brilliant coworker Erik Onnen gave a short talk on how he’s beginning to use Kafka at Urban Airship. (Expect a long form talk on that in the future!)

Thanks to everyone who showed up. I had a great time and have high hopes for the upcoming meetings. (Sign up for the mailing list!)

The slides may come across as overly negative. After all Urban Airship is actively moving away from MongoDB for our largest and busiest pieces of data. So I want to make 2 things very clear:

1. I like MongoDB and would like to use it again in the future. There’s a lot I don’t like about it, but I can’t think of any “perfect” piece of software.
2. The IO situation in EC2, particularly EBS’s poor performance (RAIDing really doesn’t help) made life with MongoDB miserable. This story may have been very different if we were running MongoDB on bare metal with fast disks.

Mike Herrick, the VP of Engineering at Urban Airship, put me on the spot at the end of my talk by asking me by asking me: “Knowing what you know now, what would you have done differently?”

I didn’t have a good answer, and I still don’t. Despite all of the misadventures, MongoDB wasn’t the wrong choice. Scaling systems is just hard, and if you want something to work under load, you’re going to have to learn all of its ins and outs. We initially started moving to Cassandra, and while it has tons of wonderful attributes, we’re running into plenty of problems with it as well.

So I think the answer is knowing then what I know now. In other words: Do your homework. That way we could have avoided these shortcomings and perhaps still be happy with MongoDB today. Hopefully these slides will help others in how they plan to use MongoDB so they can use it properly and happily.

Note: I added lots of comments to the speaker notes, so you’ll probably want to view those while looking at the slides.

Traditionally with SQL databases pagination has looked something like this:

There are previous and next links as well as links for jumping right to the beginning and end. Pretty boring stuff.

What’s interesting is that this standard interface is disappearing in favor of something like this:

Twitter

Facebook

And soon beta testers of Urban Airship’s push service for Android will see a More link on the page that lists devices associated with their app:

The simplest possible explanation for this dumbing down of pagination is that count (for total pages) and skip/offset are expensive operations.

Not only are those operations expensive, but in eventually consistent databases, which many modern non-relational databases are, they’re extremely expensive, if not impossible, to perform.

Cassandra

At Urban Airship we, like Facebook, use Cassandra: a distributed column-based database. This deals two deadly blows to traditional pagination:

1. No way to count columns in a row (without reading every column).
2. No way to skip by numeric offset (so you can’t say, skip to page 5).

In Cassandra columns are ordered, so you start reading from the beginning and read N+1 columns where N is the number of items you’d like to display. The last column’s key is then used to determine whether the More link is enabled, and if so, what key to start the next “page” at.

Both of those are solvable problems if you really need them, but I would suspect you would end up creating a column count cache as well as some sort of table of contents for the various page offsets. Not what I want to spend my time implementing.

The fact of the matter is that for many use cases, a simple More button works just as well (if not better) than traditional pagination. It’s also far cheaper to implement, which means more developer time free to work on features and more hardware resources available to push your 140 character insights around the web.

MongoDB

I should note that MongoDB is fairly unique in the non-relational database world as its dynamic querying features include count and skip operations. However, as with any database, you’ll want to make sure these queries hit indexes.

Sadly MongoDB currently doesn’t have the distributed features necessary to automatically handle data too big for a single server.

Currently I don’t use many JavaScript functions, but when I do I like to store them on the server so that they’re accessible when I’m poking around in a console.

I use something similar to the following function to load all of my JS functions onto the server when my app starts:

import os
import pymongo
import pkg_resources
 
# Relative to distribution's root
SCRIPT_DIR = os.path.join('model', 'js')
 
def init_js(db):
    '''Initializes server-side javascript functions'''
    scripts = filter(
            lambda f: f.endswith('.js'),
            pkg_resources.resource_listdir(__name__, SCRIPT_DIR)
        )
    for script in scripts:
        # Name the function after the script name
        func_name, _ = script.split('.', 1)
        script_path = os.path.join(SCRIPT_DIR, script)
 
        # Create a pymongo Code object
        # otherwise it will be stored as a string
        code = pymongo.code.Code(
                pkg_resources.resource_string(__name__, script_path))
 
        # Upsert the function
        db.system.js.save({ '_id': func_name, 'value': code, })

However, using server-side functions from Python is awkward at best. Say I have the JavaScript function:

add.js

function(x, y) {
    return x + y;
}

To run that function via PyMongo requires wrapping the function call with placeholder parameters in a Code object and passing in values as a dict:

var1 = 1
var2 = 2
result = db.eval(pymongo.code.Code('add(a, b)', {'a': var1, 'b': var2,}))
assert result == 3

Update: See MongoDB dev Mike Dirolf comment to see a much more concise way of executing server-side functions.

Bearable for simple functions, but having to manually map parameters to values is tiresome and error prone with longer function signatures.

What I wanted was something more natural like:

var1 = 1
var2 = 2
result = db.add(var1, var2)
assert result == 3

I use a simple PyMongo Database object wrapper to make my life easier:

import string
 
from pymongo.code import Code
 
class ServerSideFunctions(object):
    def __init__(self, db):
        self.db = db
 
    def func_wrapper(self, func):
        '''Returns a closure for calling a server-side function.'''
        params = [] # To keep params ordered
        kwargs = {}
        def server_side_func(*args):
            '''Calls server side function with positional arguments.'''
            # Could be removed with better param generating logic
            if len(args) > len(string.letters):
                raise TypeError('%s() takes at most %d arguments (%d given)'
                        % (func, len(string.letters), len(args)))
 
            # Prepare arguments
            for k, v in zip(string.letters, args):
                kwargs[k] = v
                params.append(k) 
 
            # Prepare code object
            code = Code('%s(%s)' % (func, ', '.join(params)), kwargs)
 
            # Return result of server-side function
            return self.db.eval(code)
        return server_side_func
 
    def __getattr__(self, func):
        '''Return a closure for calling server-side function named `func`'''
        return self.func_wrapper(func)
 
dbjs = ServerSideFunctions('foo')
var1 = 1
var2 = 2
result = dbjs.add(var1, var2)
assert result == 3

I’m tempted to monkey-patch PyMongo’s Database class to add a ServerSideFunctions instance directly as a js attribute, so then I could drop the confusing dbjs variable and just use:

assert db.js.add(1,2) == 3

If someone knows of a better way to access server-side MongoDB functions from Python, please let me know!

I modified this code to remove code specific to my project, so please let me know if there are errors.