I’ve started toying with a convenience wrapper class for mmap.mmap (at least the Unix version):

My original goal was to automatically grow the mmap whenever the user attempts to write beyond the current size of the mmap file, but that’s going to take carefully wrapping quite a few methods (write, __setitem__, and maybe get/read methods too).

If it becomes useful, I may use it in mmstats.

Feedback welcome!

Update: Discovered the hard way (segfaults) that resizing mmaps is tricky: the region can be moved but data will be copied. However, any existing pointers (from ctypes..from_buffer in my case) will now point to freed memory and segfault upon use.

tl;dr – If at all possible, precompute the size of your mmap before using it.

But first things first: I need to put data into shared memory from Python. mmap is an excellent widely-implemented POSIX system call for creating a shared memory space backed by an on-disk file.

Usually in the UNIX world you have 2 ways of accessing/manipulating data: memory addresses or streams (files). Manipulating data via memory addresses means pointers, offsets, malloc/free, etc. Stream interfaces manipulate data via read/write/seek system calls for files and send/recv/etc for sockets.

mmap gives you both interfaces. A memory mapped file can be manipulated via read/write/seek or by directly accessing its mapped memory region. The advantage of the latter is that this memory region is in userspace — meaning you can manipulate a file without incurring the overhead of write system calls for every manipulation.

Anyway, enough exposition, let’s see some code. (Despite mmap’s nice featureset, I’m only using it as a simple memory sharing mechanism anyway.) The following code shares a tiny bit of data between 2 Python processes using the excellent mmap module in the stdlib. a.py writes to the memory mapped region, and b.py reads the data out. ctypes allows for an easy way to create values in a memory mapped region and manipulate them like “normal” Python objects.

These code samples were written using Python 2.7 on Linux. They should work fine on any POSIX system, but Windows users will have to change the mmap calls to match the Windows API.

a.py

#!/usr/bin/env python
import ctypes
import mmap
import os
import struct
 
 
def main():
    # Create new empty file to back memory map on disk
    fd = os.open('/tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR)
 
    # Zero out the file to insure it's the right size
    assert os.write(fd, '\x00' * mmap.PAGESIZE) == mmap.PAGESIZE
 
    # Create the mmap instace with the following params:
    # fd: File descriptor which backs the mapping or -1 for anonymous mapping
    # length: Must in multiples of PAGESIZE (usually 4 KB)
    # flags: MAP_SHARED means other processes can share this mmap
    # prot: PROT_WRITE means this process can write to this mmap
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
 
    # Now create an int in the memory mapping
    i = ctypes.c_int.from_buffer(buf)
 
    # Set a value
    i.value = 10
 
    # And manipulate it for kicks
    i.value += 1
 
    assert i.value == 11
 
    # Before we create a new value, we need to find the offset of the next free
    # memory address within the mmap
    offset = struct.calcsize(i._type_)
 
    # The offset should be uninitialized ('\x00')
    assert buf[offset] == '\x00'
 
    # Now ceate a string containing 'foo' by first creating a c_char array
    s_type = ctypes.c_char * len('foo')
 
    # Now create the ctypes instance
    s = s_type.from_buffer(buf, offset)
 
    # And finally set it
    s.raw = 'foo'
 
    print 'First 10 bytes of memory mapping: %r' % buf[:10]
    raw_input('Now run b.py and press ENTER')
 
    print
    print 'Changing i'
    i.value *= i.value
 
    print 'Changing s'
    s.raw = 'bar'
 
    new_i = raw_input('Enter a new value for i: ')
    i.value = int(new_i)
 
 
if __name__ == '__main__':
    main()

b.py

import mmap
import os
import struct
import time
 
def main():
    # Open the file for reading
    fd = os.open('/tmp/mmaptest', os.O_RDONLY)
 
    # Memory map the file
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
 
    i = None
    s = None
 
    while 1:
        new_i, = struct.unpack('i', buf[:4])
        new_s, = struct.unpack('3s', buf[4:7])
 
        if i != new_i or s != new_s:
            print 'i: %s => %d' % (i, new_i)
            print 's: %s => %s' % (s, new_s)
            print 'Press Ctrl-C to exit'
            i = new_i
            s = new_s
 
        time.sleep(1)
 
 
if __name__ == '__main__':
    main()

(Note that I cruelly don’t clean up /tmp/mmaptest after the scripts finished. Consider it a 4KB tax for anyone who runs arbitrary code they found on the Internet without reading it first.)

Writing asynchronous IO code is fun; handling signals is not. signalfd allows you to move your signal handling code into your main event loop instead of hooking up global handlers and using the featureless set_wakeup_fd function to break the main loop.

Luckily Jean-Paul Calderone had already created a great Python wrapper for the signalfd and sigprocmask system calls. Unfortunately it doesn’t include a way to parse the siginfo_t structure which contains all the useful information about the signal you’re handling.

I’ve added a helper to do just that in a branch:

https://code.launchpad.net/~schmichael/python-signalfd/helpers

A sample program would look like:

 
import os
import select
import signal
 
import signalfd
 
 
def sigfdtest():
    # Catch them all!
    sigs = []
    for attr in dir(signal):
        if attr.startswith('SIG') and not attr.startswith('SIG_'):
            sigs.append(getattr(signal, attr))
 
    sfd = signalfd.create_signalfd(sigs)
    print 'Capturing: %r' % sorted(sigs)
 
    while 1:
        print 'selecting - pid: %d' % os.getpid()
        r = select.select([sfd], [], [])[0]
        for s in r:
            assert s is sfd, 'Python nicely re-uses the fd instance'
            sig = signalfd.read_signalfd(sfd)
            print sig
 
 
if __name__ == '__main__':
    sigfdtest()

When run you can throw some signals at it:

Capturing: [6, 14, 7, 17, 17, 18, 8, 1, 4, 2, 29, 6, 9, 13, 29, 27, 30, 3, 64, 34, 11, 19, 31, 15, 5, 20, 21, 22, 23, 10, 12, 26, 28, 24, 25]
selecting - pid: 6523
^CSIGINT
selecting - pid: 6523
^CSIGINT
selecting - pid: 6523
SIGHUP
selecting - pid: 6523
Killed

Of course you’ll need to use an uninterpretable signal like KILL to exit.

Jean-Paul attempted to get signalfd included in Python 2.7′s signal module, and it was slated for inclusion in 3.2. However, given that 3.2 was just released without it, I’m guessing the attempt to get this functionality into Python’s stdlib has been forgotten.

Up next: eventfd perhaps?

I should mention Jason Kirtland informed me after the meeting that FastCGI supports persistent connections (and a host of other features) between a load balancer and backend app servers.

Currently I don’t use many JavaScript functions, but when I do I like to store them on the server so that they’re accessible when I’m poking around in a console.

I use something similar to the following function to load all of my JS functions onto the server when my app starts:

import os
import pymongo
import pkg_resources
 
# Relative to distribution's root
SCRIPT_DIR = os.path.join('model', 'js')
 
def init_js(db):
    '''Initializes server-side javascript functions'''
    scripts = filter(
            lambda f: f.endswith('.js'),
            pkg_resources.resource_listdir(__name__, SCRIPT_DIR)
        )
    for script in scripts:
        # Name the function after the script name
        func_name, _ = script.split('.', 1)
        script_path = os.path.join(SCRIPT_DIR, script)
 
        # Create a pymongo Code object
        # otherwise it will be stored as a string
        code = pymongo.code.Code(
                pkg_resources.resource_string(__name__, script_path))
 
        # Upsert the function
        db.system.js.save({ '_id': func_name, 'value': code, })

However, using server-side functions from Python is awkward at best. Say I have the JavaScript function:

add.js

function(x, y) {
    return x + y;
}

To run that function via PyMongo requires wrapping the function call with placeholder parameters in a Code object and passing in values as a dict:

var1 = 1
var2 = 2
result = db.eval(pymongo.code.Code('add(a, b)', {'a': var1, 'b': var2,}))
assert result == 3

Update: See MongoDB dev Mike Dirolf comment to see a much more concise way of executing server-side functions.

Bearable for simple functions, but having to manually map parameters to values is tiresome and error prone with longer function signatures.

What I wanted was something more natural like:

var1 = 1
var2 = 2
result = db.add(var1, var2)
assert result == 3

I use a simple PyMongo Database object wrapper to make my life easier:

import string
 
from pymongo.code import Code
 
class ServerSideFunctions(object):
    def __init__(self, db):
        self.db = db
 
    def func_wrapper(self, func):
        '''Returns a closure for calling a server-side function.'''
        params = [] # To keep params ordered
        kwargs = {}
        def server_side_func(*args):
            '''Calls server side function with positional arguments.'''
            # Could be removed with better param generating logic
            if len(args) > len(string.letters):
                raise TypeError('%s() takes at most %d arguments (%d given)'
                        % (func, len(string.letters), len(args)))
 
            # Prepare arguments
            for k, v in zip(string.letters, args):
                kwargs[k] = v
                params.append(k) 
 
            # Prepare code object
            code = Code('%s(%s)' % (func, ', '.join(params)), kwargs)
 
            # Return result of server-side function
            return self.db.eval(code)
        return server_side_func
 
    def __getattr__(self, func):
        '''Return a closure for calling server-side function named `func`'''
        return self.func_wrapper(func)
 
dbjs = ServerSideFunctions('foo')
var1 = 1
var2 = 2
result = dbjs.add(var1, var2)
assert result == 3

I’m tempted to monkey-patch PyMongo’s Database class to add a ServerSideFunctions instance directly as a js attribute, so then I could drop the confusing dbjs variable and just use:

assert db.js.add(1,2) == 3

If someone knows of a better way to access server-side MongoDB functions from Python, please let me know!

I modified this code to remove code specific to my project, so please let me know if there are errors.

Here are the results of my little test script lxml-v-etree.py (times are in milliseconds):

name           generate | tostring | total | write | parse | find | total
------------------------+----------+-------+-------+-------+------+------
xml.cElementTree    132 |   2430   |  2562 |  2433 |   158 |   58 |   216
xml.cElementTree    112 |   2384   |  2497 |  2387 |   158 |   25 |   183
xml.cElementTree    113 |   2393   |  2507 |  2396 |   161 |   25 |   187
xml.ElementTree     591 |   2571   |  3163 |  2574 |  3613 |   25 |  3638
xml.ElementTree     619 |   2567   |  3187 |  2570 |  3589 |   55 |  3644
xml.ElementTree     609 |   2578   |  3188 |  2581 |  3564 |   55 |  3619
lxml                333 |     75   |   409 |    82 |   200 |    0 |   201
lxml                355 |     93   |   448 |    95 |   182 |   32 |   214
lxml                310 |     94   |   404 |    96 |   156 |   56 |   213
------------------------+----------+-------+-------+-------+------+------
name           generate | tostring | total | write | parse | find | total
------------------------+----------+-------+-------+-------+------+------

Note that the first “total” is “generate + tostring” while the second “total” is for the 2 parsing related tests (previous 2 columns summed).

My parsing tests are basically “etree.parse” and then running “Element.getchildren()” 3 times, which is ridiculously simplistic and should probably be ignored. My writing tests are far more thorough/realistic.

I’m running Python 2.6.2 with lxml 2.1.5 and libxml2 2.6.32 on Ubuntu 9.04 x86_64.