But first things first: I need to put data into shared memory from Python. mmap is an excellent widely-implemented POSIX system call for creating a shared memory space backed by an on-disk file.

Usually in the UNIX world you have 2 ways of accessing/manipulating data: memory addresses or streams (files). Manipulating data via memory addresses means pointers, offsets, malloc/free, etc. Stream interfaces manipulate data via read/write/seek system calls for files and send/recv/etc for sockets.

mmap gives you both interfaces. A memory mapped file can be manipulated via read/write/seek or by directly accessing its mapped memory region. The advantage of the latter is that this memory region is in userspace — meaning you can manipulate a file without incurring the overhead of write system calls for every manipulation.

Anyway, enough exposition, let’s see some code. (Despite mmap’s nice featureset, I’m only using it as a simple memory sharing mechanism anyway.) The following code shares a tiny bit of data between 2 Python processes using the excellent mmap module in the stdlib. a.py writes to the memory mapped region, and b.py reads the data out. ctypes allows for an easy way to create values in a memory mapped region and manipulate them like “normal” Python objects.

These code samples were written using Python 2.7 on Linux. They should work fine on any POSIX system, but Windows users will have to change the mmap calls to match the Windows API.

a.py

#!/usr/bin/env python
import ctypes
import mmap
import os
import struct
 
 
def main():
    # Create new empty file to back memory map on disk
    fd = os.open('/tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR)
 
    # Zero out the file to insure it's the right size
    assert os.write(fd, '\x00' * mmap.PAGESIZE) == mmap.PAGESIZE
 
    # Create the mmap instace with the following params:
    # fd: File descriptor which backs the mapping or -1 for anonymous mapping
    # length: Must in multiples of PAGESIZE (usually 4 KB)
    # flags: MAP_SHARED means other processes can share this mmap
    # prot: PROT_WRITE means this process can write to this mmap
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
 
    # Now create an int in the memory mapping
    i = ctypes.c_int.from_buffer(buf)
 
    # Set a value
    i.value = 10
 
    # And manipulate it for kicks
    i.value += 1
 
    assert i.value == 11
 
    # Before we create a new value, we need to find the offset of the next free
    # memory address within the mmap
    offset = struct.calcsize(i._type_)
 
    # The offset should be uninitialized ('\x00')
    assert buf[offset] == '\x00'
 
    # Now ceate a string containing 'foo' by first creating a c_char array
    s_type = ctypes.c_char * len('foo')
 
    # Now create the ctypes instance
    s = s_type.from_buffer(buf, offset)
 
    # And finally set it
    s.raw = 'foo'
 
    print 'First 10 bytes of memory mapping: %r' % buf[:10]
    raw_input('Now run b.py and press ENTER')
 
    print
    print 'Changing i'
    i.value *= i.value
 
    print 'Changing s'
    s.raw = 'bar'
 
    new_i = raw_input('Enter a new value for i: ')
    i.value = int(new_i)
 
 
if __name__ == '__main__':
    main()

b.py

import mmap
import os
import struct
import time
 
def main():
    # Open the file for reading
    fd = os.open('/tmp/mmaptest', os.O_RDONLY)
 
    # Memory map the file
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
 
    i = None
    s = None
 
    while 1:
        new_i, = struct.unpack('i', buf[:4])
        new_s, = struct.unpack('3s', buf[4:7])
 
        if i != new_i or s != new_s:
            print 'i: %s => %d' % (i, new_i)
            print 's: %s => %s' % (s, new_s)
            print 'Press Ctrl-C to exit'
            i = new_i
            s = new_s
 
        time.sleep(1)
 
 
if __name__ == '__main__':
    main()

(Note that I cruelly don’t clean up /tmp/mmaptest after the scripts finished. Consider it a 4KB tax for anyone who runs arbitrary code they found on the Internet without reading it first.)

I’m anxious to extend the API and add features, but I have no clue when I’ll have time to touch it again. In the mean time I’ve pushed tc-rest to bitbucket.org if you want to take a look.

Probably not, but I did: tc-rest.tar.gz

Components:

• TokyoCabinet – my favorite persistent key/value database
• pytc – a wonderful Python wrapper for TC
• fapws3 – a fast libev based HTTP/WSGI server
• simpleson – (or Python >= 2.6) for encapsulating HTTP responses
• okapi – a fantastic little static HTML page for testing HTTP APIs

Getting TokyoCabinet+pytc to work inside a virtualenv was a bit tricky, so check out my run.sh script if you’re having trouble getting it to start.

Once you get it started, load okapi in your browser:

http://localhost:8080/static/okapi.html

And then create a database by doing a POST like:

http://localhost:8080/foo/

And finally store/get keys and values using GET and POST requests like:

http://localhost:8080/foo/bar/
http://localhost:8080/foo/baz/

Doing a GET request to a database URL lists keys.

At any rate, I had big dreams for building a system where you would store JSON, specify indexes on certain keys, and the server would maintain those indexes for you by creating ad hoc TokyoCabinet databases.

Instead I ended up wasting most of my time learning how to write a low-level WSGI app. I should have just used CherryPy or Django from the beginning, but I had never written a pure WSGI app before. It was a good lesson even if it meant not getting some of my features implemented.

I’ll probably keep playing with this idea, but the next version will probably be based on some existing framework. Parsing environ['PATH_INFO'] and running start_response(...) manually gets old fast.

fapws3 is pretty neat, but had lots of annoying rough edges. I had to manually create a README file because its setup.py expects one to exist. Then I had to manually allow DELETE HTTP methods in fapws/base.py, otherwise it would return an HTML error message for me! That was a bit shocking since I was working under the assumption fapws3 is just a low-level HTTP/WSGI server.

Update

1. If you’re new to TokyoCabinet, I posted my presentation on it that I did at Portland Python meetup.
2. Someone want to benchmark this for me? Might be interesting since its made with the fastest libs available in Python for their respective tasks. I’m just feeling lazy at this point.

It was definitely just an introductory talk, so if you’re already familiar with these projects it probably won’t be very interesting.

• Slides (html)
• Single page (HTML version)
• Single page (PDF version)
• reStructuredText (rst) source

Errata: My slides state PyTyrant 1.1.11 doesn’t work with Tokyo Tyrant 1.1.17 — this is wrong. I don’t know what I was doing yesterday, but today all my tests work on Tokyo Tyrant 1.1.17.

Here’s my lightly modified version of Eric’s script:

import wsgiserver
import sys
import os
import django.core.handlers.wsgi
 
if __name__ == "__main__":
    # Setup paths - a bit hackish, but works for me.
    # Assumes an absolute path is stored in <project>.local_settings.ROOT
    sys.path.append(os.path.realpath(os.path.dirname(__file__)))
    from foo.local_settings import ROOT
    sys.path.append(ROOT)
 
    # Startup Django
    os.environ['DJANGO_SETTINGS_MODULE'] = 'foo.settings'
    server = wsgiserver.CherryPyWSGIServer(
        ('0.0.0.0', 8888),  # Use '127.0.0.1' to only bind to the localhost
        django.core.handlers.wsgi.WSGIHandler()
    )
    try:
        server.start()
    except KeyboardInterrupt:
        print 'Stopping'
        server.stop()

I also went with the latest stable version of CherryPy’s wsgiserver instead of checking out trunk like Eric’s post suggested.

Then I just enabled mod_proxy in Apache and setup the following VirtualHost:

<Proxy *>
    Order allow,deny
    Allow from all
</Proxy>
<Location "/">
    ProxyPass http://127.0.0.1:8888/
    ProxyPassReverse http://127.0.0.1:8888/
</Location>

If you’re cool you’ll write some sort of system specific script to launch your web app on boot. In a pinch, you can always use a crontab:

@reboot /usr/bin/python /path/to/app.py &

YMMV

* To mod_wsgi’s credit, it took about 10 seconds to compile, generated a Universal binary, and in general Just Worked.

** I’m already a CherryPy fan thanks to dowski, so it wasn’t a hard decision.

The Problem

A memory leak. Apache is eating up memory so quickly that I need to restart it every couple days or risk my entire server grinding to a halt as it starts swapping wildly. I’ve poured over log files and pmap output, but I still can’t figure out where the problem lies. Curse you monolithic in-process architecture!

Actually I know what my problem is, I’m running a mess of modules:

• ssl – 2 certificates on 2 ports
• php5 – blerg, who doesn’t have to run this?
• suphp – I suspect this is my problem, but I can’t prove it. A client’s 3rd party web application requires it, but I think its easily replaceable with FastCGI.
• wsgi – No complaints. Python apps are out-of-process thankfully.
• proxy – Again no complaints. Can’t imagine how this module could cause any problems except it does proxy some large (multi-megabyte, not huge) POSTs at times. I can’t imagine a memory leak could slip into this module without a lot of people noticing.

Solution A: Apache+FastCGI

I love the idea of putting each web application in its own process and letting Apache just act as an HTTP router. FastCGI seems to have all the features I need, and I’m not really worried about the CPU overhead incurred by IPC.

However, there are 2 competing FastCGI modules for Apache, and I have no idea what to choose. Anecdotally the official mod_fastcgi is buggy and fastcgi.com is a spam infested wasteland. However, I’ve found no authoritative source saying: “fastcgi is dead, long live fcgid!” (Lame excuse, I know.)

Solution B: Lighttpd

I know Lighty is the darling of Rails sites, but whenever I stop by its site I’m greeted with a list of recently fixed security bugs, and now it seems as though they’re rewriting the core!

I’m sure Lighty is a high quality intelligently engineered project, but it seems to be the definition of immature. Not necessarily bad (in fact it usually means its progressing quickly!), but perhaps not as reliable as good old workhorses like Apache.

Solution C: Cherokee

I’ve been following Cherokee for some time now and running it locally on my workstation. I love the web interface. I’m usually a very anti-webmin, pro-vim kind of guy, but I’m sick of editing Apache’s config files. I do it about once a month and therefore it always takes lots of double-checking the docs. I don’t know why, but its configuration has just never felt natural to me.

However, the lead Cherokee developer’s bravado is by the most off-putting aspect of the project. He mocks modwsgi and posts simplistic benchmarks showing Cherokee to be the fastest web server, but meanwhile Cherokee churns out numerous bug patch releases in-between feature releases and has yet to reach 1.0 status.

It seems like an excellent project technically, but I’m afraid there will be negative consequences for the lead developers hubris. (I’m not meaning to insult the guy. He’s probably a far better hacker than I’ll ever be. Self-promotion just makes me uncomfortable.)

Solution D: nginx

I don’t know much about nginx except that it works. Basically all I’ve heard about it is:

• It works.
• Its fast. Really fast.

While “working” is definitely my primary objective, nginx seems a bit bare bones for me. I just don’t think I’m the target demographic. I’d kind of like for my web server to handle spawning and kill of FastCGI processes.

nginx feels like git to me. Those who know it: use it and love it. Those who don’t: stand in fear and awe of its unbridled power.

…or maybe its just a nice simple barebones HTTP server…

Conclusions?

I think Solution A: FastCGI is the most sensible. Apache has always served me well, and the memory leak is most likely due to that shoddy suphp module.

Moving my web applications to FastCGI is also the best way to prepare to move to one of these 2nd generation web servers.

However, I’m getting kind of sick of Apache, and the ambiguousness of which FastCGI solution to choose is fairly annoying.

So dear lazyweb, for your everyday web developer consultant looking to run a bunch of PHP and Python web applications, what HTTP server stack should I use? (Debian Lenny packages are a plus.)