I should mention Jason Kirtland informed me after the meeting that FastCGI supports persistent connections (and a host of other features) between a load balancer and backend app servers.

So let’s hack nginx shall we?

I know a bit of C but am primarily a Python developer, so hacking an established C project doesn’t come easily to me. To make matters worse, as far as I can tell there’s no official public source repository (but there’s a mirror) and it seems to be mainly developed by the creator, Igor Sysoev. At least the code looks clean.

First Pass

I had nginx-0.8.54.tar.gz handy from compiling the source and nginx was nice enough to log an error for PUTs without Content-Length:

client sent PUT method without “Content-Length” header while reading client request headers, client: …, server: , request: “PUT / HTTP/1.1″ …

So let’s use ack to find it:

A quick vim +1532 src/http/ngx_http_request.c later and we’re looking at the problem:

    if (r->method & NGX_HTTP_PUT && r->headers_in.content_length_n == -1) {
        ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "client sent %V method without \"Content-Length\" header",
                  &r->method_name);
        ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED);
        return NGX_ERROR;
    }

This code returns the 411 Length Required response for PUTs lacking a Content-Length header. Remove it, recompile (make -j2 && sudo make install && sudo service nginx restart), and test:

$ curl -vk -X PUT -u '...:...' http://web-0/api/device_tokens/FE66489F304DC75B8D6E8200DFF8A456E8DAEACEC428B427E9518741C92C6660
* About to connect() to web-0 port 80 (#0)
* Trying 10.... connected
* Connected to web-0 (10....) port 80 (#0)
* Server auth using Basic with user '...'
> PUT /api/device_tokens/FE66489F304DC75B8D6E8200DFF8A456E8DAEACEC428B427E9518741C92C6660 HTTP/1.1
> Authorization: Basic ...
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: web-0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/0.8.54
< Date: Tue, 28 Dec 2010 23:06:54 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Authorization,Cookie,Accept-Encoding
<
* Connection #0 to host web-0 left intact
* Closing connection #0


Success! Now create a patch diff -ru nginx-0.8.54 nginx > fix-empty-put.patch and post it to the nginx-devel mailing list.

Now to play Minecraft for 12 hours as you wait for the Russian developers to wake up and take notice of your patch. Possibly sleep.

Second Pass: Fixing WebDAV

A positive reply from Maxim Dounin to my patch! I don't use WebDAV though, but if I want this patch accepted I better make sure it doesn't break official modules.

This time around I wanted to work locally, so I installed nginx with the following configuration:


./configure --prefix=/home/schmichael/local/nginx --with-debug --user=schmichael --group=schmichael --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_dav_module


Note that I set the prefix to a path in my home directory, turned on debugging and the dav module, and set nginx to run as my user and group. A quick symlink from /home/schmichael/local/nginx/sbin/nginx to ~/bin/nginx, and I can start and restart nginx quickly and easily. More importantly I can attach a debugger to it.

The importance of being able to attach a debugger became clear as soon as I tested dav support (with their standard config):

$ curl -v -X PUT http://localhost:8888/foo
* About to connect() to localhost port 8888 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8888 (#0)
> PUT /foo HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: localhost:8888
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection #0


My patch was causing a segfault in the dav module that killed nginx's worker process. Bumping up my error logging to debug level didn't give me many clues:

2010/12/28 10:32:55 [debug] 15548#0: *1 http put filename: "/home/schmichael/local/nginx/dav/foo"
2010/12/28 10:32:55 [notice] 15547#0: signal 17 (SIGCHLD) received
2010/12/28 10:32:55 [alert] 15547#0: worker process 15548 exited on signal 11


Time to break out the debugger! While I've used gdb --pid to attach to running processes before, I'd just installed Eclipse to work on some Java code and wondered if it might make debugging a bit easier.

After installing plugins for C/C++, Autotools, and GDB, I could easily import nginx by creating a "New Makefile project with existing code":

Now create a new Debug Configuration:

Note on Linux systems (at least Ubuntu): by default PTRACE is disabled in the kernel. Just flip the 1 to 0 in /etc/sysctl.d/10-ptrace.conf and run sudo sysctl -p /etc/sysctl.d/10-ptrace.conf to allow PTRACE.

Finally click "Debug" and select the nginx worker process from your process list:

By default GDB will pause the process it attaches to, so make sure to click the Resume button (or press F8) to allow nginx to continue serving requests.

Crashing nginx
Now cause the segfault by running our curl command curl -v -X PUT http://localhost:8888/foo. This time curl won't return because gdb/Eclipse caught the segfault in the nginx child process, leaving the socket to curl open. A quick peek in Eclipse shows us exactly where the segfault occurs:

Eclipse makes it quick and easy to interactively inspect the variables. Doing that I discovered the culprit was the src variable being uninitialized. Bouncing up the stack once you can see dav's put handler expects to be given a temporary file (&r->request_body->temp_file->file.name) full of PUT data (of which we sent none), and it copies that to the destination file (path).

Bounce up the stack again to ngx_http_read_client_request_body and you can see this relevant code:

if (r->headers_in.content_length_n < 0) {

nginx's core HTTP module short circuits a bit when there's no Content-Length specified. It skips the temp file creation because there's no data to put into the temp file!

So we have our problem:

1. The dav module put handler expects a temp file containing the data to be saved.
2. The http module doesn't create a temp file when there's no body data.

The 2 solutions I can think of are:

1. Always create a temp file, even if it's empty.
2. Add a special case to the dav module's put handler for when the temp file doesn't exist.

I really don't want to hack the core http module just to make a sub-module happy. It makes sense that no temporary file exists when there's no body data. Sub-modules shouldn't be lazy and expect it to exist. So I decided to try #2.

The Fix

You can see my implementation of solution #2 on GitHub. Simply put, if the temp file exists, follow the existing logic. If the temp file does not exist we have a PUT with an empty body: use nginx's open wrapper to do a create or truncate (O_CREAT|O_TRUNC) on the destination file (since an empty PUT should create an empty file).

I don't know if this is the best solution or even a correct one, but it appears to work and was a fun journey arriving at it. You can follow the discussion on the mailing list.

Updated to switch from bitbucket to github.

The full list of sessions is up (or here), and I’m very excited just to be attending the conference. Congratulations to all of the speakers!

Some of the sessions I’m particularly excited to attend are:

• An Introduction to Machine Learning by John Melesky – A Portland Python User Group regular and really smart guy in general. Looking forward to getting my feet wet with machine learning.
• Drop ACID and think about data by Bob Ippolito – I think the Portland Python crew that went to Pycon helped convince Bob to bring his presentation to OS Bridge, and I’m glad they did! I work with non-relational databases everyday, but I still design my apps with a relational data model in mind. This should be interesting.
• The Linux Kernel Development model by Greg Kroah-Hartman – Its Greg KH! Come on!
• How Idealist.org uses technology to change the world by Michel Pelletier – Another Portland Pythoneer and really smart guy. Eager to here more about open source at work in an organization. Its always fun to watch Michel present anyway.
• Web Testing with Windmill by Mikeal Rogers – Hardly a day goes by that I don’t tell myself I’m going to improve my test coverage and skills. Windmill looks like an excellent testing tool to help me accomplish that.

There are many many more sessions I’m excited about, but I’m too sick of copying and pasting to mention anymore right now.

So sign up to attend the Open Source Bridge Conference in Portland, OR this June!

The Problem

A memory leak. Apache is eating up memory so quickly that I need to restart it every couple days or risk my entire server grinding to a halt as it starts swapping wildly. I’ve poured over log files and pmap output, but I still can’t figure out where the problem lies. Curse you monolithic in-process architecture!

Actually I know what my problem is, I’m running a mess of modules:

• ssl – 2 certificates on 2 ports
• php5 – blerg, who doesn’t have to run this?
• suphp – I suspect this is my problem, but I can’t prove it. A client’s 3rd party web application requires it, but I think its easily replaceable with FastCGI.
• wsgi – No complaints. Python apps are out-of-process thankfully.
• proxy – Again no complaints. Can’t imagine how this module could cause any problems except it does proxy some large (multi-megabyte, not huge) POSTs at times. I can’t imagine a memory leak could slip into this module without a lot of people noticing.

Solution A: Apache+FastCGI

I love the idea of putting each web application in its own process and letting Apache just act as an HTTP router. FastCGI seems to have all the features I need, and I’m not really worried about the CPU overhead incurred by IPC.

However, there are 2 competing FastCGI modules for Apache, and I have no idea what to choose. Anecdotally the official mod_fastcgi is buggy and fastcgi.com is a spam infested wasteland. However, I’ve found no authoritative source saying: “fastcgi is dead, long live fcgid!” (Lame excuse, I know.)

Solution B: Lighttpd

I know Lighty is the darling of Rails sites, but whenever I stop by its site I’m greeted with a list of recently fixed security bugs, and now it seems as though they’re rewriting the core!

I’m sure Lighty is a high quality intelligently engineered project, but it seems to be the definition of immature. Not necessarily bad (in fact it usually means its progressing quickly!), but perhaps not as reliable as good old workhorses like Apache.

Solution C: Cherokee

I’ve been following Cherokee for some time now and running it locally on my workstation. I love the web interface. I’m usually a very anti-webmin, pro-vim kind of guy, but I’m sick of editing Apache’s config files. I do it about once a month and therefore it always takes lots of double-checking the docs. I don’t know why, but its configuration has just never felt natural to me.

However, the lead Cherokee developer’s bravado is by the most off-putting aspect of the project. He mocks modwsgi and posts simplistic benchmarks showing Cherokee to be the fastest web server, but meanwhile Cherokee churns out numerous bug patch releases in-between feature releases and has yet to reach 1.0 status.

It seems like an excellent project technically, but I’m afraid there will be negative consequences for the lead developers hubris. (I’m not meaning to insult the guy. He’s probably a far better hacker than I’ll ever be. Self-promotion just makes me uncomfortable.)

Solution D: nginx

I don’t know much about nginx except that it works. Basically all I’ve heard about it is:

• It works.
• Its fast. Really fast.

While “working” is definitely my primary objective, nginx seems a bit bare bones for me. I just don’t think I’m the target demographic. I’d kind of like for my web server to handle spawning and kill of FastCGI processes.

nginx feels like git to me. Those who know it: use it and love it. Those who don’t: stand in fear and awe of its unbridled power.

…or maybe its just a nice simple barebones HTTP server…

Conclusions?

I think Solution A: FastCGI is the most sensible. Apache has always served me well, and the memory leak is most likely due to that shoddy suphp module.

Moving my web applications to FastCGI is also the best way to prepare to move to one of these 2nd generation web servers.

However, I’m getting kind of sick of Apache, and the ambiguousness of which FastCGI solution to choose is fairly annoying.

So dear lazyweb, for your everyday web developer consultant looking to run a bunch of PHP and Python web applications, what HTTP server stack should I use? (Debian Lenny packages are a plus.)