Traditionally with SQL databases pagination has looked something like this:

There are previous and next links as well as links for jumping right to the beginning and end. Pretty boring stuff.

What’s interesting is that this standard interface is disappearing in favor of something like this:

Twitter

Facebook

And soon beta testers of Urban Airship’s push service for Android will see a More link on the page that lists devices associated with their app:

The simplest possible explanation for this dumbing down of pagination is that count (for total pages) and skip/offset are expensive operations.

Not only are those operations expensive, but in eventually consistent databases, which many modern non-relational databases are, they’re extremely expensive, if not impossible, to perform.

Cassandra

At Urban Airship we, like Facebook, use Cassandra: a distributed column-based database. This deals two deadly blows to traditional pagination:

1. No way to count columns in a row (without reading every column).
2. No way to skip by numeric offset (so you can’t say, skip to page 5).

In Cassandra columns are ordered, so you start reading from the beginning and read N+1 columns where N is the number of items you’d like to display. The last column’s key is then used to determine whether the More link is enabled, and if so, what key to start the next “page” at.

Both of those are solvable problems if you really need them, but I would suspect you would end up creating a column count cache as well as some sort of table of contents for the various page offsets. Not what I want to spend my time implementing.

The fact of the matter is that for many use cases, a simple More button works just as well (if not better) than traditional pagination. It’s also far cheaper to implement, which means more developer time free to work on features and more hardware resources available to push your 140 character insights around the web.

MongoDB

I should note that MongoDB is fairly unique in the non-relational database world as its dynamic querying features include count and skip operations. However, as with any database, you’ll want to make sure these queries hit indexes.

Sadly MongoDB currently doesn’t have the distributed features necessary to automatically handle data too big for a single server.

Luckily mxDateTime and SQLObject’s DateTimeCol both support full dates with times and time zone. Unfortunately its not immediately obvious from SQLObject’s lackluster documentation how to use mxDateTime instead of Python’s built-in datetime.

A little searching brought me to a mailing list post about how to use mxDateTime by default in SQLObject. (I don’t know why the sample code includes the conditional as I would think you’d want your code to outright fail if you’re unable to use the datetime library you expect.)

So my model’s code looks something like this:

from sqlobject import *
from sqlobject import col
 
col.default_datetime_implementation = MXDATETIME_IMPLEMENTATION
 
class Foo(SQLObject):
    timestamp = DateTimeCol(default=DateTimeCol.now)

Then my parsing code looks something like this:

import model
from mx import DateTime
 
timestamp = '2007-07-31T16:05:00.000-05:00'
bar = model.Foo(timestamp=DateTime.DateTimeFrom(timestamp))
print 'UTC Timestamp:', bar.timestamp
print 'Local Timestamp:', bar.timestamp.localtime()

Basically once you use the magic line col.default_datetime_implementation = MXDATETIME_IMPLEMENTATION, everything Just Works.

• Full text search is now a built-in feature
• Support for the SQL/XML standard, including a new xml builtin data type (For the enterprisy users)
• enum data types (I’m not a big fan of enums in a database, but lots of people like them.)
• UUID data type, similar to that defined by RFC 4122. (Very useful feature in some clustering situations. Some people even argue GUIDs/UUIDs should replace integer surrogate keys most of the time.)
• ORDER BY … NULLS FIRST/LAST (A minor little feature, but I remember wishing for it once upon a time.)
• And some excellent performance improvements, especially if you’re data doesn’t have to be flushed to disk to finish a transaction (potentially dangerous, but acceptable for many use cases).

While I haven’t had the pleasure of using PostgreSQL for a few months, I prefer it over MySQL and recommend it to anyone looking for an open source database with advanced features and excellent reliability.

drupal-db-maint.sh (I posted a copy to pastebin since WordPress makes the code impossible to read.)

Then just run crontab -e to edit your crontab and add a line similar to:
46 4 * * * /path/to/drupal-db-maint.sh

That will optimize your database nightly at 4:46 AM server time (usually UTC).

If you don’t like my solution, there are plenty of more elegant session cleaning solutions out there. I’m just more comfortable writing shell scripts and crontabs than Drupal modules that use hook_cron.

Posting code in WordPress so that its readable is pretty much impossible. Any suggestions?

Go download it using BitTorrent to be cool.

Tip #1:  You do not have to download all of the CDs, just get one and Debian’s installer will download the rest.)

I’m a happy and loyal Debian user and strongly encourage any Linux power users to try out Etch. Ubuntu is still my recommendation for users who want things to “Just Work.” However, I highly recommend Debian for any kind of server (file, print, application, database, web, e-mail).

Tip #2: Check out Debian Multimedia for all of your proprietary codec needs!

Mini-Review

I’ve been using Etch (and Sid) for a while now, so its tough to do a real review from a “normal” end user’s perspective.

Some highlights:

• Developer’s should be happy as PHP, Python, Mono, MySQL, and PostgreSQL are all fairly up-to-date and extremely easy to install.
• Gnome is stable and speedy.
• The graphical installer brings Debian out of the 1990s and offers tons of features.
• As usual, Debian’s package repository is huge. Forget vendor-supplied RPMs or custom installers, if its not in Debian’s repository (or Debian Multimedia), you probably don’t need it.

Some downsides:

• Relatively ancient version of the standard C library (2.3.6) means poor compatibility with 3rd party binaries (which Debian has always hated anyway).
• Gnome is 2 stable versions behind in Debian (2.14 vs. 2.18). This is very frustrating as Gnome constantly produces nice incremental improvements.
• Debian just isn’t as easy to use by default as Ubuntu. Debian needs to install and integrate some of the more cutting edge desktop apps like Beagle, Tomboy, Deskbar, and Ubuntu’s nicer package management interfaces.
• Secure APT is a great feature, but the error messages it can produce are going to scare the pants off of new users. More effort needs to be made to integrate all of the various places user’s encounter public key crypto. Right now its very schizophrenic and confusing: GPG errors are too common, and command line magic that even confuses me is necessary to add new repositories.
• Not related to the release, Debian’s support system is a confusing jumble of web interfaces, mailing lists, IRC channels, and an excellent wiki (that I rarely remember to use). Hopefully this is something a new Debian DPL will address.

I should mention this was my fault. I really should have turned fsync off to let the server cache the trickle of ICMP packets that keep our firewall awake all night. Instead the old Quantum Fireball Plus LM harddrive wrote every byte right as it came in.

Luckily the harddrive still worked enough that I could attach it to another computer via a IDE-USB adapter and copy off any data I wanted to keep.

Below are the SMART statistics for the drive as collected by smartmontools and munin. As I was able to retrieve data off the disk, I would say SMART did its job even if it didn’t give me much warning.