But first things first: I need to put data into shared memory from Python. mmap is an excellent widely-implemented POSIX system call for creating a shared memory space backed by an on-disk file.

Usually in the UNIX world you have 2 ways of accessing/manipulating data: memory addresses or streams (files). Manipulating data via memory addresses means pointers, offsets, malloc/free, etc. Stream interfaces manipulate data via read/write/seek system calls for files and send/recv/etc for sockets.

mmap gives you both interfaces. A memory mapped file can be manipulated via read/write/seek or by directly accessing its mapped memory region. The advantage of the latter is that this memory region is in userspace — meaning you can manipulate a file without incurring the overhead of write system calls for every manipulation.

Anyway, enough exposition, let’s see some code. (Despite mmap’s nice featureset, I’m only using it as a simple memory sharing mechanism anyway.) The following code shares a tiny bit of data between 2 Python processes using the excellent mmap module in the stdlib. a.py writes to the memory mapped region, and b.py reads the data out. ctypes allows for an easy way to create values in a memory mapped region and manipulate them like “normal” Python objects.

These code samples were written using Python 2.7 on Linux. They should work fine on any POSIX system, but Windows users will have to change the mmap calls to match the Windows API.

a.py

#!/usr/bin/env python
import ctypes
import mmap
import os
import struct
 
 
def main():
    # Create new empty file to back memory map on disk
    fd = os.open('/tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR)
 
    # Zero out the file to insure it's the right size
    assert os.write(fd, '\x00' * mmap.PAGESIZE) == mmap.PAGESIZE
 
    # Create the mmap instace with the following params:
    # fd: File descriptor which backs the mapping or -1 for anonymous mapping
    # length: Must in multiples of PAGESIZE (usually 4 KB)
    # flags: MAP_SHARED means other processes can share this mmap
    # prot: PROT_WRITE means this process can write to this mmap
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
 
    # Now create an int in the memory mapping
    i = ctypes.c_int.from_buffer(buf)
 
    # Set a value
    i.value = 10
 
    # And manipulate it for kicks
    i.value += 1
 
    assert i.value == 11
 
    # Before we create a new value, we need to find the offset of the next free
    # memory address within the mmap
    offset = struct.calcsize(i._type_)
 
    # The offset should be uninitialized ('\x00')
    assert buf[offset] == '\x00'
 
    # Now ceate a string containing 'foo' by first creating a c_char array
    s_type = ctypes.c_char * len('foo')
 
    # Now create the ctypes instance
    s = s_type.from_buffer(buf, offset)
 
    # And finally set it
    s.raw = 'foo'
 
    print 'First 10 bytes of memory mapping: %r' % buf[:10]
    raw_input('Now run b.py and press ENTER')
 
    print
    print 'Changing i'
    i.value *= i.value
 
    print 'Changing s'
    s.raw = 'bar'
 
    new_i = raw_input('Enter a new value for i: ')
    i.value = int(new_i)
 
 
if __name__ == '__main__':
    main()

b.py

import mmap
import os
import struct
import time
 
def main():
    # Open the file for reading
    fd = os.open('/tmp/mmaptest', os.O_RDONLY)
 
    # Memory map the file
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
 
    i = None
    s = None
 
    while 1:
        new_i, = struct.unpack('i', buf[:4])
        new_s, = struct.unpack('3s', buf[4:7])
 
        if i != new_i or s != new_s:
            print 'i: %s => %d' % (i, new_i)
            print 's: %s => %s' % (s, new_s)
            print 'Press Ctrl-C to exit'
            i = new_i
            s = new_s
 
        time.sleep(1)
 
 
if __name__ == '__main__':
    main()

(Note that I cruelly don’t clean up /tmp/mmaptest after the scripts finished. Consider it a 4KB tax for anyone who runs arbitrary code they found on the Internet without reading it first.)

Writing asynchronous IO code is fun; handling signals is not. signalfd allows you to move your signal handling code into your main event loop instead of hooking up global handlers and using the featureless set_wakeup_fd function to break the main loop.

Luckily Jean-Paul Calderone had already created a great Python wrapper for the signalfd and sigprocmask system calls. Unfortunately it doesn’t include a way to parse the siginfo_t structure which contains all the useful information about the signal you’re handling.

I’ve added a helper to do just that in a branch:

https://code.launchpad.net/~schmichael/python-signalfd/helpers

A sample program would look like:

 
import os
import select
import signal
 
import signalfd
 
 
def sigfdtest():
    # Catch them all!
    sigs = []
    for attr in dir(signal):
        if attr.startswith('SIG') and not attr.startswith('SIG_'):
            sigs.append(getattr(signal, attr))
 
    sfd = signalfd.create_signalfd(sigs)
    print 'Capturing: %r' % sorted(sigs)
 
    while 1:
        print 'selecting - pid: %d' % os.getpid()
        r = select.select([sfd], [], [])[0]
        for s in r:
            assert s is sfd, 'Python nicely re-uses the fd instance'
            sig = signalfd.read_signalfd(sfd)
            print sig
 
 
if __name__ == '__main__':
    sigfdtest()

When run you can throw some signals at it:

Capturing: [6, 14, 7, 17, 17, 18, 8, 1, 4, 2, 29, 6, 9, 13, 29, 27, 30, 3, 64, 34, 11, 19, 31, 15, 5, 20, 21, 22, 23, 10, 12, 26, 28, 24, 25]
selecting - pid: 6523
^CSIGINT
selecting - pid: 6523
^CSIGINT
selecting - pid: 6523
SIGHUP
selecting - pid: 6523
Killed

Of course you’ll need to use an uninterpretable signal like KILL to exit.

Jean-Paul attempted to get signalfd included in Python 2.7′s signal module, and it was slated for inclusion in 3.2. However, given that 3.2 was just released without it, I’m guessing the attempt to get this functionality into Python’s stdlib has been forgotten.

Up next: eventfd perhaps?

I should mention Jason Kirtland informed me after the meeting that FastCGI supports persistent connections (and a host of other features) between a load balancer and backend app servers.

Good news

• ATI graphics card with proprietary binary driver – just works (much better than 9.04 as well).
• Suspend & hibernate work flawlessly.
• Boots fast.
• Upgrade worked flawlessly.
• Sound (better volume panel!), wifi, USB, network printer, bluetooth (option to turn it on/off!), webcam, etc. all still work.
• New theme is nice.
• I manually upgraded my filesystem to ext4 and grub from version 1 to 2. It was a bit scary but worked out fine. Can’t really feel a difference, but my laptop is mainly a dumb terminal for running web browsers and ssh.

Empathy

Ubuntu replaced Pidgin with Empathy as the default IM client in Ubuntu 9.10. I think because Empathy supports voice/video chat and Pidgin doesn’t? I’ve used video chat once in my life and that was through Skype on Linux. It worked great, but it’s really not a feature I care about.

So for someone like me who doesn’t care about Empathy’s singular advantage over Pidgin, Empathy is a major step backward. At first it was extremely crashy, but a recent update seems to have fixed that. However, now it mysteriously loses messages. I’m a very light IM user, but it would start silently missing messages a few hours into the day every day.

I’ve switched back to Pidgin and couldn’t be happier.

Message Notification applet deal

Empathy integrates with the message notification applet deal along with Evolution. Other apps may as well, but evidently I don’t use any of them. My top panel is 75% whitespace, so the singular benefit of Ubuntu’s consolidated message notification applet was completely lost on me.

So much like Empathy, I removed this specialized applet as well. I’m much happier with per-application icons anyway and fail to see what the benefit of consolidating them is (unless you have a really cramped top panel).

New Theme

I like it, but then I installed Chromium. Now the bold window titles and expansive title bars in Metacity look bulky and antiquated. Gnome really needs to evolve their window manager and default UI. Chrome is an excellent example of how to design a compact, minimal, yet still pleasant and intuitive user interface.

Ubuntu Software Center

What a curious little replacement for the old Add/Remove Applications program. I think I see the direction they’re headed, but it definitely feels like rolled out a beta program to replace a perfectly functional and stable one.

The left pane with expansive whitespace and 2 options hints that there might be more categories in the future, but right now it just looks like a mistake. Like maybe something isn’t working properly, and I’m not seeing all the options I should see.

At any rate, I hope Ubuntu adds an App Store that even includes evil proprietary software. I’d love to be able to plunk down a few bucks for a game like Braid directly from Ubuntu Software Center. That’d be great! Maybe Canonical could even pocket a few pennies and start making Linux-on-the-desktop profitable. Now I’m just dreaming though…

Bottom Line

9.10 is a solid and safe upgrade for any users of previous versions. Not sure there’s anything new to win over users from OS X or Windows though.

The Win

Essentially what I want for Gnome 3.0 is what Tomboy just got: a web interface called Snowy. Part synchronization, part collaboration, all “cloud” for those buzzword minded among us.

The Fail

Luckily there’s already Conduit which I think does the synchronization bit… I say “I think” because I’ve never actually gotten Conduit to work. I drag & drop F-Spot onto the canvas**, neat. I drag & drop Network to the right of F-Spot, neat. Now what? There’s no configure option for Network. Heck, there’s not even a Remove*** for Network. I right click and select Synchronize in hopes it will prompt me for details, but instead it happily reports synchronization is finished. Huh?!

I poke around in Preferences hoping for guidance on how to sync to another computer over a network. Wow. The first tab is fine, but the other 2 tabs are just long lists of text. Wait a second, if you scroll horizontally to the right in the Data Providers tab you can see a checkbox to load/unload them! Why you’d want this option other than for development, I’m not sure, but its there if you know where to look.

At this point Conduit has failed to gain my trust, so I give up. I’m fine with Pidgin being crash happy or Eclipse being ridiculously complex to configure. The former isn’t mission critical and the latter is intended for developers.

However, I’m expected to entrust all of my files to Conduit. Files and account information for network resources if I could figure out how to get that to work. I have pretty high expectations for anything I’m entrusting my files and passwords too. Right now Conduit seems more like a demo of GTK+’s canvas drawing abilities (it is pretty looking) than a powerful synchronization tool.

The Dream

My expectations for web enabling Gnome are insanely high. Synchronization isn’t enough (although it’d be an excellent start). I need web enabled collaboration features as well. I need it to Just Work™ as well as be Secure By Default™.

This means:

• All applications can synchronize their settings and data.
• All data has some sort of web viewable format and ideally a way to interact with it on the web as well.
• By default all settings and data should be private.
• Fine-grained CRUD permissions for all settings and data. The Create and Update are key here: not only should I be able to give other users the ability to edit my files but Grandma should be able to upload photos to my photo album.
• While I’m dreaming can I add versioning support (like Dropbox)? Cool, thanks.

The Implementation

Unfortunately I think Conduit is the wrong way to go. The concept is sound, but Worse Is Better™: each application needs to be allowed to implement their own integration with the cloud synchronization and collaboration system. There’s too many application specific tasks to accomplish intuitive synchronization and collaboration features outside the main application.

Tomboy has had pretty good synchronization for a while now. Firefox is getting it with Weave.

In a way F-Spot and Banshee support 1-way synchronization and collaboration. F-Spot can upload photos to a variety of services which have all (or at least most) of the features I’m pining for. Banshee’s ability to download Podcasts and music synchronize to iPods (note: I’ve never used that feature) is a sort of primordial synchronization. If only Banshee Web had survived (and wasn’t written in ASP.NET, the ickiest of all webdev platforms). Abiword has been toying with cloud services for a little while as well.

Perhaps Evolution is the poster child for this: I currently use it as a Gmail and Gcal interface sometimes because there are certain use cases for which I prefer it (shocking I know!). Despite its past reputation for being bloated and buggy, it Just Works&tm; these days and integrates beautifully with the rest of the Gnome desktop (are those calendar events in my system tray calendar?! amazing! Pidgin integration?! <3 Evolution). Evolution lives completely in the cloud and synchronizes everything locally beautifully (after years and years of effort).

Why Gnome Can Do It

Gimp is actually the first Gnome app to really make me realize working on the web can and should be as easy as working locally. All it did was utilize Gnome’s VFS to edit and save images via FTP or SSH. With Gnome’s screenshot tool, I can save a screenshot to a web accessible location in 1 keypress and 2 clicks. Amazing! Thats almost as fast and easy as using a pastebin tool on the command line for sharing text.

Snowy seems like an excellent poster child for the web interface I’m whining about. I haven’t used it yet, but the very idea that Tomboy already has an open source reference implementation is an excellent start. 3rd parties could implement the protocols and offer competing services.

In an ideal world this would create a revenue stream for open source applications: services like Dropbox could support the synchronization and collaboration protocols and in return support development of the projects. The more people who use apps that integrate seamlessly with Dropbox, the more people who pay for Dropbox… or box.net or Amazon or… you get the idea.

FreeDesktop.org Does What?!

That’s right, FreeDesktop.org needs to lead the charge here. Its not enough that the protocols are open, application developers need libraries to make synchronization and collaboration easy. In that way Conduit is right: each application should not completely reinvent the wheel.

FreeDesktop.org needs to follow Mozilla’s lead. Mozilla realized early on that in order for their browser business to succeed, they needed open standards to succeed. FreeDesktop.org needs to do the same. A KDE application should be able to synchronize to the same service as a Gnome app (this is where Conduit’s monolithic-master-synchronization-app model seems too optimistic to succeed). OSX and Windows apps should be able to use the base synchronization libraries as well, even if they don’t use GTK+ or QT.

At the end of the day I should be able to sync my photos, documents, and mp3s between my laptop, desktop, cell phone, Kindle, etc as well as any relevant application settings (Tomboy for Kindle?! yes please!). …and so much more.

* Not always true. For example: posting opinions on a piece of software long after the roadmap has been made.
** Strike 1 against Conduit: “Canvas” isn’t a very end user friendly term IMHO. A minor complaint, but as confusing as the UI already is, it needs all the help it can get. Perhaps the problem isn’t the name “Canvas” but rather that the fact that it doesn’t even need to be said. Remove all references to it and just put a textbox that says “Drop data providers here.” when the canvas is empty. The “Clear canvas” option can just be “Clear” as there’s really only 1 thing that can be cleared.
*** Strike 1.5: “Remove” would be better than “Delete”. Delete makes me wonder if its going to erase my files! Remove seems more descriptive and precise. Also, drop “item” from the end of each right click menu option. Its redundant when every option operates on the item you clicked.

Enter Dropbox: a service for synchronizing files between computers. It works great in Gnome (Windows and Mac OSX as well).

I use Dropbox for synchronizing my Tomboy notes by telling Tomboy to synchronize to a Local Folder: ~/Dropbox/Tomboy

My one gripe is that you have to go into a note to synchronize. Someone has filed a bug and submitted a patch, so hopefully it will be fixed soon.