UPDATE: related random links and pictures.

There’s plenty of fun python code running behind the scenes (and some perl and java too), which you can read about in a previous entry on this site. If you look closely, you can even see me in the studio in the backstage photo gallery.

I’m most definitely not a GUI person, and I find Powerpoint quite distressing. I’m most happy authoring in VIM. This is the first time I’ve used Eric Meyer’s S5 HTML slide templates, and it’s been a very satisfying experience.

If you’re at the workshop, do come and have a chat. Should be a great Saturday.

UPDATE: Ross Burton picked up the script and made a couple of enhancements. Miguel de Icaza posted a C# version.

To run, you’ll need PyGtkMoz, Gtk and the Python Imaging Library. Because it’s a GTK app, it needs an X server. To run headless, it’ll need an X server like Xnest or VNC. VNC’s working well for me.

The way it works is very simple:

Starting with the example.py shipped with PyGtkMoz, I created a stripped-down browser window app that loads a URL given on the commandline. By connecting the net_stop signal to a method, you can tell when network activity has finished. I couldn’t find a way to be notified when rendering has finished (images decompressed, etc) so I put in a 3 second pause here.

Following clues in a GTK mailing list post, I wrote these lines to save a PNG of my widget’s window:

window = self.widget.window
(x,y,width,height,depth) = window.get_geometry()
pixbuf = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB,False,8,width,height)
pixbuf.get_from_drawable(window,self.widget.get_colormap(),0,0,0,0,width,height)
pixbuf.save("screenshot.png","png")

After that, producing a thumbnail just takes a few more lines of PIL code to open, thumbnail and save the PNG under a new name:

At home I play all my music from the commandline: my mp3 jukebox has no fancy user interface, I just SSH into it. Audioscrobbler have several player plugins for download, but no commandline interface. I wrote myself some python code to play MP3s and talk to audioscrobbler so that I could keep my profile up to date.

Popping onto the audioscrobbler IRC channel, I found Russ and RJ, two key guys behind the system. They were very helpful, sorting me out with a client ID for testing and giving debugging tips.

As so often happens, the python community had already supplied all the libraries I need to make the guts of my system: pymad to decode MP3s, pyao for audio output, pyid3lib to read ID3 tags. Building the code was just a matter of following the protocol documentation and writing about 100 lines to stick it all together.

The Audioscrobbler protocol is very nice. Top marks for using the web the way it was designed, letting me GET the details I need to start a session (MD5 challenge, URL to submit to, etc) and then POST my information to affect my user profile. The protocol used within the transaction doesn’t use RDF or XML, which would be my choice, but it is nice and simple, reminiscent of traditional text-based internet protocols such as SMTP and POP3.

Now I’ve reached this simple goal (a commandline MP3 player with no controls that submits ‘now playing’ information), I’d like to take it further. If I had a newer iPod, it’d be great to write a bit of code to submit tracks I’ve been playing on the move to my profile, but sadly my iPod is of too early a generation to have that feature. More interestingly, I look forward to being able to make use of audioscrobbler’s analysis to build my own playlists and discover new music. For now this is mostly only doable through the audioscrobbler site, but there are also data dumps available. It’d be great to have a combination of the two – an online REST query interface that gives machine-readable results.

Oh, and don’t forget… donate to Audioscrobbler and keep a great project online.

This is a long piece… skip to the end if you want to see how it went on the day.

The project

I work for the beeb at Radio and Music Interactive, in the software architecture team. We’re responsible for the behind-the-scenes plumbing, running the systems behind things like LiveText, and providing toolkits for the guys on the apps team who build public services. When Radio 1 came up with the takeover day (ten hours of radio with music selected entirely by the listeners via text message and phone), they asked us to build them something to deal with the expected deluge of messages.

As you probably know, text messaging is huge in the UK and it’s been taken up enthusiastically by the BBC’s radio networks. Radio 1′s incoming SMS provider have a web console that’s used by the broadcast assistants and DJs in the studio. It works rather like an email inbox, and like most email clients the user interface treats the incoming messages as a chronological stream of text rather than a database to be mined for information. This works for running a regular radio show, but the takeover day needs special treatment.

Since the web console had been enough for the networks so far, this was our department’s first project based on machine-processing of SMS. Knowing how important text is to the radio networks, we wanted to build a foundation of components that would support future applications. As every software engineer knows, it’s hard to “build one to throw away” after the users have got their hands on it.

The build

In our team the language of choice is Python, whereas the apps team prefer Perl. Contrary to what some might expect, this mixed economy hasn’t yet given us any interop problems. We’re very keen on using abstractions such as asynchronous messaging, RESTful web services and relational databases to act as language-agnostic intermediaries for our data flows.

My immediate thoughts about the architecture were that it should be built from a series of loosely-coupled layers. Even solving the simple problem of how to pull in a high-volume external XML feed of text messages and redistribute it to applications would show benefits later if it was simple to hook new code into. Further layers could process the data, with a web user interface layered on top of that. Working from a specification of the XML format used by the provider, Paul Clifford quickly built a gateway that collected messages and rebroadcast them over a message bus. Using asynchronous messaging as a transport gives us a good level of resilience and clusterability for free, while keeping the logic very simple. Building on this, he then started work on code to push messages sent to Radio 1 into a database and analyse their contents.

We wanted to provide simple, useful features to the users such as freetext search, but we thought we could do more given that the incoming texts would have a certain amount of implicit structure. The programme was going to ask people to text in “Artist – Track – Name – Dedication”, but we assumed that there would be a lot of variation in the accuracy of the texts and a lack of consistent punctuation to act as delimiters. We planned a system of ‘fuzzy matching’ to group together texts using stopwords, sounds-alike phonetic matching and statistical analysis of text prefixes. If you see enough text strings beginning with words that sound like “Bob Dylan” then you can start to guess that Bob Dylan might be an artist, rather than a track called Dylan by a band named Bob. Paul did some great work on refining these techniques, and an on-air test a few weeks before the real broadcast showed that we could pretty accurately infer that “Weezer”, “Wheezer”, “Weazer” and “Wheeser” were all the creators of tracks called “Buddy Holly”, “Budy Holly” and “Buddy Holy”.

While Paul worked on the analysis, I created a Perl wrapper around the database so that the apps team could get to work on the user interface. At the same time, I passed on a few lines of sample client code to Matt Webb to see what he could do with it. When working on a toolkit or API, I always like to have more than one client using it, to make sure that the requirements of the main project haven’t kept the interface from being generic and useful in other contexts. He built a nice rolling graph of texts received per minute, tapping in at the message bus layer. As with any messaging system, adding a new message recipient affected neither the code providing the messages nor any other clients of the bus. Neil Slater and Conal Jones in the apps team did great work building a web interface with guidance from the Radio 1 team, and within a few weeks we were ready to go live.

On air

Chris Moyles likes to break things. He once managed to get listeners to send 14,000 texts at once. When I arrived, Aled the BA asked me how many messages the new system could take, and told me that Chris was going to do his best to overload it. At 8.45am, Chris started trailing the feature, and the text messages started trickling in. Then flooding. I was impressed: on a UK Bank Holiday, thousands of people were not just listening but involved enough to get on their phones and interact with the programme.

Chris and his team did a great job of understanding our system. They caught onto the freetext search right away and used it to find interesting tracks, and messages to read out from people who requested them. We put a ‘quick stats’ box on every screen of the app, which they used to goad on the listeners: “3000 text messages so far. That is simply not enough. I want that quadrupled!”. Once there were enough messages in the system for the pattern matching to kick in, they used it to navigate through the data and see which tracks by which artists were getting the most attention. As the listeners realised that the playlist had gone out the window and they could request anything, we got some great stuff coming in. Our system even got a little mention on air as Chris teased us for matching an Elvis Costello track to a bunch of requests for Elvis.

As the programme started I was sitting nervously next-door to the studio tailing logfiles and watching process tables, but as it went on and the code coped with everything they could throw at it, I relaxed and enjoyed the show. I don’t normally listen to the Radio 1 Breakfast Show but listening to this gave me new respect for the skills of a daytime DJ, sitting in a tiny room in a basement working a crowd of millions that they never see.

If you want to hear the show, there’s a Listen Again stream available from the BBC until April 16th. The request section is in the last hour. Tune into Radio 1 on Monday April 12th from 10am for the ten hour version.

UPDATE: Applications have now closed.

FURTHER UPDATE: for unknown reasons, this page is (at the time of writing) number one hit on google for the search term work at the bbc (and perhaps some similar terms). For those who come here looking for work, I suggest you look at the BBC Jobs site or perhaps BBC Talent, where budding writers, presenters and DJs are sought.

Although source for an earlier version was previously available, this new version 1.0 includes all the improvements made by myself and Dave Beckett for Planet RDF. I’ve also refactored the code to use an external configuration file, and packaged it up using python distutils with a simple control script. This means that a single global installation of the system can be used to run multiple clogica sites on a single machine. It’ll also make it easy to build distribution packages for Redhat and Debian.

There’s a useful RDF Interest Group Geo namespace for describing simple lat/long/alt properties of points, so using
PyGarmin I wrote some code to pull off the waypoints and tracklogs and produce a model like this.

The current XMLTV DTD turns out to be pretty RDF-friendly in its structure. This XML example looks very similar as RDF (graph). The major changes I made in the translation were in assigning URIs to channels and programme categories, using identifiers such as http://xmlns.com/2003/rdftv/channel#bbc3.bbc.co.uk and http://xmlns.com/2003/rdftv/category/ananova#News/CurrentAffair, and giving the tv data a namespace under http://xmlns.com. The conversion is performed by some XSLT.

Taking a complete TV feed as RDF, I can now write query code using Redland and Python to print out details of all films on TV this week.

My main need for a remote was for controlling slides when doing presentations, so I looked into how to send fake keypresses to X applications. My presentation format of choice is PDFs, so I just needed to be able to send PgUp and PgDown events to Acrobat Reader.

X has a useful extension called XTEST that is designed to help automated testing of GUI apps. It has a fairly simple API, and with the help of the source of the X11::GUITest CPAN module I created a python extension that lets me write code such as keycontrol.tapkey('PGU') to simulate tapping on the PgUp key.

The resulting code defines an Ericsson class that can be subclassed to receive events such as cursor_left and camera when the keys are pressed on the phone. It helps to have the keypad lock activated when using it so that you don’t end up triggering events on the phone itself.

Libby Miller took a photo of me using it at the XML Summer School Semantic Web forum, where it came in very handy. The phone is just visible in my right hand.