This is a long piece… skip to the end if you want to see how it went on the day.

The project

I work for the beeb at Radio and Music Interactive, in the software architecture team. We’re responsible for the behind-the-scenes plumbing, running the systems behind things like LiveText, and providing toolkits for the guys on the apps team who build public services. When Radio 1 came up with the takeover day (ten hours of radio with music selected entirely by the listeners via text message and phone), they asked us to build them something to deal with the expected deluge of messages.

As you probably know, text messaging is huge in the UK and it’s been taken up enthusiastically by the BBC’s radio networks. Radio 1′s incoming SMS provider have a web console that’s used by the broadcast assistants and DJs in the studio. It works rather like an email inbox, and like most email clients the user interface treats the incoming messages as a chronological stream of text rather than a database to be mined for information. This works for running a regular radio show, but the takeover day needs special treatment.

Since the web console had been enough for the networks so far, this was our department’s first project based on machine-processing of SMS. Knowing how important text is to the radio networks, we wanted to build a foundation of components that would support future applications. As every software engineer knows, it’s hard to “build one to throw away” after the users have got their hands on it.

The build

In our team the language of choice is Python, whereas the apps team prefer Perl. Contrary to what some might expect, this mixed economy hasn’t yet given us any interop problems. We’re very keen on using abstractions such as asynchronous messaging, RESTful web services and relational databases to act as language-agnostic intermediaries for our data flows.

My immediate thoughts about the architecture were that it should be built from a series of loosely-coupled layers. Even solving the simple problem of how to pull in a high-volume external XML feed of text messages and redistribute it to applications would show benefits later if it was simple to hook new code into. Further layers could process the data, with a web user interface layered on top of that. Working from a specification of the XML format used by the provider, Paul Clifford quickly built a gateway that collected messages and rebroadcast them over a message bus. Using asynchronous messaging as a transport gives us a good level of resilience and clusterability for free, while keeping the logic very simple. Building on this, he then started work on code to push messages sent to Radio 1 into a database and analyse their contents.

We wanted to provide simple, useful features to the users such as freetext search, but we thought we could do more given that the incoming texts would have a certain amount of implicit structure. The programme was going to ask people to text in “Artist – Track – Name – Dedication”, but we assumed that there would be a lot of variation in the accuracy of the texts and a lack of consistent punctuation to act as delimiters. We planned a system of ‘fuzzy matching’ to group together texts using stopwords, sounds-alike phonetic matching and statistical analysis of text prefixes. If you see enough text strings beginning with words that sound like “Bob Dylan” then you can start to guess that Bob Dylan might be an artist, rather than a track called Dylan by a band named Bob. Paul did some great work on refining these techniques, and an on-air test a few weeks before the real broadcast showed that we could pretty accurately infer that “Weezer”, “Wheezer”, “Weazer” and “Wheeser” were all the creators of tracks called “Buddy Holly”, “Budy Holly” and “Buddy Holy”.

While Paul worked on the analysis, I created a Perl wrapper around the database so that the apps team could get to work on the user interface. At the same time, I passed on a few lines of sample client code to Matt Webb to see what he could do with it. When working on a toolkit or API, I always like to have more than one client using it, to make sure that the requirements of the main project haven’t kept the interface from being generic and useful in other contexts. He built a nice rolling graph of texts received per minute, tapping in at the message bus layer. As with any messaging system, adding a new message recipient affected neither the code providing the messages nor any other clients of the bus. Neil Slater and Conal Jones in the apps team did great work building a web interface with guidance from the Radio 1 team, and within a few weeks we were ready to go live.

On air

Chris Moyles likes to break things. He once managed to get listeners to send 14,000 texts at once. When I arrived, Aled the BA asked me how many messages the new system could take, and told me that Chris was going to do his best to overload it. At 8.45am, Chris started trailing the feature, and the text messages started trickling in. Then flooding. I was impressed: on a UK Bank Holiday, thousands of people were not just listening but involved enough to get on their phones and interact with the programme.

Chris and his team did a great job of understanding our system. They caught onto the freetext search right away and used it to find interesting tracks, and messages to read out from people who requested them. We put a ‘quick stats’ box on every screen of the app, which they used to goad on the listeners: “3000 text messages so far. That is simply not enough. I want that quadrupled!”. Once there were enough messages in the system for the pattern matching to kick in, they used it to navigate through the data and see which tracks by which artists were getting the most attention. As the listeners realised that the playlist had gone out the window and they could request anything, we got some great stuff coming in. Our system even got a little mention on air as Chris teased us for matching an Elvis Costello track to a bunch of requests for Elvis.

As the programme started I was sitting nervously next-door to the studio tailing logfiles and watching process tables, but as it went on and the code coped with everything they could throw at it, I relaxed and enjoyed the show. I don’t normally listen to the Radio 1 Breakfast Show but listening to this gave me new respect for the skills of a daytime DJ, sitting in a tiny room in a basement working a crowd of millions that they never see.

If you want to hear the show, there’s a Listen Again stream available from the BBC until April 16th. The request section is in the last hour. Tune into Radio 1 on Monday April 12th from 10am for the ten hour version.

Every night after midnight, hits.pl runs and inspects the previous day’s Apache access log. Every entry has a predictable URL of the form http://www.hackdiary.com/archive/000articleID.html, so the code looks for hits on those URLs and parses out the article ID. It updates a dbm file, adding the day’s hits to hits already recorded for each item.

After that’s finished, hits2rss.pl is run and translates the dbm file into an RSS file by looking up the article titles via the MT XML-RPC interface. An MT plugin parses that file when the index page is rebuilt and makes the sidebar.

Fun hacks like this make me wish that MT had a license that allowed distribution of modifications to its core. Although the plugin interface is flexible, it’s a dead-end in the long run without the ability to dig in and change things under the skin.

The slides (in PDF created using AxPoint) are brief but hopefully useful. At the same meeting, Celia talked about her work at the BBC on modelling Eastenders plots in RDF and Jo talked about spacenamespace, mudlondon and collaborative mapping. It was great to chat to people in the pub afterwards and see a lot of enthusiasm for the ideas.

The code uses RDF::Core::Parser from CPAN, which is a nice simple RDF parser that works much like SAX does for XML. You register an Assert callback with the parser which gets called once for each RDF statement found in the RDF it parses.

A very satisfying hack. RDF and HTTP makes this stuff so easy.

As an example, in this entry, I put this in my keywords:

<dc:identifier> "some unique id" .
<rdfs:seeAlso> <http://www.movabletype.org> .

and two extra triples get created in the RDF for this entry.

It’s used in the MT RDF template by putting <$MTEntryKeywords ntriples="1"$>. Of course any namespace prefix used in the Keywords field must be declared in the RDF template beforehand. Apart from that it should be pretty resilient against breaking because it ignores any line that doesn’t match the syntax exactly.

rdfparse-perl.tar.gz

The XSLT is taken from the rubyrdf distribution. The perl code uses XML::LibXML and XML::LibXSLT.

Subject: [Bots] gatebot – making bots URL-addressable
From: Matt Biddulph
Date: Mon Aug 26 14:41:05 2002

I’ve put up a new experimental service that allows code to talk to IRC
bots over the web without making a direct IRC connection.

You can try it out here via a dodgy web form interface:

http://www.picdiary.com/bot/index.html

But it’s really meant as some kind of web API for access by code (e.g.
so web apps could ask the foafbot for people-related information). It
works like this:

POST to
http://www.picdiary.com/bot/servlet/GatebotServlet/some.irc.server/somebotnick
with a query of "say=what%20you%want%20to%say". The web service sends a
proxy bot onto some.irc.server and privmsgs somebotnick with what you
want to say.

You get back a simple HTML document, with a 201 Created status to
indicate the creation of a new session URL for you to read back what is
said. This session URL is given both in the HTML (for browser-based
testing) and in the HTTP Location: header. Every time you hit this URL,
it’ll give you a line-by-line dump of what have been privmsged back to
the proxy bot (obviously this will be blank if you hit the URL quickly -
if you expect an answer then poll it at some regular interval).

If nothing is said to the bot for 60 seconds, it leaves the IRC network
but the URL remains available ‘indefinitely’ (it’s kept in memory and
will die when my webserver restarts). You should also be able to say
further things to the same bot in the same session by POSTing to the
session URL with "say=whatever"

Source code (perl for the bot bits, java servlet for the web bits) is
here:
http://www.picdiary.com/~mattb/bots/gatebot-0.2.tar.gz
Apologies for any hard-coded paths or other cruftiness. The version of
Bot::BasicBot in there is slightly modified to listen on STDIN so that
the web layer can communicate with its proxy bots by spawning processes
and using STDIN/STDOUT.

There are a couple of pieces of demo client code in the tarball.
getkarma asks the infobot ‘robert’ on #pants for karma of things via a
commandline interface. There’s also an infobotkarmabot that does the
same thing but with an IRC interface.

Here’s the IRC karma bot connecting to #bots on london.rhizomatic.net:

–> robertpro (~robertpro@217.158.195.49) has joined #bots
<mattb> robertkarma test
<robertpro> robert says test has karma of 1

and the action this triggered on #gatebot (where proxy bots hang out
while they’re privmsging) on the irc network where #pants lives:

–> EUUREbot (tomcat4@pod-124.dolphin-server.co.uk) has joined #gatebot
<EUUREbot> robert said ‘test has karma of 1′

Feel free to play with this, and flame me royally if it breaks or sucks.

Cheers,
Matt.

and a later response to a followup:

On Mon, 2002-08-26 at 17:16, Mark Fowler wrote:
> Hmm. Any reason this can’t be a GET? Aside from the breaking of the
> idempotent nature of requests I mean (pah, RFCs.) It’d just be nice so
> that you can use it with Mozilla’s bookmark keywords function (where you
> can type ‘$keyword $foo’ and it looks up a url you’ve associated with
> $keyword changing ‘%s’ for a URI encoded version of $foo.)

Yeah, preserving the semantics of POST seems quite important here – this
is initiating an action, which causes the creation of a resource (both
in the sense of the URL and the process actually forked). This resource
then becomes readable in an idempotent way, its content changing as it
(hopefully) interacts with the remote bot it’s talking to.

In order to support an application such as ‘return me this information’
for bookmark-keywords purposes, I would rather a wrapper application
were used that was truly GETtable – it would make the initial POST and
poll the resulting GETtable session until it returned some useful
information. Only then would it return 200 OK and some content to the
client. A directly GETtable gatebot service would 99% of the time simply
return a blank screen to the browser because it has no way of knowing
how long to wait (if it should wait for a response at all).