When I’m hiring, one of the things I always want to see is evidence of personal projects. Over the last two years, GitHub has become an amazing treasure trove of code, with the best social infrastructure I’ve ever seen on a developer site. GitHub profiles let the user set their location, so I started with a few web searches for Berlin developers. This finds hundreds of interesting people, but how do I prioritise them?

Another thing that I look for when building a good team is someone’s personal network. I’ve always believed strongly in spending lots of time at conferences meeting passionate people who are smarter than me. A good developer can make themselves even more productive by knowing who to email, IM or DM to answer a question when they’re stuck.

A recent article by Stowe Boyd on centrality and influence in social networks reminded me of some of the network analysis we use behind the scenes calculating recommendations for the Dopplr Social Atlas. So I wrote some code to query the GitHub API and analyse the social graph of the Berlin subset of their users.

The JRuby code uses Yahoo BOSS to do the web search. After querying the GitHub API for each user’s followers it builds an in-memory graph using the Java Universal Network/Graph Framework. Then it ranks each user node in the graph using the Betweenness Centrality algorithm. You can see the simple source code on my github.

To sanity-check the results I ran it for a couple of cities I already know well: London and San Francisco. Here are the top 5 for each, which seem quite plausible to me:

San Francisco

1. Chris Wanstrath, GitHub
2. Tatsuhiko Miyagawa, Six Apart
3. Leah Culver, Six Apart
4. Square Inc
5. Aman Gupta, ruby eventmachine maintainer

London

1. James Darling
2. London Ruby User Group
3. Mark Norman Francis
4. Dan Webb (recently moved to Twitter in SF)
5. Carlos Villela, Thoughtworks

My choice of metric biases these lists towards connectedness and influence — it can’t measure ability. It’s only measuring GitHub users, and they are biased towards Ruby, Perl and Javascript. But seeing names there that I trust gives me confidence that it’ll help me find interesting people in Berlin.

Hopefully some of those people are reading this blog post right now. Others outside Berlin might be interested to know that Nokia does a superb job of relocating people, with everything taken care of by shipping companies and local agents. If you love the web, Javascript, mobile, user experience, social networks, location, enormous datasets and currywurst, you should get in touch.

Version 1.10 was released last week, and hidden among the exciting new visual modeling possibilities of shiny rendering and flexible objects was Second Life’s own XMLHTTPRequest: llHTTPRequest. Using asynchronous callbacks, it gives the platform an important new capability: communication with the web on demand. A lot of what we are learning about AJAX makes sense here, in this world of Asynchronous Lindenscriptinglanguage And Some-sort-of-data (ALAS!)

I’ve spent a few hours hacking on some toy objects with this new capability, starting with the mashup de rigeur: Flickr integration. My home in SL now sports a simple picture frame. Touch it and it looks up your avatar name to see what your favourite Flickr tag is, picks a random picture with that tag from Flickr and displays it on its surface. If it hasn’t met you before, it asks you to tell it what tag to use.

Because Second Life is so wonderfully visual, here’s a little demo movie that I recorded with Tom Coates:

how it works

UPDATE: source code is available

LSL is a rather limited C-like scripting language with a couple of interesting features (like its native support for state-machines to respond to events). As far as I’ve been able to work out, it lacks the ability to parse XML or JSON, and has no native associative array type. Objects can store some state, but it is a little fragile in the face of resets. I decided not to try accessing the Flickr API directly, or using local storage. Instead, I created a stateful web companion for my new object with a bit of serverside Rails.

Strings and lists are the most useful datatypes available, and there’s a function similar to perl’s ‘split’ for turning strings into lists. This makes pipe-delimited a reasonable format for simple data. I created the following web resources:

/seen?user=SomeAvatar: Records that the object has sensed the presence of SomeAvatar

/touched?user=SomeAvatar: In response to a ‘touch’ event, consults the database for the user’s tag and asks the Flickr API for a random photo with that tag. Returns a string such as http://flickr.com/some/photo.jpg|SomeAvatar|, or UNKNOWN.

/set_tag?user=SomeAvatar&tag=sausages: Records SomeAvatar’s favourite tag

The HTTP system is nicely responsive: using the web as my object’s outboard brain added only a tiny bit of latency to the mix. The asynchronous model allowed other processing to continue while waiting for a response. With these URLs ready to respond, I wired up the appropriate Second Life events using llSensorRepeat and sensor for presence, llListen and listen to respond to spoken commands, and touch_start for the physical interface. The llParcelMediaCommandList features are confusing (and only work on land you own, with movie streaming enabled in the client), but I found the source code for Freeview to be a useful reference.

If you’re interested in this code, or would like to see a demo, you can find me from time to time in SL as Matt Basiat.

(Maximize your browser window! Apologies for the slow playback speed of the movie; although you’re welcome to browse the javascript, it’s something of a pain to get it running on your own browser. I’m looking at how I can turn it into a reusable and configurable Firefox extension, but for now it’s just a demo built with Greasemonkey.)

UPDATE: I had to demo this to a mixed audience at the BBC this afternoon, so I put together some quick slides to help me explain the step-by-step process that goes on behind the scenes. Perhaps someone else will find them useful too.

So, what are you seeing in this movie? It’s nothing more than a bit of DHTML trickery that imports a subset of del.icio.us functionality into an existing website. I chose BBC Radio 3 because it has a wealth of content with plenty of potential for horizontal navigation, and because it has a clearly-defined canonical URL per programme and thereby gains the maximum benefit from being tagged. By creating a symbiotic relationship between the two sites in your browser, you gain an overlaid cross-site navigation that doesn’t exist in the site as it currently stands, and del.icio.us users see your tagging of Radio 3 pages in the wider context.

There are several things that I enjoy in this demo. In no particular order:

• I like the immediate feedback that you can get from adding a tag to a programme. Decide that ‘cello’ is relevant, and within seconds you see a bunch of other cello programmes. It’s common for content management systems to demand ‘metadata’ or ‘keywords’ of you when you file content, but rare that there’s an easy way to get a feel for what value you’ve added by doing so.
• This was my first real attempt to wrangle the XMLHTTPRequest system, and it was a satisfying one. I did learn one or two things, including some problems with asynchronous and synchronous modes of operation.
• Looking beyond the specific application (tagging) used here, notice the two-way benefit that came from the mashup of one site’s service with another’s content. I like the idea that domain-specific use on Radio 3 leads to general usefulness on del.icio.us.

There are many more possibilities to explore. The demo uses a single user on del.icio.us for all tagging. Imagine instead being able to select between different tag sets to overlay – one to guide newcomers to classical music, another designed for experts and old hands, a third to explore the history of a particular instrument or musical movement.

Guessing that the post would be perfect linkfodder, I was curious to see how the del.icio.us community would take the bait. It went down pretty well. To date, it’s been posted by 153 people.

I knocked up a bit of code to list the tags that the item got, something like this:

tags = {}
for post in d.Href("http://www.hackdiary.com/archives/000060.html"):
for tag in post.tags:
tags[tag.name] = tags.get(tag.name,0) + 1

And what was the emergent categorisation of my piece? It was a pretty broad list, demonstrating several different styles of tagging strategy:

• 97 times python
• 64 times del.icio.us
• 37 times delicious
• 22 times programming
• 10 times hacks
• 7 times tools
• 6 times web
• 4 times software
• 4 times dev
• 3 times rdf
• 3 times bookmarks
• 2 times unread
• 2 times social
• 2 times semweb
• 2 times library
• 2 times hacking
• 2 times hack
• 2 times development
• 2 times del
• 2 times api
• ***
• *checkout
• *touch
• blog
• blogentry
• blognew
• bookmark
• booty
• checkout
• clipz
• code
• collaboration
• collaborativefiltering
• computer
• cool
• del.icio.us social
• delic
• delicios
• devel
• extensions
• geek_stuff
• generalreference
• greatminds
• icio.us
• images
• metabookmark
• metadelicious
• namespaces
• newnet
• news/articles
• read
• readlater
• rss
• ruby
• semanticweb
• socialsoftware
• todo
• viapopular
• webdev
• webservices/delicious
• webtech
• xml
• yummy
• z:active
• z:del.icio.us_experiments

I was also curious about what the pattern of linking would be over time. Mostly I expected the standard power law curve, reflecting an initial flurry of attention followed by a swift drop-off. I put together a quick chart in openoffice:

My hypothesis appears to be largely borne out by the results, but there are some ripples in the tail of the curve where the post regains momentary popularity then tails off again. I’m guessing this happens when the link is discovered by a new del.icio.us sub-community (python programmers, etc) who read each other’s del.icio.us streams via rss or inbox.

A few more numbers: of 153 posts, only 19 used a title other than the page title I used in the post. Only 44 posts gave an extended description. Most users (64) assigned just two tags, with the rest distributed like this:

>>> print len(delicious.Href("http://www.vim.org/").posts())
8

or:

>>> for post in delicious.users.mattb(delicious.tags.hackney):
...   print post.description,post.href
 
Phil Gyford selling his flat http://www.gyford.com/strandbuilding/
POGO http://www.pogocafe.co.uk/
hackneycentral http://www.hackneycentral.org/
Armadillo http://www.armadillorestaurant.co.uk/

or perhaps do something much, much cooler. delicious.py is for you.

UPDATE: Since I published this code, Joshua Schachter has made the rules around use of del.icio.us APIs clearer. So that you can stay within these limits, you should be aware that no call to a method in delicious.py will cause more than one HTTP request to del.icio.us. This means that it’s left up to you to time your requests appropriately and politely, but at least you know that the code won’t spam del.icio.us of its own accord.

As with audioscrobbler a few months back, del.icio.us has been around for a while but last week really started to show up on the radar. I’ve been poking it with bits of code in the hope of achieving enlightenment.

I started by posting a few things: doing some linklogging, scripting an extraction of my items from The Daily Chump into a tag of their own, and extracting and posting pictures from my photo site under a username of their own with simple tags derived from the foaf and wordnet RDF annotations in that site’s RSS. I pointed to the picdiary experiment on the Semantic Web Interest Group IRC channel, and got some interesting discussion out of it.

The python wrapper that I used for most of this is fairly simple in structure. It uses a combination of Redland‘s RDF parsing (for the pleasantly fulsome RSS bits) and libxml2‘s HTML parser for the bits I needed to screenscrape. Href, User and Tag objects are used throughout, and can be used in combination. You can ask a User for all the posts from the user, or restrict it by passing one or more Tags into its post() method, and the converse for the Tags. It’s got some nice python automagic, creating Tag and User objects when you use syntax like “delicious.tags.foo” or “delicious.users.bar” and providing __iter__ methods that allow the “for post in delicious.tags.foo” usage.

There’s a bit of an example in the module’s __main__ code, which will take a username and find all the users that also posted two or more URLs posted by that user, and tell you what tags others used for the URLs posted by that user.

This code has evolved out of my own needs, rather than having a goal in particular. If you find it useful, I’d love to hear from you.

Although the layout was quite simple (two columns of text and images), the automatic HTML export was pretty bad – full of ugly FONT tags and with the images in all the wrong places. I ended up starting again from scratch, hand-producing XHTML and CSS with nothing but a text editor and pasting the text in from the original.

The resulting HTML is valid XHTML 1.1 strict, and degrades nicely in lynx. I’ve not yet been able to check it in many other browsers, but it seems to look OK in IE6 running under win4lin (I don’t have any machines natively running Windows).

I used to shy away from frontend web work, finding it awfully fiddly and dull. Over the last year or two I’ve discovered that making simple semantic HTML and styling it with CSS can be quite a satisfying activity.

This presentation examines the problem of semantic web crawling – following links from document to document and gathering the results for searching. Unlike centralised web search facilities, semantic web agents will be distributed, personalised and often highly domain-specific. How can we hold the entire world inside our laptops?

The W3C vision for the Semantic Web is “an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation”. It is “the representation of data on the World Wide Web”, expressed using the Resource Description Framework (RDF). Just as the web grew in usefulness as it was traversed, indexed and searched by systems such as Lycos, Altavista and Google, the semantic web requires technologies that can crawl, aggregate and query the RDF data.

This talk presents a modular semantic web crawler designed to explore the provision of services to applications. It highlights differences from and similarities to existing web search systems that gather their source data from the public web.

Rather than have web crawling and aggregation built into every semantic web application, agents will be able to call on aggregation services via webservices, be notified of new resources by publish-and-subscribe mechanisms, or simply receive a stream of RDF statements as they are found. A number of different RDF storage mechanisms are tested, including traditional relational databases and RDF toolkits such as Redland.

Applications will be discussed in the areas of social networks (using the Friend Of A Friend vocabulary) and personal publishing. Models for providing centralised services to lighter-weight agents (such as mobile applications) are explored, and important issues such as trust and attribution of information are covered.

It’s now about 80% of the way there, with a colour scheme inspired by a photo of a warning sign that I took somewhere in East London last year.