The demos:

• Wordnet namespace entry for building
• Picdiary home page
• RSS for Picdiary home page
• Picdiary polaroids view
• Pictures of Edd Dumbill
• Pictures of Edd Dumbill, RSS source
• Wordnet-based search for skyscraper
• Wordnet-based search for building
• Wordnet-hierarchy-nav for structure

The logs on the IRC #www2003 channel during the talk are interesting to compare with the slides.

The talk was scheduled amongst a fun-packed Semantic Web series of Celia Romaniuk (Soap Operas and the Semantic Web), Uche Ogbuji (Akara – Part Wiki, Part Blog, Powered by XML and RDF), Jo Walsh (Collaborative Mapping with RDF) and myself.

RSS 1.0 (RDF Site Summary) is a well-known XML format commonly used for syndicating news headlines. By design it is an extensible format in which metadata expressed using any RDF vocabulary can be linked to its component items, while still maintaining compatibility with applications such as newsreaders that may be unaware of such vocabularies. This paper will discuss some interesting applications that can be built with RSS as a base. The annotation of collections of photographs is used as a case study.

Using an RDF representation of Wordnet, the lexical database of English, we attach keywords to photographs to indicate what they objects they depict. With simple inference logic, we create improved search engines over this data. For example, using the hypernym information in Wordnet to extrapolate from keywords, a search for buildings can find hotels, churches, houses and other related photographs. We can automatically build a Yahoo-like hierarchical web site of photographs organised by the meaning of their keywords.

Using the Friend Of A Friend (FOAF) vocabulary, we can record which photographs show particular people. Applications can then display related information with the photograph by using information gathered by spidering the existing network of FOAF information on the web.

Dynamically-built RSS channels are an ideal format for expressing search results; hence people who are interested could use newsreaders to subscribe to new pictures of any Wordnet noun, particular person or other metadeta criteria. This RSS is easily turned into web pages using templating tools. A Blogger-like front page can be built by building an RSS ‘meta-channel’ summary of the 10 newest photo collections.

Processing toolkits for RDF are available in many languages, and RDF’s natural integration with the web’s URI system makes it easy to build applications in a loosely-coupled style. Tools for this project built using the Redland API in Perl and the Jena toolkit in Java easily interoperate, using the RDF output of one as the input of another. New applications can be built by others using any language or environment with RDF and HTTP support (even Mozilla), without needing to be granted any special database access or other privileges.

Update: slides from this and other talks are available.

After much thought, coding and discussion over the summer with #bots people, I came to the conclusion that a good architectural style for bots is, as jo puts it, “a stateful conversational interface to an aspect of the semantic web”. Querying and updating information is modeled by HTTP GET and POST respectively. The code is structured after the Model-View-Controller pattern commonly used in GUI development: the View is a represention over IRC or another messaging system, the Controller responds to messages generated by running a grammar-based parser over the user’s input, and the Model lives on the web. The bot only keeps state relevant to the conversation (e.g. which picture we are currently talking about).

To get the bot up and running, I started by creating a GET-able query system based on jena. It has a nice RDF query language that’s somewhat SQLish, and can use berkeleydb and rdbms backends. I modeled a store of queries in RDF and wrote a java servlet to run queries and return results in naive XML or RDF formats.

With this flexible backend, I was able to write a commandline version of a bot using simple HTTP requests and the Jakarta Digester to extract information from the XML. Adding a new feature is mostly a case of augmenting the parser’s grammar and adding a new query to the query store.

Here’s a sample session with the commandline version of the bot:

> find mailto:jo@abduction.org
mailto:jo@abduction.org depicted in http://www.picdiary.com/xcom/IMG_1676.jpg
> read xcom
0: http://www.picdiary.com/xcom/IMG_1674.jpg
1: http://www.picdiary.com/xcom/IMG_1676.jpg
2: http://www.picdiary.com/xcom/IMG_1677.jpg
3: http://www.picdiary.com/xcom/IMG_1678.jpg
> current file
xcom
> use 0
Using pic with uri http://www.picdiary.com/xcom/IMG_1674.jpg
> current pic

http://www.picdiary.com/xcom/IMG_1674.jpg

> depicts
mailto:edd@usefulinc.com
mailto:ben@hammersley.com
> use 1
Using pic with uri http://www.picdiary.com/xcom/IMG_1676.jpg
> depicts
mailto:jo@abduction.org

The next step is to extend the query system to allow updates and creates, and I’ll be nearly done.

The code uses RDF::Core::Parser from CPAN, which is a nice simple RDF parser that works much like SAX does for XML. You register an Assert callback with the parser which gets called once for each RDF statement found in the RDF it parses.

A very satisfying hack. RDF and HTTP makes this stuff so easy.

Compare the HTML version to the SVG version. There’s nothing in the SVG version that couldn’t be rendered in HTML; the interest was in doing the job entirely client-side.

The system builds links into foafnaut and my wordnet picture hierarchy when it finds appropriate metadata about a picture.

Jim’s parser page says the RDF parser needs at least version 5.5 to run under IE. Javascript in SVG is a pain to debug in mozilla; you don’t seem to be able to use mozilla’s javascript console or debugger.

Subject: Re: [RSS-DEV] RSS and Libraries
From: mb@p…
Date: Tue Dec 17, 2002 9:18 pm

On Mon, Dec 16, 2002 at 05:14:33PM +0000, Libby Miller wrote:
> On Mon, 16 Dec 2002, se teague wrote:
> > I’m so happy to find this group! I’ve been developing an
> > RSS/Perl/PHP/MySQL based rss program to generate news and events for the
> > academic library I work for. i do have a question though; We haven’t
> > tackled associating images with RSS items rather than channels. Has
> > anyone done this before? Any best practices out there?
>
> I rather like what Matt Biddulph’s been doing, e.g.:
> http://www.picdiary.com/rss/barcelona_conf.rss

Thanks Libby. To (over)elaborate:

My site (http://www.picdiary.com) is just a blog of collections of
pictures. There’s not much text, the pictures are the main focus and
just have a few lines of description. For a while it’s been lacking good
search facilities, and was built out of static HTML.

I decided to use RSS 1.0 as the ‘container’ format for each picture
collection. RSS works just as well for pictures-as-items as it does for
html-pages-as-items – they’re all just web links. So I wrote a bunch of
simple perl scripts to read in RSS and display them as web pages. For
example, The link Libby gave above can be viewed at

or of course it can be viewed in a newsreader/aggregator.

If you look in the HTML source of that page, you’ll see a line like
this:

<link rel=”alternate” type=”application/rss+xml” title=”RSS”
href=”http://www.picdiary.com/rss/barcelona_conf.rss”>

This is following the rss autodiscovery standard so that spiders can
find the RSS metadata from the webpage.

Using RSS 1.0′s RDF capability, I can add all sorts of metadata to
channels and items with a few extra tags in the RSS. That particular
feed uses FOAF to say that certain pictures are of people like Libby.
Other (more recent) feeds such as
http://www.picdiary.com/rss/highwalk.rss also use Dublin Core to give
the creator and date of the channel, and Wordnet to list keywords
associated with each picture (item).

There’s a (prerelease) search engine that indexes all the RSS and uses
the Wordnet terms (and exploits Wordnet’s synonym and hypernym
relationships between words to give better search results) at
http://www.picdiary.com/cgi-bin/search.pl?word=building
If you look ‘behind the scenes’ of that search engine at
http://www.picdiary.com:8180/rss/search?word=building
then you’ll see that the search results themselves are returned as
dynamically-built RSS channels then rendered into HTML by some perl. So
you could subscribe to the search results if you wanted to.

Ramble ramble. There’s quite a lot to say, feel free to followup or mail
me off-list for more information.

Cheers,
Matt.

The pics are marked up in the RSS-based picture collections using markup like this:

<item rdf:about=”http://www.picdiary.com/kensington/img_1926.jpg”>
<foaf:depicts><wordnet:Telephone /></foaf:depicts>
</item>

Search results are actually returned in RSS; the html search page is just a wrapper around that. So you can subscribe to pictures of shops if you really want to.

Next steps:

• Add “more like this” links to picture pages that find pictures by navigating the wordnet space
• make a dmoz-style directory of pictures arranged into the wordnet hierarchy
• do similar things with FOAF person information
• do location annotation using jo’s geonet-based namespace
• tie it all together into some sort of coherent website

(It turns out my code doesn’t do the jpeg embedding quite right. It looks like the JPEG-XMP package will do it correctly.)

Subject: Adobe XMP and jpeg embedding
To: www-rdf-interest@w3.org
Date: 03 Aug 2002 19:18:39 +0100

I was discussing picture metadata with some people this morning and
decided to have a play with the Adobe XMP toolkit [1]. Here are my
notes, in case they are of use or interest to anyone.

job 1: compile under linux, which is unsupported
* the code compiles with warnings if you apply these small
changes across the .c, .cpp and .h files:
* s/xap_xpdomconf.h/XAP_XPDOMConf.h/g
* s/XPDom.h/XPDOM.h/g
* s/XAPTkinternals.h/XAPTkInternals.h/g
* tsk tsk to adobe for developing on a case-insensitive OS but
putting mixed-case filenames in their zipfiles
* also, some files use FREE, REALLOC, CALLOC and MALLOC macros
which presumably would get correctly resolved via a chain of ifdefs
and stuff if i were on windows, but for now just #define them to
regular libc free, realloc, calloc and malloc
* then compile everything with gcc/g++ version 3
* i should make a Makefile really, but i’m too dumb so I just
muck about on the cmdline
* the sample XAPDumper also needs the PacketScanner utility to
compile, but that’s also in the distribution and compiles OK
(perhaps with similar case-changes for include files)
* having done this, i end up with a bunch of apparently
working code, but I am getting some segfaults on exit
* i shall run a memory profiler when i get a round tuit

job 2: try the dumper
* as you’d hope, the PDF files that document the XMP toolkit
themselves have XMP embedded
* so I ran the dumper on XMPEmbedding.pdf and got this dump [2]
* and here’s a nice w3c rdf validator visualisation of the
metadata [3]

job 3: insert XMP into JPEG as described in XMPEmbedding.pdf
* now, time to write some C++ using the toolkit
* I used the MetaXAP object to create a new chunk of rdf
describing a jpeg as having dc:creator David Brophy
* that gets rendered to RDF/XML and the UtilityXAP object
creates an ‘xml packet’ wrapper around the metadata
* i now have a string ready to insert into a jpeg
* i use libexif5 and something named ‘libjpeg’ extracted from
the source distribution of the ‘exif’ command to add data to the
existing jpeg file
* this goes in an APP1 segment that starts with a length value
and the null-terminated string ‘http://ns.adobe.com/xap/1.0/‘,
followed by the
<?xpacket begin=’\x{FEFF}’ id=’W5M0MpCehiHzreSzNTczkc9d’?>
…
<?xpacket end=’w’ ?>
wrapper created by the toolkit
* the resulting jpeg [4] still renders correctly as a jpeg,
and also contains xmp data giving dc:creator as David Brophy
* and works as expected with dan brickley’s XMP demo:
* extracted metadata: [5], raw RDF/XML: [6]
* the slightly shoddy code that does job 3 is available [7]

References

1. http://www.adobe.com/products/xmp/
2. http://www.picdiary.com/~mattb/misc/xmpdump.txt
3. http://www.picdiary.com/~mattb/misc/xmpdump.png

4. http://www.picdiary.com/~mattb/misc/xmp.jpg
5.
http://www.w3.org/2001/sw/Europe/200206/imagemeta/extract/extract?uri=http://www.picdiary.com/~mattb/misc/xmp.jpg&html=true
6.
http://www.w3.org/2001/sw/Europe/200206/imagemeta/extract/extract?uri=http://www.picdiary.com/~mattb/misc/xmp.jpg&html=false
7. http://www.picdiary.com/~mattb/misc/xmptest.cpp

Cheers,
Matt.