Some emails and comments on Screenscraping HTML with TagSoup and XPath alerted me to the fact that the example I gave on that page has gone out of sync with the current release of JDOM and no longer works. I’ve reworked the example using Xalan 2.5.

The problem seems to be that JDOM is asking the TagSoup parser for full namespace support, which it’s not able to give. This new example uses Xalan’s SAX2DOM class to make a DOM tree out of the TagSoup SAX stream, then uses the simple XPathAPI wrapper to make the XPath call.

import java.net.URL;
import org.apache.xalan.xsltc.trax.SAX2DOM;
import org.apache.xpath.XPathAPI;
import org.apache.xpath.objects.XObject;
import org.ccil.cowan.tagsoup.Parser;
import org.w3c.dom.Node;
import org.xml.sax.InputSource;
 
public class example {
 public final static void main(String[] args) throws Exception {
  URL url = new URL("http://example.com");
  Parser p = new Parser();
  p.setFeature("http://xml.org/sax/features/namespace-prefixes",true);
  // to define the html: prefix (off by default)
  SAX2DOM sax2dom = new SAX2DOM();
  p.setContentHandler(sax2dom);
  p.parse(new InputSource(url.openStream()));
  Node doc = sax2dom.getDOM();
  String titlePath = "/html:html/html:head/html:title";
  XObject title = XPathAPI.eval(doc,titlePath);
  System.out.println("Title is '"+title+"'");
 }
}

This code example can be compiled and run with just the TagSoup classes and the Xalan 2.5 main jar on the classpath.

The system is multithreaded and so can simultaneously download from many sources while the aggregation thread does the processing. It builds a model that remembers the provenance of the RDF and takes care to delete and replace triples if it hits the same URL twice, so you can run it as often as you like to keep the data fresh without bloating the store with out-of-date information. As yet it doesn’t do anything with what it gathers; the information’s just sitting there waiting for interesting applications to be built on top of it.

To use it as distributed, set up a mysql database called “scutter” and set the username and password in the DBConnection setup in Scutter.java then recompile using ‘ant compile’ (sorry, no handy config files in this 0.1 release). Run the script scutter.sh passing in as many starting-point URLs as you like. These will be added to the queue, and any rdfs:seeAlso pointers in the downloaded RDF will be recursively followed until no more unique URLs can be found. The biggest known issue at the moment is that it doesn’t do proper management to work out when it’s run out of URLs – it just stops. The standard log4j.properties file can be edited to change what gets logged – with full debugging information turned on, you get quite a lot of output.

Plans for the future include tying FOAF-related processing into the aggregation such as smushing and mbox_sha1sum normalising, and making a publish/subscribe-based system so that people who can’t run their own aggregators can subscribe to the RDF that’s gathered.

Note: the example code below has been updated.

The commandline version looks like this:

wget -O - http://example.com | tidy -asxml - | xsltproc somexsl.xsl -

where somexsl.xsl looks something like this:

<?xml version='1.0'?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version='1.0'>
<xsl:output method="text" />
<xsl:template match="/">
<xsl:value-of select="/html/head/title" />
</xsl:template>
</xsl:stylesheet>

It’s also possible to do the same thing entirely in Java. John Cowan wrote a wonderful HTML parser called TagSoup that outputs SAX events using a do-the-best-I-can approach (“Just Keep On Truckin’” as he describes it) that attempts to make the best job of even the nastiest badly-written HTML. It produces output in cases when HTML TIDY gives up and tells you that errors in the input must be corrected before it can continue.

Because the SAX events just look like XML to any downstream code, it can be plugged into an XPath processor such as Jaxen. XPath processors need DOM trees to work with (because of the backwards-and-forwards-looking nature of the language which makes streaming processing difficult). JDOM contains a nice class called SAXBuilder that can do this SAX-to-DOM conversion, and handily Jaxen can work with JDOM trees directly. So, the Java equivalent of the commandline above is:

URL url = new URL("http://example.com");
SAXBuilder builder = new SAXBuilder("org.ccil.cowan.tagsoup.Parser"); // build a JDOM tree from a SAX stream provided by tagsoup
Document doc = builder.build(url);
JDOMXPath titlePath = new JDOMXPath("/h:html/h:head/h:title");
titlePath.addNamespace("h","http://www.w3.org/1999/xhtml");
String title = ((Element)titlePath.selectSingleNode(doc)).getText();
System.out.println("Title is "+title);

This issue has been highlighted recently by the bandwidth load caused by the growth in popularity of RSS readers, which repeatedly download RSS files looking for changes. There’s a good writeup of the details at The Fishbowl. I didn’t find any sample Java source when I went looking recently, so here’s some code.

If you’re using Jakarta Commons HttpClient and you have an etag and lastModified string cached with a document then use these lines on your GetMethod instance:

GetMethod get = new UrlGetMethod(url);
get.addRequestHeader(new Header("If-None-Match",etag));
get.addRequestHeader(new Header("If-Modified-Since",lastModified));

then check the response code like this:

client.executeMethod(get);
if(get.getStatusCode() < 300) {
  // server gave us a document
  HeaderElement[] etags = get.getResponseHeader("ETag").getValues();
  if(etags.length > 0) {
   String newEtag = etags[0].getName(); // stash this somewhere
  }

  HeaderElement[] mods = get.getResponseHeader("Last-Modified").getValues();
  if(mods.length > 0) {
   String newLastModified = mods[0].getName()); // stash this somewhere
  }
} else {
  // server didn't give us a document, no update
}

The equivalent lines (taken from nntp//rss) for the standard JDK java.net package are:

HttpURLConnection httpCon = ....
httpCon.setRequestProperty("If-None-Match", etag);
httpCon.setIfModifiedSince(lastModified);

and

if(httpCon.getResponseCode() == HttpURLConnection.HTTP_OK) {
 newEtag = httpCon.getHeaderField("ETag");
 newLastModified = httpCon.getHeaderFieldDate("Last-Modified", 0);
}
if(httpCon.getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED) {
  // no change
}

This tarball contains the source, compiled class files and support jars, along with a Jetty setup that will let you run the demo servlet without needing an install of Tomcat or any other servlet engine. Point the indexer at your IMAP host and give it a folder to start from and it will recursively build an index of subject, date, from and mail body. Run Jetty via queryserver.sh and point your browser at http://localhost:9999

The indexer uses the Message-ID as a primary key; it will only index mail it hasn’t seen before when it does a run. This means it will work nicely from a regular cronjob. The query code uses the standard Lucene query parser so will support queries such as +foo +bar, subject:fish and “phrase search”. The spider is independent of the indexer and just fires message events at a MessageListener interface, so it might be useful for other things. The main limitation at the moment (apart from some kind of nice interface) is that the code only copes with single-part messages of type text/plain. The MailDocument class is the place to start improving that.

• convert foaf:mbox entries to privacy-protected foaf:mbox_sha1sum entries.
• normalise statements of the form “PICTURE depicts PERSON” to “PERSON depiction PICTURE”.
• smush disparate references to the same person into references to a single definition of that person.
• extract depiction triples from a model and copy just the bare minimum of information related to those depictions

So I wrote foaftool, a Java class that uses Jena. The tarball also contains a couple of servlets that can be used to transform existing content on the web.

The first servlet will transform foaf:mbox triples in FOAF data into appropriately-encoded foaf:mbox_sha1sum triples. This makes Edd’s FOAF file look like this. Using an extra querystring parameter, it optionally converts foaf:depicts triples to foaf:depiction. foaf:depicts isn’t actually in the official FOAF schema at the time of writing, although it is in informal use in many places as it sometimes makes for more elegant modeling. Normalising to foaf:depictions makes working with large amounts of FOAF data simpler.

Writing the smushing code was an entertaining diversion. Smushing is important when merging multiple RDF sources. Say you have two sources, edd1.rdf and edd2.rdf, showing where to find photos of Edd. When merged, the graph structure looks like this:

This is because without smushing, the anonymous nodes that both have Edd’s email address are not equated. The smushed version (smushed on mbox_sha1sum using a foaftool servlet) looks like this:

With the data in normalised and merged form, I want to extract just the triples of the form “X foaf:depiction [picture uri]” and the related foaf:name and foaf:mbox_sha1sum triples. With the foaftool code, I can now merge and extract depictions from any number of RDF sources.

Comments and bugfixes are very welcome; the code has only been tested as far as the junit tests included in the tarball.

There are some features of the template that are specifically for deploying web apps into Tomcat. They can be easily ignored if you’re not doing that. The directory structure goes like this:

• etc – put any misc files to go in WEB-INF in here and they’ll be copied over when the project is deployed. web.xml lives in here.
• lib – put the .jar files your code depends on in here and they will be automatically put into the classpath for compilation and deployed to the right place in the webapp.
• src – your java source.
• tests – the java source of your junit tests for the code in src.
• web – the web document tree for deployment.

Here are the targets in the ant buildfile:

• compile – compiles the java from src into build.
• test – compiles the source and the tests and runs every test in tests using junit.
• clean – nukes the build and testbuild directories.
• with.jikes – add this before any target that compiles to set jikes as the build compiler.
• with.clover – add this before any target that compiles to instrument the .class files with clover.
• clover.report – runs the clover reporter. Best used after running tests, as in ant clean with.clover test clover.report.
• clover.report.html – runs the clover html reporter.
• deploy – compiles and deploys the project into a tomcat webapp directory, copying the classes, config files from etc, web tree from web and the jars from lib.
• ctags – runs ctags -R over the src and tests directories. This is run automatically by the compile targets.
• javadoc – compile javadoc into javadoc.
• run – run some code from your source with the classpath all set up correctly for your libs and source.
• pause – pauses and waits for carriage-return before running the next target. I use this to make ant more responsive; I find that it can take several seconds for ant to start up and parse its build file, and I like to run the compiler a lot while I’m coding so that I can check my tests as often as possible. In the shell, I use this line: while true; do ant with.jikes pause test; done. This means that every time you hit enter, the compile-and-test run is done then ant makes itself ready for the next run.

After much thought, coding and discussion over the summer with #bots people, I came to the conclusion that a good architectural style for bots is, as jo puts it, “a stateful conversational interface to an aspect of the semantic web”. Querying and updating information is modeled by HTTP GET and POST respectively. The code is structured after the Model-View-Controller pattern commonly used in GUI development: the View is a represention over IRC or another messaging system, the Controller responds to messages generated by running a grammar-based parser over the user’s input, and the Model lives on the web. The bot only keeps state relevant to the conversation (e.g. which picture we are currently talking about).

To get the bot up and running, I started by creating a GET-able query system based on jena. It has a nice RDF query language that’s somewhat SQLish, and can use berkeleydb and rdbms backends. I modeled a store of queries in RDF and wrote a java servlet to run queries and return results in naive XML or RDF formats.

With this flexible backend, I was able to write a commandline version of a bot using simple HTTP requests and the Jakarta Digester to extract information from the XML. Adding a new feature is mostly a case of augmenting the parser’s grammar and adding a new query to the query store.

Here’s a sample session with the commandline version of the bot:

> find mailto:jo@abduction.org
mailto:jo@abduction.org depicted in http://www.picdiary.com/xcom/IMG_1676.jpg
> read xcom
0: http://www.picdiary.com/xcom/IMG_1674.jpg
1: http://www.picdiary.com/xcom/IMG_1676.jpg
2: http://www.picdiary.com/xcom/IMG_1677.jpg
3: http://www.picdiary.com/xcom/IMG_1678.jpg
> current file
xcom
> use 0
Using pic with uri http://www.picdiary.com/xcom/IMG_1674.jpg
> current pic

http://www.picdiary.com/xcom/IMG_1674.jpg

> depicts
mailto:edd@usefulinc.com
mailto:ben@hammersley.com
> use 1
Using pic with uri http://www.picdiary.com/xcom/IMG_1676.jpg
> depicts
mailto:jo@abduction.org

The next step is to extend the query system to allow updates and creates, and I’ll be nearly done.

The class is implemented using the Java Wordnet Library package, which means it requires JDK 1.4.

The major methods are:

• public Synset lookup(String uri)

Takes a URI string such as http://xmlns.com/wordnet/1.6/Dog and returns a JWNL Synset object representing the wordnet synset of that URI.

• public String uri(Synset synset)

Takes a JWNL Synset object and returns its canonical URI string from the wordnet namespace.

The constructor takes a filename of a JWNL config file from which to configure the JWNL system.

Step 1 is a template that generates RDF for each entry (eg the metadata for this entry).
Step 2 is a simple RDF widget (src) that builds an aggregate model of the metadata for each entry.
Step 3 will be to build navigation based on that. To get a flavour of the possibilities, browse the aggregate model using BrownSauce.

There’s an extra mode in the aggregate model system that adds an inSubject reverse triple for each dc:subject triple. Automatic inferencing of that sort might be handy for simplifying query code, and it’s something I’d like to play with, particularly if based on DAML rules.

Perhaps the aggregator could add other inferred metadata based on automatic analysis of the text, or other toys.