Hackdiary

If you’re going to download a resource over HTTP from a URL more than once, there are a couple of features of HTTP you should make sure you’re using. By giving the server some metadata about what you saw when you last downloaded the resource, it can give you a status code indicating that the resource hasn’t changed and you should continue to use the version you already have.

This issue has been highlighted recently by the bandwidth load caused by the growth in popularity of RSS readers, which repeatedly download RSS files looking for changes. There’s a good writeup of the details at The Fishbowl. I didn’t find any sample Java source when I went looking recently, so here’s some code.

If you’re using Jakarta Commons HttpClient and you have an etag and lastModified string cached with a document then use these lines on your GetMethod instance:

GetMethod get = new UrlGetMethod(url);
get.addRequestHeader(new Header("If-None-Match",etag));
get.addRequestHeader(new Header("If-Modified-Since",lastModified));

then check the response code like this:

client.executeMethod(get);
if(get.getStatusCode() 0) {
   String newEtag = etags[0].getName(); // stash this somewhere
  }

  HeaderElement[] mods = get.getResponseHeader("Last-Modified").getValues();
  if(mods.length > 0) {
   String newLastModified = mods[0].getName()); // stash this somewhere
  }
} else {
  // server didn't give us a document, no update
}

The equivalent lines (taken from nntp//rss) for the standard JDK java.net package are:

HttpURLConnection httpCon = ....
httpCon.setRequestProperty("If-None-Match", etag);
httpCon.setIfModifiedSince(lastModified);

and

if(httpCon.getResponseCode() == HttpURLConnection.HTTP_OK) {
 newEtag = httpCon.getHeaderField("ETag");
 newLastModified = httpCon.getHeaderFieldDate("Last-Modified", 0);
}
if(httpCon.getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED) {
  // no change
}