Using HTTP conditional GET in java for efficient polling

April 9th, 2003  |  Published in java  |  6 Comments

If you’re going to download a resource over HTTP from a URL more than once, there are a couple of features of HTTP you should make sure you’re using. By giving the server some metadata about what you saw when you last downloaded the resource, it can give you a status code indicating that the resource hasn’t changed and you should continue to use the version you already have.

This issue has been highlighted recently by the bandwidth load caused by the growth in popularity of RSS readers, which repeatedly download RSS files looking for changes. There’s a good writeup of the details at The Fishbowl. I didn’t find any sample Java source when I went looking recently, so here’s some code.


If you’re using Jakarta Commons HttpClient and you have an etag and lastModified string cached with a document then use these lines on your GetMethod instance:

GetMethod get = new UrlGetMethod(url);
get.addRequestHeader(new Header("If-None-Match",etag));
get.addRequestHeader(new Header("If-Modified-Since",lastModified));

then check the response code like this:

client.executeMethod(get);
if(get.getStatusCode() < 300) {   // server gave us a document   HeaderElement[] etags = get.getResponseHeader("ETag").getValues();   if(etags.length > 0) {
   String newEtag = etags[0].getName(); // stash this somewhere
  }

  HeaderElement[] mods = get.getResponseHeader("Last-Modified").getValues();
  if(mods.length > 0) {
   String newLastModified = mods[0].getName()); // stash this somewhere
  }
} else {
  // server didn't give us a document, no update
}

The equivalent lines (taken from nntp//rss) for the standard JDK java.net package are:

HttpURLConnection httpCon = ....
httpCon.setRequestProperty("If-None-Match", etag);
httpCon.setIfModifiedSince(lastModified);

and

if(httpCon.getResponseCode() == HttpURLConnection.HTTP_OK) {
 newEtag = httpCon.getHeaderField("ETag");
 newLastModified = httpCon.getHeaderFieldDate("Last-Modified", 0);
}
if(httpCon.getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED) {
  // no change
}

Responses

  1. Development Notebook says:

    April 10th, 2003 at 9:42 am (#)

    Using HTTP conditional GET in java for efficient polling

    Includes code using Jakarta Commons HttpClient . hackdiary: Using HTTP conditional GET in java for efficient polling seeAlso: the fishbowl

  2. Phil Wilson says:

    April 11th, 2003 at 3:23 pm (#)

    excellent, saving the rest of us some work. cheers! :)

  3. sheril says:

    July 23rd, 2003 at 7:43 pm (#)

    What if web server not provides If-Modified -Since Response header?

  4. Kenan KARA says:

    August 7th, 2003 at 6:55 pm (#)

    thanks, for that publication.

    i want to translate that publication to turkish, and publish in my own site with the reference of hackdiary.com. Any problem?

    http://www.kenankara.com

  5. Josh's Hacking Log says:

    September 27th, 2003 at 8:01 am (#)

    Using Conditional GETs for HTTP

    In a post from earlier in the year, Matt Biddulph gives some great advice on how to be a much…

  6. BonnerJackson Aaron says:

    January 9th, 2004 at 11:35 pm (#)

    Truth is not determined by majority vote