Comments on: Update: Screenscraping HTML with TagSoup and XPath

By: Tunc

Tunc — Wed, 19 May 2004 23:59:09 +0000

I have a question.

When I do run your the example program I get
the following exception.

“Prefix must resolve to a namespace: html”

Any ideas what might be wrong.

Thanks a lot,
-tunc

By: hannes wallnoefer

hannes wallnoefer — Mon, 19 Jan 2004 15:32:53 +0000

TagSoup 0.8 does indeed have a problem with HTML entities. The fix is a one-liner: just add

theSize = 0;

at line 336 in file HTMLScanner.java after calling h.pcdata() and before falling through to A_SAVE_PUSH. I also sent an email to John Cowan about this, so hopefully, there’ll be a fixed version out soon.

By: Danny

Danny — Thu, 01 Jan 2004 16:56:13 +0000

Nice one Matt! I saw the earlier article but held off trying that approach because of the need for JDOM. I’m already using Xalan in my app, and I want to eat tag soup ;-)

By: Matt Biddulph

Matt Biddulph — Mon, 29 Dec 2003 13:12:46 +0000

That’s because I was lazy in the code example and just used the toString() method on the XObject returned from xpath evaluation. If you change the last line to:

org.w3c.dom.NodeList nodes = title.nodelist();
for(int i = 0; i Node node = nodes.item(i);
System.out.println(“Result “+i+”: ‘”+node.toString()+”‘”);
}

then it properly iterates through each separate node returned from the xpath, printing it out. You get lines such as:

Result 5: ‘

28.12.03 22:30 | Australske telemyndigheder sl | Australske telemyndigheder sl&aringr fast, at de nye 3G-sendemaster har lavere str | Australske telemyndigheder sl&aringr fast, at de nye 3G-sendemaster har lavere str&aringling end almindelige mobilmaster og f.eks. taxiradioer.

’

which contain all the tags and entities found. You could also use the methods on the Node object to traverse the tree of divs, spans and text nodes to examine the individual parts instead of using toString().

By: Guan Yang

Guan Yang — Mon, 29 Dec 2003 12:48:34 +0000

Do entities work correctly with Tagsoup? I have fetched the content of http://www.berlingske.dk/ (a Danish newspaper) and run it through the code example above. When I select the path //html:div[@html:class='headline']/following-sibling::html:div[@html:class='teaserText'] I get the following: 29.12.03 10:02 | Over 2000 mennesker er blevet reddet ud i live fra ruiner i den jordsk | Over 2000 mennesker er blevet reddet ud i live fra ruiner i den jordskælvsramte iranske by Bam. Det meddelte den iranske R | Over 2000 mennesker er blevet reddet ud i live fra ruiner i den jordskælvsramte iranske by Bam. Det meddelte den iranske Røde Halvm | Over 2000 mennesker er blevet reddet ud i live fra ruiner i den jordskælvsramte iranske by Bam. Det meddelte den iranske Røde Halvmåne mandag. It's supposed to be: | Over 2000 mennesker er blevet reddet ud i live fra ruiner i den jordsk