XPaths with xmllint

xmllint is a command-line XML tool used to validate and pretty-print XML documents. More importantly, it offers an interactive shell mode which allows you to use xpaths to print out elements. For example, //body will print out the body element of an HTML document.

I wrote a useful bash function, which uses xmllint to evaluate xpaths really easily:

xpath()
{
    if [ $# -ne 2]; then
        echo "Usage: xpath xpath file"
        return 1
    fi
    xmllint --shell $2 <<< "cat $1" | sed '/^\/ >/d'
}

Example:

sharfah@starship:~> xpath "//body" index.html
<body>Hello World!</body>

Faster XPaths with VTD-XML

I've recently started using VTD-XML for applying XPaths on large XML documents. DOM is a memory hog and is too slow. However, VTD-XML allows you to run XPaths and provides random access to nodes, similar to DOM, but much more efficiently. You can't apply XPaths with a SAX parser nor can you access nodes randomly or traverse the document easily.

VTD-XML was 60 times faster compared to DOM when processing my XML document (20MB).

This post shows you how to use VTD-XML for fast XPath evaluation.

Sample XML:
I will use the following XML document in the examples below.

<?xml version="1.0"?>
<catalog>
 <book id="bk101">
  <author>Gambardella, Matthew</author>
  <author>Doe, John</author>
  <title>XML Developer's Guide</title>
  <genre>Computer</genre>
  <price>44.95</price>
  <publish_date>2000-10-01</publish_date>
 </book>
 <book id="bk102">
  <author>Ralls, Kim</author>
  <title>Midnight Rain</title>
  <genre>Fantasy</genre>
  <price>5.95</price>
  <publish_date>2000-12-16</publish_date>
 </book>
 <book id="bk103">
  <author>Corets, Eva</author>
  <title>Maeve Ascendant</title>
  <genre>Fantasy</genre>
  <price>5.95</price>
  <publish_date>2000-11-17</publish_date>
 </book>
</catalog>

Loading the XML document:
The following code parses the XML file and creates the navigator and autopilot objects.

final VTDGen vg = new VTDGen();
vg.parseFile("books.xml", false);
final VTDNav vn = vg.getNav();
final AutoPilot ap = new AutoPilot(vn);

Selecting all titles:
Print out all the title nodes using an XPath expression of /catalog/book/title. First call selectXPath to compile the expression and then use evalXPath to move the cursor to the selected nodes in the result.

ap.selectXPath("/catalog/book/title");
while (ap.evalXPath() != -1) {
  int val = vn.getText();
  if (val != -1) {
    String title = vn.toNormalizedString(val);
    System.out.println(title);
  }
}

Selecting all book ids and authors:
This one is a bit more involved as a book can have many authors. In the code below, I first run an XPath to select the books and then iterate over the children, selecting the author nodes.

ap.selectXPath("/catalog/book");
while (ap.evalXPath() != -1) {
  int val = vn.getAttrVal("id");
  if(val != -1){
    String id = vn.toNormalizedString(val);
    System.out.println("Book id: " + id);
  }

  if(vn.toElement(VTDNav.FIRST_CHILD,"author")){
    do{
      val = vn.getText();
      if(val != -1){
        String author = vn.toNormalizedString(val);
        System.out.println("\tAuthor:" + author);
      }
    }while(vn.toElement(VTDNav.NEXT_SIBLING,"author"));
  }
  vn.toElement(VTDNav.PARENT);
}

The output is:

Book id: bk101
 Author:Gambardella, Matthew
 Author:Doe, John
Book id: bk102
 Author:Ralls, Kim
Book id: bk103
 Author:Corets, Eva

Related Posts:
Using XPath with DOM

Using XPath in Java

Given the following xml document:

<hosts>
  <host name="starship" port="8080"/>
  <host name="firefly" port="8180"/>
</hosts>

this is how you can use the javax.xml.xpath library to run an XPath query in order to obtain a list of host names:

//create a document
DocumentBuilderFactory domFactory = 
                     DocumentBuilderFactory.newInstance();
domFactory.setNamespaceAware(true);
DocumentBuilder builder = domFactory.newDocumentBuilder();
Document doc = builder.parse("file.xml");

//create the xpath expression
XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("//host/@name");

//run the xpath query
Object result = expr.evaluate(doc, XPathConstants.NODESET);

//read results
NodeList nodes = (NodeList) result;
for (int i = 0; i < nodes.getLength(); i++) {
    System.out.println(nodes.item(i).getNodeValue());
}