XPath is a useful tool to have when web scraping since it gives you more flexibility when specifying document locations than CSS selectors offer. If you’re searching for a lesson, check out this one on XPath with good examples.
In this article, we’ll use the Scrapy Selector API as an example to provide some XPath usage recommendations we discovered to be helpful.
Avoid using contains(.//text(), ‘search text’) in your XPath conditions.
Use contains(., ‘search text’) instead.
Why is this: The result of the expression .//text() is a node-set, which is a collection of text items. Additionally, the text for the first element is the only one that appears when a node-set is transformed to a string, which occurs when it is supplied as an argument to a string function like contains() or starts-with().
>>> from scrapy import Selector
>>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
>>> xp = lambda x: sel.xpath(x).extract() # let's type this only once
>>> xp('//a//text()') # take a peek at the node-set
[u'Click here to go to the ', u'Next Page']
>>> xp('string(//a//text())') # convert it to a string
[u'Click here to go to the ']
However, a node that has been transformed to a string combines the text of both itself and all of its descendants:
>> xp('//a[1]') # selects the first a node
[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>']
>>> xp('string(//a[1])') # converts it to string
[u'Click here to go to the Next Page']
So, in general:
GOOD:
>>> xp("//a[contains(., 'Next Page')]")
[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>']
BAD:
>>> xp("//a[contains(.//text(), 'Next Page')]")
[]
GOOD:
>> xp("substring-after(//a, 'Next ')")
[u'Page']
BAD:
>>> xp("substring-after(//a//text(), 'Next ')")
[u'']
You can read more detailed explanations about string values of nodes and node-sets in the XPath spec. http://www.w3.org/TR/xpath/#dt-string-value
Beware of the difference between //node[1] and (//node)[1] //node[1] selects all the nodes occurring first under their respective parents.
(//node)[1] selects all the nodes in the document, and then gets only the first of them.
>>> from scrapy import Selector
>>> sel=Selector(text="""
....: <ul class="list">
....: <li>1</li>
....: <li>2</li>
....: <li>3</li>
....: </ul>
....: <ul class="list">
....: <li>4</li>
....: <li>5</li>
....: <li>6</li>
....: </ul>""")
>>> xp = lambda x: sel.xpath(x).extract()
>>> xp("//li[1]") # get all first LI elements under whatever it is its parent
[u'<li>1</li>', u'<li>4</li>']
>>> xp("(//li)[1]") # get the first LI element in the whole document
[u'<li>1</li>']
>>> xp("//ul/li[1]") # get all first LI elements under an UL parent
[u'<li>1</li>', u'<li>4</li>']
>>> xp("(//ul/li)[1]") # get the first LI element under an UL parent in the document
[u'<li>1</li>']
Also,
//a[starts-with(@href, ‘#’)][1] gets a collection of the local anchors that occur first under their respective parents.
(//a[starts-with(@href, ‘#’)])[1] gets the first local anchor in the document.
When selecting by class, be as specific as necessary
If you want to select elements by a CSS class, the XPath way to do that is the rather verbose:
*[contains(concat(‘ ‘, normalize-space(@class), ‘ ‘), ‘ someclass ‘)]
Let’s cook up some examples:
>>> sel = Selector(text='<p class="content-author">Someone</p><p class="content text-wrap">Some content</p>')
>>> xp = lambda x: sel.xpath(x).extract()
BAD: doesn’t work because there are multiple classes in the attribute
>>> xp("//*[@class='content']")
[]
BAD: gets more than we want
>>> xp("//*[contains(@class,'content')]")
[u'<p class="content-author">Someone</p>']
GOOD:
>>> xp("//*[contains(concat(' ', normalize-space(@class), ' '), ' content ')]")
[u'<p class="content text-wrap">Some content</p>']
And many times, you can just use a CSS selector instead, and even combine the two of them if needed:
ALSO GOOD:
>>> sel.css(".content").extract()
[u'<p class="content text-wrap">Some content</p>']
>>> sel.css('.content').xpath('@class').extract()
[u'content text-wrap']
Read more about what you can do with Scrapy’s Selectors here.
Learn to use all the different axes
It is useful to understand how to use the axes; you may quickly review this by working through the tutorial’s examples. http://scrapy.readthedocs.org/en/latest/topics/selectors.html#nesting-selectors
In particular, you should note that following and following-sibling are not the same thing, this is a common source of confusion. The same goes for preceding and preceding-sibling, and also ancestor and parent.
Useful trick to get text content
Here is another XPath trick that you may use to get the interesting text contents:
//*[not(self::script or self::style)]/text()[normalize-space(.)]
This excludes the content from script and style tags and also skip whitespace-only text nodes.
Source: https://stackoverflow.com/a/19350897/2572383