<includepath="functions.xml"/><!-- defines search keyword and start URL --><setvar="search">platon</set><!-- xpath fails no warning--><setvar="url"><xpathexpression="//noscript//a/@href[1]"><html-to-xml><httpurl="http://images.google.com/images?q=${search}&hl=en&btnG=Search+Images"/></html-to-xml></xpath></set><!-- collects all image URLs --><setvar="imgLinks"><callname="download-multipage-list"><call-paramname="pageUrl"><getvar="url"/></call-param><call-paramname="nextXPath">//td[.='Next']/a/@href</call-param><call-paramname="itemXPath">//img[contains(@src, 'images?q=tbn')]/@src</call-param><call-paramname="maxloops">5</call-param></call></set><!-- download images and saves them to the files --><loopitem="link"index="i"filter="unique"><list><getvar="imgLinks"/></list><body><fileaction="write"type="binary"path="/home/metronome/harvest/google_images/${search}_${i}.gif"><httpurl="${sys.fullUrl(url,link)}"/></file></body></loop>
</config>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
<config xmlns="http://web-harvest.sourceforge.net/schema/2.1/core" xmlns:var="http://web-harvest.sourceforge.net/schema/2.1/var" xmlns:p="http://web-harvest.sourceforge.net/schema/2.1/param">
<!--
Download multi-page list of items.
@param pageUrl - URL of starting page
@param itemXPath - XPath expression to obtain single item in the list
@param nextXPath - XPath expression to URL for the next page
@param maxloops - maximum number of pages downloaded
@return list of all downloaded items
-->
<functionname="download-multipage-list"><return><whilecondition="${pageUrl.toString().length()!=0}"maxloops="${maxloops}"index="i"><empty><setvar="content"><html-to-xml><httpurl="${pageUrl}"/></html-to-xml></set><setvar="nextLinkUrl"><xpathexpression="${nextXPath}"><getvar="content"/></xpath></set><setvar="pageUrl"><template>${sys.fullUrl(pageUrl.toString(),nextLinkUrl.toString())}</template></set></empty><xpathexpression="${itemXPath}"><getvar="content"/></xpath></while></return></function>
</config>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
<!-- collects all tables for individual products --><setvar="products"><xpathexpression ='//li[@class="hproduct" or @class="hproduct first" or @class="hproduct last"]'><html-to-xml><httpurl ="http://shopping.yahoo.com/s:Digital%20Cameras:4168-Brand=Canon:browsename=Canon%20Digital%20Cameras:refspaceid=96303108;_ylt=AnHw0Qy0K6smBU.hHvYhlUO8cDMB;_ylu=X3oDMTBrcDE0a28wBF9zAzk2MzAzMTA4BHNlYwNibmF2"/></html-to-xml></xpath></set><!-- iterates over all collected products and extract desired data --><fileaction="write"path="/home/metronome/harvest/canon/single_page_catalog.xml"charset="UTF-8"><![CDATA[ <catalog> ]]><loopitem="item"index="i"><list><getvar="products"/></list><body><xquery><xq-paramname="item"type="node()"><getvar="item"/></xq-param><xq-expression><![CDATA[ declare variable $item as node() external; let $name := data($item//*[@class='title']) let $desc := data($item//*[@class='desc']) let $price := data($item//*[@class='price']) return <product> <name>{normalize-space($name)}</name> <desc>{normalize-space($desc)}</desc> <price>{normalize-space($price)}</price> </product> ]]></xq-expression></xquery></body></loop><![CDATA[ </catalog> ]]></file>
</config>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It is in fact the "namespacesaware" attribute of html-to xml processor whose default value is :
true in 2.0 doc ( namual )
true in 1.1 code ( xsd )
but is false in 2.0 code !!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'v tried several configs and most of them return empty xpath
I've modified the google_image example , it runs , but does not collect anything
<config xmlns="http://web-harvest.sourceforge.net/schema/2.1/core" xmlns:var="http://web-harvest.sourceforge.net/schema/2.1/var" xmlns:p="http://web-harvest.sourceforge.net/schema/2.1/param">
</config>
<config xmlns="http://web-harvest.sourceforge.net/schema/2.1/core" xmlns:var="http://web-harvest.sourceforge.net/schema/2.1/var" xmlns:p="http://web-harvest.sourceforge.net/schema/2.1/param">
<!--
Download multi-page list of items.
</config>
But this one works though
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/2.1/core" xmlns:var="http://web-harvest.sourceforge.net/schema/2.1/var" xmlns:p="http://web-harvest.sourceforge.net/schema/2.1/param">
</config>
In fact , after many tests , the Bug seems to be in
<html-to-xml>
htmlcleaner generate wrong xml documents
then the parsing fails
In fact , after many tests , the Bug seems to be in
<html-to-xml>
htmlcleaner generate wrong xml documents
then the parsing fails
It is in fact the "namespacesaware" attribute of html-to xml processor whose default value is :
true in 2.0 doc ( namual )
true in 1.1 code ( xsd )
but is false in 2.0 code !!