I want to scrape data from a site that requires an http-param named "ADD". I have 1000's of addresses in a file (either a CSV file or an XML file, whichever makes more sense). For each address i need to submit the address and then data-mine the returned data. Any idea how i can write a config file to submit each of those addresses?
<var-def name="baseUrl">http://www5.metrokc.gov/reports/</var-def><var-def name="property">property_report.asp</var-def><var-def name="id">1</var-def><var-def name="address">12325 38TH AVE NE</var-def><var-def name="content"><html-to-xml><!-- Gets the data from the county's site --><http method="post" url="${sys.fullUrl(baseUrl,property)}"><http-param name="ADD"><var name="address"/></http-param><http-param name="T1">Search</http-param></http></html-to-xml></var-def><file action="write" path="KingCounty/data/parcel.xml" charset="UTF-8"><xquery><xq-param name="doc"><var name="content"/></xq-param><!-- replace the "item" below with the xpath to the table containing the values --><!-- <xq-param name="item" type="node()"><var name="/html/body/div/table[11]/tbody"/></xq-param> --><xq-expression><![CDATA[
declare variable $doc as node() external;
let $address := data($doc/html/body/div/table[11]/tbody/tr[1]/td[2])
let $parcelNo := data($doc/html/body/div/table[2]/tbody/tr[1]/td[4]/b/font)
let $yr := data($doc/html/body/div/table[11]/tbody/tr[4]/td[2])
let $sf := data($doc/html/body/div/table[11]/tbody/tr[6]/td[2])
return
<parcel><parcelNo>{normalize-space($parcelNo)}</parcelNo><address>{normalize-space($address)}</address><yr>{normalize-space($yr)}</yr><sf>{normalize-space($sf)}</sf></parcel>
]]></xq-expression></xquery></file>
</config>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I want to scrape data from a site that requires an http-param named "ADD". I have 1000's of addresses in a file (either a CSV file or an XML file, whichever makes more sense). For each address i need to submit the address and then data-mine the returned data. Any idea how i can write a config file to submit each of those addresses?
Any ideas would be appreciated.
Here's my file to download a single address:
<?xml version="1.0" encoding="UTF-8"?>
<config charset="ISO-8859-1">
</config>