WebHarvest - web data extraction tool / Discussion / Help: Help harvesting Google Scholar

Rodrigo Rech - 2006-12-20

Hello! I'm trying to harvest Google Scholar (http://scholar.google.com)

The data i'm interested: title of the reference and the number of citations.

The problem is that sometimes the text of a reference's title is a link, sometimes just a text, without any html tags around it.

The code below shows the XML version generated by Web-Harvest of the HTML sent by Google Scholar:

<p class="g">
<span class="w">
<a
href="/url?sa=U&q=http://www8.org/w8-papers/2b-customizing/user/user.html">
User Adaptable Multimedia resentations for the WWW // THIS IS THE TITLE!!!!!
</a>
</span>
...
</p>

To harvest the title, i'm using the following Xpath expression:
let $title := data($item//span[@class='w']/a)

It works fine in this situation!

But sometimes the title is at another place:

<p class="g">
<font size="-2">
<b>[CITATION]</b>
</font>
Modeling of Courses through Workflow using the standard SVG/XML // THIS IS THE TITLE NOW!!!!!
<font size="-1">
...
</p>

There are no html tags around the title... And i can't figure out the Xpath expression that would grab it.

Any ideas?

I can send the full web-harvest configuration file i'm using, just send me a message here. Of course, the file can be distributed in the next versions of Web-Harvest, no problem!

Thanks a lot!!!
Rodrigo - rorech@inf.ufrgs.br

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Rodrigo Rech - 2006-12-22
  
  Still can't figure out a Xpath expression to harvest the title of the reference below :(
  
  <p class="g">
  <font size="-2">
  <b>[CITATION]</b>
  </font>
  Modeling of Courses through Workflow using the standard SVG/XML
  <font size="-1">
  
  ...
  
  </font>
  
  </p>
  
  I have tried many expressions, the best currently is:
  let $title2 := data($item//text()[2])
  
  But then i can't use {normalize-space($title2)}, an exception is throw saying that there are more than one item.
  
  Thanks for any help!!!
  Rodrigo
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Rodrigo Rech - 2006-12-20
  
  Example of a configuration file i'm using to harvest Google Scholar:
  
  <?xml version="1.0" encoding="UTF-8"?>
  
  <config charset="UTF-8">
  
  <include path="functions.xml"/> <var-def name="url"> <template>http://scholar.google.com/scholar?q=valdeni&amp;hl=en&amp;lr=&amp;btnG=Search</template> </var-def> <var-def name="referencias"> <call name="download-multipage-list"> <call-param name="pageUrl"><var name="url"/></call-param> <call-param name="nextXPath">//td[.='Next']/a/@href</call-param> <call-param name="itemXPath">//p[@class="g"]</call-param> <call-param name="maxloops">5</call-param> </call> </var-def> <file action="write" path="scholar/scholar.xml" charset="UTF-8"> <![CDATA[ <?xml version="1.0" encoding="UTF-8"?> <scholar> ]]> <loop item="item" index="i"> <list><var name="referencias"/></list> <body> <xquery> <xq-param name="item"><var name="item"/></xq-param> <xq-expression><![CDATA[ let $title := data($item//span[@class='w']/a) let $citacoes := data($item//font[@size='-1']/font[@color='#7777CC']/a[contains(@href, 'cites=')]) return <referencia> <title>{normalize-space($title)}</title> <citacoes>{normalize-space($citacoes)}</citacoes> </referencia> ]]></xq-expression> </xquery> </body> </loop> <![CDATA[ </scholar> ]]> </file>
  
  </config>
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Rodrigo Rech - 2006-12-20
  
  Another configuration file, just generate a XML file representing the HTML of Google Scholar:
  
  <?xml version="1.0" encoding="UTF-8"?>
  
  <config charset="UTF-8">
  
  <var-def name="url"> <template>http://scholar.google.com/scholar?q=valdeni&amp;hl=en&amp;lr=&amp;btnG=Search</template> </var-def> <file action="write" path="scholar/layoutScholar.xml" charset="UTF-8"> <html-to-xml> <http url="${url}" charset="UTF-8"/> </html-to-xml> </file>
  
  </config>
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Help harvesting Google Scholar

Forums

Help

Help harvesting Google Scholar document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Help harvesting Google Scholar