GOOGLE, METAR, TAF, and WEATHER commands broken.

The IRC's Talking Robot

Status: Beta

Brought to you by: luizcb

#3 GOOGLE, METAR, TAF, and WEATHER commands broken.

Milestone: 8rc4

Status: done

Owner: onefang

Labels: None

Priority: high

Updated: 2014-03-10

Created: 2006-12-12

Creator: onefang

Private: No

The google command returns nothing. Likely google have changed things, again. Same applies to all the other web scraping commands. Fair enough, web page scraping is fragile.

Related

Discussion

onefang - 2014-02-24

Milestone: --> 8rc4

Priority: 5 --> high
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-02-24

I think all the web scraping functions are broken. Might be better to replace them with what ever sort of web API the various web sites provide. Might also be good to replace them all with a generic Lua thing if we add Lua scripting.

Either way, until we move to using web API (a big job) any fixes here will be temporary, as web page scraping is fragile.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

summary: google comand broken. --> GOOGLE, METAR, TAF, and WEATHER commands broken.

Description has changed:

Diff:

--- old +++ new @@ -1 +1 @@ -The google command returns nothing. Likely google have changed things, again. +The google command returns nothing. Likely google have changed things, again. Same applies to all the other web scraping commands. Fair enough, web page scraping is fragile.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

They all need to be addressed to the bot.

WEATHER works fine.

METAR and TAF use the same web site, which has changed, but now they have a new XML based API - http://www.aviationweather.gov/adds/dataserver

GOOGLE got complicated, lots of javascript and stuff to wade through, but they might have an API.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

Google has an API, but it's a paid for service. https://developers.google.com/custom-search/json-api/v1/overview

Web page scraping like we did in the past is apparently against the Google Terms Of Service.

Might be time to drop the GOOGLE command and find something else.

http://stackoverflow.com/questions/6084096/what-free-web-search-apis-are-available

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

Lots of choices - http://www.programmableweb.com/apis/directory/1?apicat=Search

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

https://duckduckgo.com/api seems OK, though it's not a web search engine as such. It does provide "instant answers", which fits better with what Darkbot does. On the other hand, the result for http://api.duckduckgo.com/?q=darkbot&format=xml leaves a lot to be desired. lol

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

We will go with converting METAR and TAF to use their new XML API, and using XML with duckduckgo. I can reuse my XMLame parser I wrote long ago.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

status: open --> working-on-it

assigned_to: David Seikel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-09

https://sourceforge.net/p/darkbot/darkbot/ci/c647a6e1ee5ce82924a3cb41889ce4accbdfc979/ fixes METAR and TAF.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-10

GOOGLE is now WEBSEARCH in commit https://sourceforge.net/p/darkbot/darkbot/ci/891588a96a38b9ae4c819add3213d1ec98efa1e5/

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

onefang - 2014-03-10

status: working-on-it --> done
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Log in to post a comment.