#7 Add Lua scripting, and convert the web scrapers to use it.

[onefang]

This is a copy of something I posted elsewhere, which takes a while to get to the point -

Having a quick look at our current web site wrappers, we basically have web page scrapers written in C. Web page scraping is fragile, and that can be seen in our google support having broken in the past, and now being broken again. Each time one of the handful of web sites we scrape changes their web pages, we would need to fix the scraping function. This is why things like REST, SOAP, and JSON where
invented. Some web sites publish a stable API that can be used over HTTP to access their functions directly instead of trying to scrape details from ever changing web pages. It would be better if we used those sorts of API.

Allura and Redmine at least have published REST + JSON API to get to most of their functionality. That made it easy to write my migration script. Allura has 500 error code issues at the best of times, sometimes it gets worse. So my script tries to work around that.

I initially looked for a C library, but the one I had my eye on didn't actually support REST or JSON yet, but had those as a TODO item. It would not be suitable for use in Darkbot though. Too many other required libraries. So I next looked for high level scripting language solutions. Lua is my current favourite scripting language, and I found a suitable REST + JSON Lua library, so that's what I used for my migration script.

Darkbot doesn't have Lua, but I think it would be a good idea to add it. Adding a scripting language to Darkbot makes it easy for users to extend the functionality, and if we convert things like google and weather to that scripting language, then people can use them as examples. Then it would be easy enough to use parts of my migration script as support for Allura and Redmine ticket systems. We could even convert the math stuff to just pass maths to Lua and let it deal with that. It would give us a much more sophisticated maths function,
essentially for free. Lua is well sandboxed, could even let ordinary IRC users write short scripts.

I'm not gonna try to sell you on Lua specifically right now, except to say that it's what I would choose coz I really like it, it has a lot to offer, and I've already added Lua scripting to two other projects. It's something to think about later, more important stuff to do first.