An attempt at a Xanalogical Web, comprised of a CGI system with both native and HTML interfaces, a set of standards, a backend supporting Transclusion, versioning, and a custom browser with support for Transpointing and editing.
A Lua-based crawling scripting language and leveraging selenium
I needed a way to crawl a site, crawling using commands. I would put commands in a file or DB to use selenium to interpret the HTML and Javascript. The best would be to have a complete language with conditionals and looping. I'm a java developper and I needed that the crawler to run in a Spring-Boot application. So I decided to use a Lua interpreter in Java to build a crawling tool based on Selenium.
The trick here is to add the crawling commands into the Lua interpreter.