#56 Configuration based Robots.txt

[Peter Ansell]

The webapp configuration files need to be extended to include support for robots.txt query and global level instructions:

Example:

global: /queryplan/, /admin/
query: /links/, /search/