Search-Spider

This software is an enhanced version of code authored by Manu Konchady and published in Dr Dobbs in 2007.
This code is best described as experimental (ver .00). It is also inefficient, clumsy, and slow. You will need at least a working knowledge of Perl, postgresql and sql, and of boolean algebra, just to get it to run in its present form, and it runs only on Linux. So why bother?
Perhaps you want to find out how a search engine works;
You may be curious and want to estimate the efficiency of your searches;
Maybe you have a particular kind of research you want to complete;
You might need a particular boolean query for an efficient search;
You may want complete control of where your spider searches;
You may want to avoid, as far as possible, searches routed through proprietary sites.

If any of the above reasons strike a chord, SSpider may be what you are searching for.