Larbin is a Web crawler intended to fetch a large number of Web pages, it should be able to fetch more than 100 millions pages on a standard PC with much u/d. This set of PHP and Perl scripts, called webtools4larbin, can handle the output of Larbin and p

http://blogs.ibibo.com/hackdaweb/How-to-get-anything-from-the-Web.html

Database, Dynamic Content, Indexing/Search