Larbin is an HTTP Web crawler with an easy interface that runs under Linux. It can fetch more than 5 million pages a day on a standard PC (with a good network).
Project Admins:
ailleret
Operating System:
All BSD Platforms (FreeBSD/NetBSD/OpenBSD/Apple Mac OS X), All POSIX (Linux/BSD/UNIX-like OSes), FreeBSD, Linux
License:
GNU General Public License (GPL)
Category:
WWW/HTTP
Buy expert services from Sourceforge.net Marketplace. Support from the people who know.