#1 Questions

[Frieda Chrysler]

Hi!

I'm looking for a web crawler that'll handle a group that
I'm developing a website for. There are literally
thousands of pages on the domain, some dynamically
driven, some not. I need the crawler to index via the
links, rather than indexing files themselves, as we are
using a CMS for the main part of our website.

The site is http://new.ucip.com (and currently under
develoment). We are a large RPG that has been around
for 8 years and we have massive amounts of data to
index and make searchable.

Is this something that can do the job? Or do you have
any suggestions for one that might be able to handle
this?

Thanks!
Frieda