The determination of the time it will take to complete the whole crawling-process is almost impossible since the number of douments to request is always unkknown (it can increase after every crawled document). Also, the size of the documents in the queue is unknown.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Is it possible to see the time for the crawl?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi!
What exaclty do yo mean?
The time a cawling-process took after it is finshed?
Or the time remaining till it finishes?
Or something else?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
The time for the download for a single website and the average time to dowload all.
Hi,
please take a look at the documentation of the PHPCrawlerDocumentInfo-Class, section "benchmarks", maybe you'll find there what you need.
http://phpcrawl.cuab.de/classreferences/PHPCrawlerDocumentInfo/overview.html
Also see the PHPCrawlerProcessReport-Class, section "benchmarks" too.
http://phpcrawl.cuab.de/classreferences/PHPCrawlerProcessReport/overview.html
The determination of the time it will take to complete the whole crawling-process is almost impossible since the number of douments to request is always unkknown (it can increase after every crawled document). Also, the size of the documents in the queue is unknown.