Methabot Web Crawler Icon

User Ratings

ease 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 0 / 5
features 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 0 / 5
design 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 0 / 5
support 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 0 / 5
Write a Review

User Reviews

  • 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5

    Methabot is great! Thanks.

    Posted 06/06/2013
  • 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5

    I looked hard for a web crawler that was as flexible as I wanted. Methabot delivered. It is not full featured so far, but allows the addition of features in a very intelligent and flexible fashion that far outstripped other crawlers and made me very happy. Configure methabots behavior by writing configuration files and javascript parsers. Javascript parsers can be written in a snap. Mix and match the parsers into a chain where each parser can operate on or even change the text as it goes down the line. Write parsers in c language or in javascript. Javascript parsers can be written very quickly. The behavior of the crawler is way more configurable than other parsers. You can fire off different scripts for different file types and switch crawling and parsing behavior in a mechanistic fashion on the fly, use your own very rapidly written javascript E4x parser to intelligently decide which links you want to follow, Extract exactly the information you want in a way that's easy to understand. Methabot rocks. I can't tell you how much better of a crawler this is than the other ones I have looked at. I could not use them to do my project without drastically altering them. Methabot changed all that. The learning curve is not high to make this thing do what you want (assuming you can deal with sql scripts already). Version 1.6 supports mysql with javascript parsers, but 1.7 does not (yet). I wrote a stopwords parser that subtracts common words from the text.. message me if you want it. I'm mgshightech. (I may post it here on sf). version 1.7 supports distributed crawling farms frameworking. redirect the output of Javascript parsers into files to write csv files, or sql scripts for indexing. I may write an indexing parser for this in c language down the road. Enjoy both the speed and power of c language and the quick development times of javascript as this system gives you both. My complements to the original software engineers on a fine architecture. This thing will go far. mgshightech dated 9-21-11

    Posted 09/21/2011

Thanks for helping keep SourceForge clean.

Screenshot instructions:
Red Hat Linux   Ubuntu

Click URL instructions:
Right-click on ad, choose "Copy Link", then paste here →
(This may not be possible with some types of ads)

More information about our ad policies

Briefly describe the problem (required):

Upload screenshot of ad (required):
Select a file, or drag & drop file here.

Please provide the ad click URL, if possible:

Get latest updates about Open Source Projects, Conferences and News.

No, thanks