Search Results for "crawling"

YaCy is a free search engine that anyone can use to build search the internet (www and ftp) or to create a search portal for others (internet or intranet). The scale of YaCy is limited only by the number of users and can index billions of web pages. In p2p mode it is fully decentralized, all users of the search engine network are equal and it is not possible for anyone to censor the content of the distributed index.

...Powered by Headless Chrome, the crawler provides simple APIs to crawl dynamic websites. Support both depth-first search and breadth-first search algorithm. Save screenshots for the crawling evidence, emulate devices and user agents, priority queue for crawling efficiency, obey robots.txt, and more. The static crawlers are based on simple requests to HTML files. They are generally fast, but fail scraping the contents when the HTML dynamically changes on browsers. Dynamic crawlers based on PhantomJS and Selenium work magically on such dynamic applications. ...

...The goal is to expose a few simple methods that mimic user actions (like goto, type and click), with an API that feels synchronous for each block of scripting, rather than deeply nested callbacks. It was originally designed for automating tasks across sites that don't have APIs, but is most often used for UI testing and crawling. Segment started with an open source project. Since then, we’ve open sourced hundreds of our repos. We want to continue supporting the community by publishing our code and other developers’ awesome open source projects. We use these open source projects as the foundations of our infrastructure to handle billions of API calls per day and to allow us to rapidly build and test code on the client.

...Added Facebook Bot Support for "Facebot/1.0". Add this code to your website to prevent unauthorized stealing of your valuable content & block malicious bots from crawling your site.

This module provides attack surface reduction enhancements against the HTTP Flood Attacks at the web application level. Massive crawling/scanning tools, HTTP Flood tools can be detected and blocked by this module via htaccess, firewall or iptables, etc. (like mod_evasive) You can use this module by including "iosec.php" to any PHP file which wants to be protected. You can test module here: http://www.iosec.org/test.php (demo) Watch the Proof of Concept video: http://goo.gl/dSiAL Hakin9 IT Security Magazine Article about IOSEC http://goo.gl/aQM4Di (different format -> http://goo.gl/JKMUPN) IJNSA Article at http://goo.gl/LLxRdX WP Plugin Page http://goo.gl/nF5nD CHANGES v.1.8.2 - Iptables Auto Ban Bash Script Included - Token Access via Implicit Deny - Reverse Proxy Support - reCAPTCHA Support Do you want more features? ...

A simple library for crawling the web. This library will give you the ability to create macros for crawling web site and preforming simple actions like preforming "log in" and other simple actions in web sites.

Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net

Agent based Regional Crawler strategy implementation - gathers users' common needs and interests in a certain domain. It crawls based on these interests, instead of crawling the web without any predefined order.

Combine is an open system for crawling Internet resources. It can be used both as a general and focused crawler. If you want to download Web-pages pertaining to a particular topic (like 'Carnivorous Plants') Then Combine is the system for you!

MPI implementation using PHP, project still under development... This will give you a speed-up if your code relies alot on bandwidth(eg: crawling the web), i think its useless if your program doesnt require internet.