Showing 245 open source projects for "crawler"

View related business solutions
  • MongoDB Atlas runs apps anywhere Icon
    MongoDB Atlas runs apps anywhere

    Deploy in 115+ regions with the modern database for every enterprise.

    MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
    Start Free
  • Build Securely on Azure with Proven Frameworks Icon
    Build Securely on Azure with Proven Frameworks

    Lay a foundation for success with Tested Reference Architectures developed by Fortinet’s experts. Learn more in this white paper.

    Moving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
    Download Now
  • 1
    Addons for IOSEC - DoS HTTP Security

    Addons for IOSEC - DoS HTTP Security

    IOSec Addons are enhancements for web security and crawler detection

    IOSEC PHP HTTP FLOOD PROTECTION ADDONS IOSEC is a php component that allows you to simply block unwanted access to your webpage. if a bad crawler uses to much of your servers resources iosec can block that. IOSec Enhanced Websites: https://www.artikelschreiber.com/en/ https://www.unaique.net/en/ https://www.unaique.com/ https://www.artikelschreiber.com/marketing/ https://www.paraphrasingtool1.com/ https://www.artikelschreiben.com/ https://www.buzzerstar.com/ https://iosecaddons.sourceforge.io/ https://www.artikelschreiber.com/opensource/ https://www.sebastianenger.com/ Added Setup Instructions for Wordpress Content Management System. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 2

    sitecheck

    Modular web site spider for web developers.

    More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation errors and accessibility errors. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 3
    GHZ Tools v0.6 Build 9645 Release Data (02/09/2014) 7zPass: MHg2NzY4N0E3NDZGNkY2QzczMzAzNj== (base64/hex) Properties: 1)- Brute Forcer: WordPress Joomla 4images osCommerce Drupal, Razor Ftp cPanel Whmcs DirectAdmin Authentication Bypass SSH Authentication vBulletin Kleeja OpenCart WordPress Xmlrpc 2)- Remote Exploits: JCE Webdav 3)- SQL Injector: Auto SQL Injection 4)- Hash Cracker: MD2 MD4 MD5 SHA1 MD5(MD5(PASS)) SHA1(SHA1(PASS)) 5)- URL Fuzzer: URL Fuzzer 6)- Web Scanner: RFI/LFI URL Scanner Web Extractor Open Port Scanner URL Crawler SQLi Scanner
    Downloads: 0 This Week
    Last Update:
    See Project
  • 4

    Domain Analyzer Security Tool

    Finds all the security information for a given domain name

    Domain analyzer is a security analysis tool which automatically discovers and reports information about the given domain. Its main purpose is to analyze domains in an unattended way.
    Downloads: 0 This Week
    Last Update:
    See Project
  • Build Agents and Models on One Platform Icon
    Build Agents and Models on One Platform

    Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

    Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
    Try It Free
  • 5
    Zoozle Search & Download Suchmaschine

    Zoozle Search & Download Suchmaschine

    Zoozle 2008 - 2010 Webpage, Tools and SQL Files

    Download search engine and directory with Rapidshare and Torrent - zoozle Download Suchmaschine All The files that run the World Leading German Download Search Engine in 2010 with 500 000 unique visitors a day - all the tools you need to set up a clone. Code Contains: - PHP Files for zoozle - Perl Crawler for gathering new content to database and all other cool tools i have created https://www.artikelschreiber.com/en/ https://www.unaique.net/en/ https://www.unaique.com/ https://www.unaique.de/ https://www.openinsider.de/ https://www.artikelschreiber.com/ https://zoozle.sourceforge.io/ https://www.artikelschreiben.com/ https://www.artikelschreiber.com/opensource/ It was very successful and 2008/2009 one of the biggest German download engines! ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 6
    webStraktor is a programmable World Wide Web data extraction client. Its purpose is to scrape HTML based content via the HTTP protocol and extract relevant information. webStraktor features a scripting language to facilitate the collection, the extraction and the storage of information available on the web, including images. The scripting language uses elements of the Regular Expression and xPath syntax. The webStraktor scripting language has a small instruction set and its syntax is easy...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 7
    ARCOMEM

    ARCOMEM

    Semantic and social web crawling

    ...Throughout the project a large number of components have been developed to collect content from Web and Social Web, to analyse it from semantic and social perspectives and to enable Web archive access by different facets. The whole system based on the Heritrix crawler is released as open source to the public. Since many components or composite tools are of interest also for other areas and usage scenarios, the ARCOMEM consortium defined a number of pre-packaged tools which can be used independently from each other. By combining all packages the full ARCOMEM system can be build. The following major packages will be released in the coming weeks as pre-compiled packages with source code. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 8

    pytomo

    Tomography tool working as a YouTube crawler

    Usage: start_crawl.py [-b --batch] [-u max_crawled_url] [-r max_rounds] [-l, --loop|--no-loop] [-R --related|--no-related] [-p max_per_url] [-P max_per_page] [-s {youtube, dailymotion}] [--snmp] [-t time_frame] [-n ping_packets] [-D download_time] [-S delay_between_requests] [-x, --no-log-ip] [-c, --no-centralize] [--http-proxy=http://proxy:8080] [--provider=MY_ISP] [--download-extra-dns] [-L log_level] [-f, --input_file input_file_list] [input_urls]
    Downloads: 0 This Week
    Last Update:
    See Project
  • 9

    SauceWalk Proxy Helper

    Enumeration and automation of file discovery for your sec tools.

    ...The remote agent can be used to identify target files and folders on a live system via a PHP script on the target server(ASP/JSP coming soon). The advantage of this tool is that it allows access to files and folders (for example include or plugin folders) which are not usually seen via a spider or crawler to be security tested with traditional tools. The Py version is on its way soon.
    Downloads: 0 This Week
    Last Update:
    See Project
  • Custom VMs From 1 to 96 vCPUs With 99.95% Uptime Icon
    Custom VMs From 1 to 96 vCPUs With 99.95% Uptime

    General-purpose, compute-optimized, or GPU/TPU-accelerated. Built to your exact specs.

    Live migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
    Try Free
  • 10
    anonme.sh

    anonme.sh

    anonymous tools [uncontinued]

    anonme.sh {bash script} V1.0 Operative Systems Suported: Linux Dependencies: slowloris macchanger decrypter.py description of the script * this script makes it easy tasks such as DoS attacks, change you MAC address, inject XSS on target website, file upload vulns, MD5 decrypter, webcrawler (scan websites for vulns) and we can use WGET to download files from target domain or retrieve the all website... tutorial:http://www.youtube.com/watch?v=PrlrBuioCMc
    Downloads: 0 This Week
    Last Update:
    See Project
  • 11

    A simple Crawler

    We can make a simple crawler with using Java Servlet & JSP . A crawl

    This project consists of 1 main directory, 4 sub directories , 1 main page(html) , 1 servlet DD , 4 java files , 7 classes , and 1 document file as the list below: - [hw5] --- index.html - readme.txt - [WEB-INF] --- web.xml - [classes] --- [mvc] --- HelloController.java - HelloController.class - HelloModel.java - HelloMOdel.class - HelloView.java - HelloView.class - HelloResult.java - HelloResult.class - Bfs.class - Queue.class - WebSource.class - [hw5] - [WEB-INF] - [classes] - [mvc] - index.html ( first page for crawler ) - web.xml ( the configuraion of all servlet ) - HelloController.java ( process the HTTP request and response ) - HelloModel.java ( main process and crawler , url match ) - HelloView.java ( show the result of crawler and search) - HelloResult.java ( show the search result)
    Downloads: 0 This Week
    Last Update:
    See Project
  • 12
    xpider

    xpider

    An extensible web spider (crawler) for Joomla!

    The extensible web spider (Xpider) is Joomla! component that tries to make the crawling of external webpages possible for you. It is possible to create a Spider and give it some Tasks (data to find) and some Seeds (web addresses) to search on. The Spider's Finding (the result of finding the tasks) is possible to link to a database.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 13

    Battlelance

    Multiplayer dungeon crawler

    Battlelance is a turn based multiplayer RPG game based on DnD and the likes. You can control a wizard, fighter, thief, cleric or a whole party of these and pit them against a dungeon master or other characters/parties. You can also take up the role as dungeon master and spawn enemies, traps and do other nasty tricks to make life of the characters as difficult as possible.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 14
    Constellio Enterprise Search engine

    Constellio Enterprise Search engine

    Open source Search Engine and Enterprise Search

    Constellio is an enterprise search engine that allows companies to search all their organization's information through a single interface (Web, CRM, ERP, ECM, Mail etc.). Constellio is Based on Apache Solr and Google Search Appliance's connector. Constellio has a powerful web crawler.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 15
    TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 17 different attributes.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 16

    Ancient World Of Generica

    A Current Text combat game transitioning to Dungeon Crawler/Roguelike

    This is a little game i have programmed for a while in python and compiled with pyinstaller. It is a very early alpha with only a few feature to boot in the first internet release version. These include: -Rudimentry Saving and loading -Weapons, armor and potions(but no inventory to store them in as of yet) - A modding interface for the above -Basic Text combat and morale implemented for a basic monster logic - Basic procedural generation for the above -Classes and Races This at the...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 17
    Blog Crawler System
    Downloads: 0 This Week
    Last Update:
    See Project
  • 18
    Arch Crawler

    Arch Crawler

    Arch Crawler is a pre-configured install of Arch Linux

    Arch Crawer is an pre-configured Arch Linux install that is based around the fluxbox window manager. http://www.archlinux.org https://wiki.archlinux.org/index.php/Fluxbox
    Downloads: 0 This Week
    Last Update:
    See Project
  • 19
    Yet another web crawler? Yes, but this ones uses the full power of regular expressions to accept or reject, examine or ignore, save or refuse pages. You also use MIME types to do all this. Powerful and flexible.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 20
    Downloads: 0 This Week
    Last Update:
    See Project
  • 21

    tescoShooping

    crawler - http://ezakupy.tesco.pl/

    crawler - http://ezakupy.tesco.pl/ aplikacja z funkcją cachowania obrazków, gromadzi je w katalogu cache. w celu wyczyszczenia należy skasować zawartość. z tego też względu za pierwszym razem kategorie mogą się długo ładować
    Downloads: 0 This Week
    Last Update:
    See Project
  • 22
    This 5 generation selenium web crawler crawl through web page of a host website searching for static and dynamic links and able to detect honeypot links.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 23
    A web search engine and crawler written in java/mysql, fulltext and vertical search, word segmentation system .
    Downloads: 0 This Week
    Last Update:
    See Project
  • 24
    Ani-Shell
    Ani-Shell is a PHP remote shell, basically used for remote access and security pen testing. Ani-Shell provides a robust and a basic interface to access the file system, do some networking tweaks and even test your server for some common security vulnerabilities. The developer has tried to follow a coding standard which makes the code a little clean and easier to understand, Note: How you use this shell is exactly on you, and author pays no responsibility for what you use it for and what may...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 25
    PRO-Search is a crawler of FTP servers, SMB shares, HTTP, dc++ networks, ... with powerful web search and navigation interface
    Downloads: 0 This Week
    Last Update:
    See Project
MongoDB Logo MongoDB