Showing 245 open source projects for "crawler"

View related business solutions
  • Ship Agents Faster Icon
    Ship Agents Faster

    Transform your applications and workflows into powerful agentic systems at global scale.

    Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
    Get Started Free
  • Forever Free Full-Stack Observability | Grafana Cloud Icon
    Forever Free Full-Stack Observability | Grafana Cloud

    Our generous forever free tier includes the full platform, including the AI Assistant, for 3 users with 10k metrics, 50GB logs, and 50GB traces.

    Built on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
    Create free account
  • 1

    Web Crawler Security Tool

    A web crawler oriented to information security.

    ...Many of the old features has been reimplemented and the most interesting one is the capability of the crawler to search for directory indexing.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 2

    The Secret of Zankas

    A student game: dungeon crawler

    This is a student game made by: Edmund Lewry, Steepana Stevenson, Adebayo Banjo and Nadia Jahan. Check the dev blog for installation instructions.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 3

    Python Crawler Library

    Python Web Crawler Library

    A simple library for crawling the web. This library will give you the ability to create macros for crawling web site and preforming simple actions like preforming "log in" and other simple actions in web sites.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 4
    The “Media Crawler” is an extensible Eclipse RCP based desktop application which will crawl a given file system, extract metadata from files, map metadata to internal schemas and store the metadata in a databse. This project is ANDS-funded.
    Downloads: 1 This Week
    Last Update:
    See Project
  • Secure File Transfer for Windows with Cerberus by Redwood Icon
    Secure File Transfer for Windows with Cerberus by Redwood

    Protect and share files over FTP/S, SFTP, HTTPS and SCP with the #1 rated Windows file transfer server.

    Cerberus supports unlimited users and connections on a single IP, with built-in encryption, 2FA, and a browser-based web client — all deployable in under 15 minutes with a 25-day free trial.
    Try for Free
  • 5
    The archive-crawler project is building Heritrix: a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 6
    RiverGlass EssentialScanner is an open source web and file system crawler which indexes the text content of discovered files so they can be retrieved and analyzed. It provides simple scanner capabilities as part of larger enterprise search solutions.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 7
    Hound is an application written in python which crawls websites, collecting data such as valid URLs for that site, email addresses, etc. The plugin architecture enables the user to develop new ones easily.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 8
    This game is a mouse-driven dungeon crawler. The player moves the PC around, buys equipment, and attacks enemies using the mouse. More info on the wiki page. Use git to get the code.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 9
    Started as a MSc Project, it is a Tweet crawler and a search engine based on finding relationships to the found results with the help of a page graph generated by the crawling system.
    Downloads: 0 This Week
    Last Update:
    See Project
  • Stop Cyber Threats with VM-Series Next-Gen Firewall on Azure Icon
    Stop Cyber Threats with VM-Series Next-Gen Firewall on Azure

    Native application identity and user-based security for your Azure cloud

    Gain integrated visibility across all traffic in a single pass. Deploy Palo Alto Networks VM-Series to determine application identity and content while automating security policy updates via rich APIs.
    Get a free trial
  • 10
    http://ProxyGuard.net - Proxy Guard, the Block Proxy Script, blocks unwanted proxies from visiting your site. Our crawler add hundreds of proxy sites and IP based proxies to our database daily.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 11
    This is a futur C library like liburl to simply get webpage. The main goal is to make it the most easier possible. this library also provide some basics functions to create crawler. There is also some kind of string.h
    Downloads: 0 This Week
    Last Update:
    See Project
  • 12
    a minimal Java web crawler
    Downloads: 0 This Week
    Last Update:
    See Project
  • 13
    PHP, mySQL and AJAX web crawler
    Downloads: 0 This Week
    Last Update:
    See Project
  • 14
    .Net crawler is a 3 part system written in asp.net It includes a Crawler, Indexing System, and a front end search engine based on lucene. You can also visit us at http://www.gurucoding.com
    Downloads: 0 This Week
    Last Update:
    See Project
  • 15
    Crawler
    Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page. More open source at https://github.com/fcc.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 16
    Crawler to index all websites using google services.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 17
    Pandor
    Pandor is a free search engine written in C + + with spell checker, crawler, search system, ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 18
    A web crawler for crawling news from user specific websites and keywords using PHP and third party software
    Downloads: 0 This Week
    Last Update:
    See Project
  • 19
    Ex-Crawler
    Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net
    Downloads: 0 This Week
    Last Update:
    See Project
  • 20
    crowdspider is a multi-thread web crawler. crowdspider is (just) a web crawler, NOT an indexer. You have to write some code yourself in order to save pages or index them in a database.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 21
    A school project consisting of a crawler, a server and a searchpage.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 22
    ItSucks
    This project is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates. All backend functionalities are also available in a separate library.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 23
    Folksonomy Web Crawler
    A Web crawler prototype designed to index pages of certain resource sharing platforms based on folksonomy tags. The results are displayed in an Excel spreadsheet.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 24
    This is a simple webcrawler for FaceBook (TM) written in Java. The crawler will surf the public user pages (this means that you do not need to provide ann account) to reconstruct the friendship graph for further studies and analises
    Downloads: 0 This Week
    Last Update:
    See Project
  • 25
    MuSE-CIR is a Multigram-based Search Engine and Collaborative Information Retrieval system. Written in Java /JSP, supports any JDBC connectable database - thoroughly tested only with OracleXE, and somewhat with MySQL, JSP on Apache Tomcat 5.5
    Downloads: 0 This Week
    Last Update:
    See Project
MongoDB Logo MongoDB