Search Results for "web crawler bot"
Sort By:
26 projects for "web crawler bot" with 2 filters applied:
-
Secure File Transfer for Windows with Cerberus by RedwoodCerberus supports unlimited users and connections on a single IP, with built-in encryption, 2FA, and a browser-based web client — all deployable in under 15 minutes with a 25-day free trial.
-
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
3
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
4
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
grammY
The Telegram Bot Framework
grammY makes it easy to create Telegram bots. Both for beginners and at scale. You want grammY because it is easy to use. It is very powerful and always up to date. It has the best documentation in town. It is extremely efficient and scales up effortlessly. It has a thriving ecosystem of plugins, a friendly community chat, seamless integrations with web frameworks and databases, and so much more. -
6
Cap CAPTCHA
The privacy-first, self-hosted CAPTCHA for the modern web
...It replaces invasive tracking-based CAPTCHA solutions with a lightweight approach based on proof-of-work and client-side instrumentation. The system can be deployed independently, giving developers full control over user data and verification processes. It integrates easily into web applications and focuses on maintaining usability while preventing automated abuse. Cap avoids reliance on centralized providers, reducing dependency on external services and improving privacy compliance. Its design reflects a shift toward decentralized, user-respecting security mechanisms. It is particularly suited for developers who want transparent and customizable bot protection. -
7
Zendriver
A blazing fast, async-first, undetectable webscraping
Zendriver is a modern Python web automation and scraping framework that leverages the Chrome DevTools Protocol to provide fast, asynchronous control over real browser instances. Unlike traditional tools that rely on Selenium or WebDriver, Zendriver communicates directly with the browser through CDP, enabling higher performance and more precise control over browser behavior. -
8
React ChatBotify
A modern React library for creating flexible and extensible chatbots
Welcome to React ChatBotify, an intuitive and versatile library that allows you to build responsive chatbots with ease. Whether you're aiming to create a simple FAQ bot or a sophisticated conversational interface integrated with Large Language Models (LLMs), React ChatBotify provides the tools and flexibility you need to bring your vision to life. -
9
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
10
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
11
reCAPTCHA
PHP client library for reCAPTCHA, a free service
...The ecosystem supports mobile and enterprise variants, but the repo focuses on common web integrations and best practices for verifying the token securely. Deployed correctly, reCAPTCHA reduces credential stuffing, bot sign-ups, and scraping without degrading the experience for typical users. -
12
tirreno
Security framework for devs and product teams
tirreno is a security framework. tirreno [tir.ˈrɛ.no] helps understand, monitor, and protect your product from threats, fraud, and abuse. While classic cybersecurity focuses on infrastructure and network perimeter, most breaches occur through compromised accounts and application logic abuse that bypasses firewalls, SIEM, WAFs, and other defenses. tirreno detects threats where they actually happen: inside your product. tirreno is a few-dependency, "low-tech" PHP/PostgreSQL application.... -
13
koa-isbot
Fast Middleware detect bot crawler for Koa
Koa detects robots. Fast Middleware detects bot crawler for Koa. -
14
Mojo-Webqq
Perl-based smart QQ client framework
Mojo-Webqq is an unofficial Perl client and bot framework for the QQ messaging platform’s web endpoints, built on the Mojolicious real-time web stack. It manages login flows, session refresh, and long-polling or websocket-style message reception to deliver events into an asynchronous event loop. Plugins and hooks let developers implement auto-replies, keyword filters, logging, forwarding, and integrations with external services or databases. -
15
A Java Web crawler: multi-threaded, scalable, with high performance, extensible and polite. It can be used to crawl and index any web or enterprise domain and is configurable through a XML configuration file.
-
16
DotNetWikiBot Framework
Make robots for MediaWiki-powered sites!
The DotNetWikiBot Framework was developed so that it can offer a helping hand with many complicated and routine tasks of wiki site development and maintenance. DotNetWikiBot Framework is a cross-platform full-featured client API, that allows you to build programs and web robots easily to manage information on MediaWiki-powered sites. DotNetWikiBot Framework can also be used for learning C# and .NET. -
17
sitecheck
Modular web site spider for web developers.
More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation... -
18
Zoozle Search & Download Suchmaschine
Zoozle 2008 - 2010 Webpage, Tools and SQL Files
Download search engine and directory with Rapidshare and Torrent - zoozle Download Suchmaschine All The files that run the World Leading German Download Search Engine in 2010 with 500 000 unique visitors a day - all the tools you need to set up a clone. Code Contains: - PHP Files for zoozle - Perl Crawler for gathering new content to database and all other cool tools i have... -
19
** Guys I have built a much more powerful Fully Featured CMS system at: https://github.com/MacdonaldRobinson/FlexDotnetCMS Macs CMS is a Flat File ( XML and SQLite ) based AJAX Content Management System. It focuses mainly on the Edit In Place editing concept. It comes with a built in blog with moderation support, user manager section, roles manager section, SEO / SEF URL
-
20
Globsy is a PHP bot framework for the Google Talk network. Globsy can be executed from a regular browser on a PHP enabled server, connect to the Google Talk network and continue running in the background, without the browser needing to remain open.
-
21
yuik is an abstract widget library for creating GUIs. The actual frontend can be chosen at runtime which allows the program to run under Tk, as a shell program, and even as a web-application. More frontends like an irc-bot, wxWidgets, or XUL are planned.
-
22
-
23
JIB - Java IM Bot, aimed to built a pluggable, customizable, expandable and dynamic bot framework for instant messager. With JIB user can control remote IM Bots over web based interface, configure bots to perform various tasks or custom behavior.
-
24
Spiritbot is a suite of tools designed for use in software-engineering, CodeGenUML includes an XMI/MOF/UML reader and exporter. An IRC ticker bot is also provided to aid in the development process.
-
25