Search Results for "web crawler bot"
Sort By:
Showing 27 open source projects for "web crawler bot"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
SafeLine
Serve as a reverse proxy to protect your web services from attacks
SafeLine is a self-hosted WAF(Web Application Firewall) to protect your web apps from attacks and exploits. A web application firewall helps protect web apps by filtering and monitoring HTTP traffic between a web application and the Internet. It typically protects web apps from attacks such as SQL injection, XSS, code injection, os command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP-flood, bot abuse, among others. ... -
2
3X-UI
Xray panel supporting multi-protocol multi-user expire day
3X‑UI is an enhanced fork of the X‑UI project, serving as a multifunctional web panel to manage VPN services. It supports multi‑protocol configurations (like VMess, VLESS, Trojan, Shadowsocks, WireGuard), offers traffic and user limits, SSL management, multilingual support, and integrates Telegram notifications for system events. Advanced, open-source web-based control panel designed for managing Xray-core server. It offers a user-friendly interface for configuring and monitoring various VPN... -
3
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
4
Kong
The Cloud-Native API Gateway
Kong is a next generation cloud-native API platform for multi-cloud and hybrid organizations. When building for the web, mobile, or Internet of Things, you’ll need a common functionality to run your software, and Kong is that solution. Kong acts as a gateway, connecting microservices requests and APIs natively while also providing load balancing, logging, monitoring, authentication, rate-limiting, and so much more through plugins. Kong is highly extensible as well as platform agnostic,... -
Stop vibe-debugging.AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.
-
5
WFDownloader App
Free batch downloader for image, wallpaper, video, audio, document,
Use as an image gallery, wallpaper, audio/music, video, document, and other media bulk downloader from supported websites. Also use to download sequential website urls that have a certain pattern (e.g. image01.png to image100.png). Also use app's built-in site crawler for advanced link search or extraction. There is also special support for forum media downloading, forum thread offline archiving, rss feed downloading, and open directory downloading. It's a programmable downloader and also... -
6
AlertsOnWings
Pinger alert
AlertsOnWings is web-based interface for viewing a `heartbeat` of network by pinging hosts. No extra files, only DB. All reports generated "on-the-fly". And we got personal cabinet for each user. -
7
PHP mini vulnerability suite
Multiple server/webapp vulnerability scanner
github: https://github.com/samedog/phpmvs -
8
taskt
Free and open-source robotic process automation (rpa)
taskt (formerly sharpRPA) is the first truly free, easy-to-use, and open-source process automation client built on the .NET Framework in C#. taskt allows you to build and design process automation without needing to write application code. taskt allows you to automate the boring stuff and create efficienies by giving you the power to craft a digital workforce that executes and performs rule-based automation. No API? No Problem! Included is a "what you see is what you get" bot designer with... -
9
OpenSearchServer Search Engine
An open source search engine with RESTFul API and crawlers
OpenSearchServer is a powerful, enterprise-class, search engine program. Using the web user interface, the crawlers (web, file, database, etc.) and the client libraries (REST/API , Ruby, Rails, Node.js, PHP, Perl) you will be able to integrate quickly and easily advanced full-text search capabilities in your application: Full-text with basic semantic, join queries, boolean queries, facet and filter, document (PDF, Office, etc.) indexation, web scrapping,etc. OpenSearchServer runs on... -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
10
pyspider
A powerful Spider(Web Crawler) system in Python
pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. -
11
diskover
File system crawler and disk space usage software
diskover is a file system crawler and disk space usage software that uses Elasticsearch to index your file metadata. diskover crawls and indexes your files on a local computer or remote storage server over network mounts. diskover helps manage your storage by identifying old and unused files and give better insights into data change "hotfiles", file duplication "dupes" and wasted space. It is designed to help deal with managing large amounts of data growth and provide detailed storage... -
12
autoreplybot
an 'auto-reply' bot for your mail server
autoreplybot is designed for mail server administrators wishing to add an easy-to-use, easy-to-implement and uninvasive "autoreply" service for their users. All you need is a PHP environement and a little postgresql database. The concept is fairly simple: * You configure your mail server to send a copy of every mail that it processes to a specific mailbox * You set up autoreplybot to poll this mailbox, either via POP3 or IMAP * Users log on the autoreplybot web panel using... -
13
DotNetWikiBot Framework
Make robots for MediaWiki-powered sites!
The DotNetWikiBot Framework was developed so that it can offer a helping hand with many complicated and routine tasks of wiki site development and maintenance. DotNetWikiBot Framework is a cross-platform full-featured client API, that allows you to build programs and web robots easily to manage information on MediaWiki-powered sites. DotNetWikiBot Framework can also be used for learning C# and .NET. -
14
Addons for IOSEC - DoS HTTP Security
IOSec Addons are enhancements for web security and crawler detection
...Added Facebook Bot Support for "Facebot/1.0". Add this code to your website to prevent unauthorized stealing of your valuable content & block malicious bots from crawling your site. -
15
GHZ Tools v0.6 Build 9645 Release Data (02/09/2014) 7zPass: MHg2NzY4N0E3NDZGNkY2QzczMzAzNj== (base64/hex) Properties: 1)- Brute Forcer: WordPress Joomla 4images osCommerce Drupal, Razor Ftp cPanel Whmcs DirectAdmin Authentication Bypass SSH Authentication vBulletin Kleeja OpenCart WordPress Xmlrpc 2)- Remote Exploits: JCE Webdav 3)- SQL Injector: Auto SQL Injection 4)- Hash Cracker: MD2 MD4 MD5 SHA1 MD5(MD5(PASS)) SHA1(SHA1(PASS)) 5)- URL Fuzzer: URL Fuzzer 6)- Web Scanner: RFI/LFI URL Scanner Web Extractor Open Port Scanner URL Crawler SQLi Scanner
-
16
Python Crawler Library
Python Web Crawler Library
A simple library for crawling the web. This library will give you the ability to create macros for crawling web site and preforming simple actions like preforming "log in" and other simple actions in web sites. -
17
The archive-crawler project is building Heritrix: a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.
-
18
RiverGlass EssentialScanner is an open source web and file system crawler which indexes the text content of discovered files so they can be retrieved and analyzed. It provides simple scanner capabilities as part of larger enterprise search solutions.
-
19
SteelBot is an open source ICQ/Jabber bot written in PHP. It is fully modular and easy using, and can use files or MySQL engines as database driver.
-
20
Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net
-
21
Combine is an open system for crawling Internet resources. It can be used both as a general and focused crawler. If you want to download Web-pages pertaining to a particular topic (like 'Carnivorous Plants') Then Combine is the system for you!
-
22
Ajax's Bot is a free online scripting service. We also do Shell hosting. All of our services are made to the best of our ability. We offer support on our irc server and on our live support and forums.
-
23
Universal information crawler is a fast precise and reliable Internet crawler. Uicrawler is a program/automated script which browses the World Wide Web in a methodical, automated manner and creates the index of documents that it accesses.
-
24
A configurable knowledge management framework. It works out of the box, but it's meant mainly as a framework to build complex information retrieval and analysis systems. The 3 major components: Crawler, Analyzer and Indexer can also be used separately.
-
25
IrcBot++ is a flexible IRC bot written in C++. It supports multi-server, remote configuration and administration. Among many other features it will include a Half-Life module, a Web based channel statistics module, an IRC-proxy module, a file-server mod