Search Results for "web crawler bot"
Sort By:
Showing 53 open source projects for "web crawler bot"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
3
Bot Framework Web Chat
A highly-customizable web-based client for Azure Bot Services
This repository contains code for the Bot Framework Web Chat component. The Bot Framework Web Chat component is a highly-customizable web-based client for the Bot Framework V4 SDK. The Bot Framework SDK v4 enables developers to model conversation and build sophisticated bot applications. This repo is part of the Microsoft Bot Framework, a comprehensive framework for building enterprise-grade conversational AI experiences. ... -
4
WebMagic
A scalable web crawler framework for Java
WebMagic is a scalable crawler framework. It covers the whole lifecycle of crawler, downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. WebMagic is a simple but scalable crawler framework. You can develop a crawler easily based on it. WebMagic has a simple core with high flexibility, a simple API for html extracting. It also provides annotation with POJO to customize a crawler, and no configuration is needed. Some other... -
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
5
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
6
Bot Framework Emulator
Locally test and debug chat bots built with the Bot Framework SDK
Bot Framework Emulator is a desktop application that allows bot developers to test and debug bots, either locally or remotely. Using the Emulator, you can chat with your bot and inspect the messages that your bot sends and receives. The Emulator displays messages as they would appear in a web chat UI and logs JSON requests and responses as you exchange messages with your bot. -
7
Slack Machine
A simple, yet powerful and extendable Slack bot
Slack Machine is a simple, yet powerful and extendable Slack bot framework. More than just a bot, Slack Machine is a framework that helps you develop your Slack workspace into a ChatOps powerhouse. Slack Machine is built with an intuitive plugin system that lets you build bots quickly but also allows for easy code organization. A plugin can look as simple as this: -
8
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
9
grammY
The Telegram Bot Framework
grammY makes it easy to create Telegram bots. Both for beginners and at scale. You want grammY because it is easy to use. It is very powerful and always up to date. It has the best documentation in town. It is extremely efficient and scales up effortlessly. It has a thriving ecosystem of plugins, a friendly community chat, seamless integrations with web frameworks and databases, and so much more. -
Your monitoring isn't a stack. It's a pile. Fix that.Replace Datadog, New Relic, and Sentry without adding three more dashboards.
-
10
Cap CAPTCHA
The privacy-first, self-hosted CAPTCHA for the modern web
...It replaces invasive tracking-based CAPTCHA solutions with a lightweight approach based on proof-of-work and client-side instrumentation. The system can be deployed independently, giving developers full control over user data and verification processes. It integrates easily into web applications and focuses on maintaining usability while preventing automated abuse. Cap avoids reliance on centralized providers, reducing dependency on external services and improving privacy compliance. Its design reflects a shift toward decentralized, user-respecting security mechanisms. It is particularly suited for developers who want transparent and customizable bot protection. -
11
HyperTAG
AI-Generated Tags and Summaries for Telegram Messages
...Additionally, it generates summaries for Web pages or YouTube videos using their links. -
12
3X-UI
Xray panel supporting multi-protocol multi-user expire day
3X‑UI is an enhanced fork of the X‑UI project, serving as a multifunctional web panel to manage VPN services. It supports multi‑protocol configurations (like VMess, VLESS, Trojan, Shadowsocks, WireGuard), offers traffic and user limits, SSL management, multilingual support, and integrates Telegram notifications for system events. Advanced, open-source web-based control panel designed for managing Xray-core server. It offers a user-friendly interface for configuring and monitoring various VPN... -
13
SiteOne Crawler (desktop app)
A free, feature-rich web analyzer and exporter/cloner you will love!
A free in-depth website analyzer providing audits of security, performance, SEO, accessibility and other technical aspects. Available as a desktop application for Windows/macOS/Linux and as a CLI tool for advanced users and CI/CD processes. It also includes an offline web page exporter (website clone, mirror). -
14
Zendriver
A blazing fast, async-first, undetectable webscraping
Zendriver is a modern Python web automation and scraping framework that leverages the Chrome DevTools Protocol to provide fast, asynchronous control over real browser instances. Unlike traditional tools that rely on Selenium or WebDriver, Zendriver communicates directly with the browser through CDP, enabling higher performance and more precise control over browser behavior. -
15
Botonic
Build chatbots and conversational experiences using React
Botonic is a full-stack Javascript framework to create chatbots and modern conversational apps that work on multiple platforms, web, mobile and messaging apps (Messenger, Whatsapp, Telegram, etc). Building modern applications on top of messaging apps like Whatsapp or Messenger is much more than creating simple text-based chatbots. Botonic is a full-stack serverless framework that combines the power of React and Tensorflow.js to create amazing experiences at the intersection of text and... -
16
DeviceDetector
The Universal Device Detection library will parse any User Agent
Device Detector is a PHP library for parsing user-agent strings to detect devices, operating systems, and browsers, widely used in analytics and tracking. -
17
React ChatBotify
A modern React library for creating flexible and extensible chatbots
Welcome to React ChatBotify, an intuitive and versatile library that allows you to build responsive chatbots with ease. Whether you're aiming to create a simple FAQ bot or a sophisticated conversational interface integrated with Large Language Models (LLMs), React ChatBotify provides the tools and flexibility you need to bring your vision to life. -
18
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
19
miniblink49
Lighter, faster browser kernel of blink to integrate HTML UI in apps
...After turning off the cross-domain switch, you can use various cross-domain functions (support cross-domain). Headless mode, which greatly saves resources and is suitable for crawlers (headless mode, be suitable for Web Crawler). -
20
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
21
reCAPTCHA
PHP client library for reCAPTCHA, a free service
...The ecosystem supports mobile and enterprise variants, but the repo focuses on common web integrations and best practices for verifying the token securely. Deployed correctly, reCAPTCHA reduces credential stuffing, bot sign-ups, and scraping without degrading the experience for typical users. -
22
Venom
Venom is the most complete javascript library for Whatsapp
Venom is a high-performance system developed with JavaScript to create a bot for WhatsApp, support for creating any interaction, such as customer service, media sending, sentence recognition based on artificial intelligence and all types of design architecture for WhatsApp. It's a high-performance alternative API to whatzapp, you can send, text messages, files, images, videos and more. Remember, the API was developed on a platform called RESTful Web services, providing interoperability between computer systems on the Internet. ... -
23
Kong
The Cloud-Native API Gateway
Kong is a next generation cloud-native API platform for multi-cloud and hybrid organizations. When building for the web, mobile, or Internet of Things, you’ll need a common functionality to run your software, and Kong is that solution. Kong acts as a gateway, connecting microservices requests and APIs natively while also providing load balancing, logging, monitoring, authentication, rate-limiting, and so much more through plugins. Kong is highly extensible as well as platform agnostic,... -
24
Simple Traffic Stats WordPress Plugin
Lightweight WordPress traffic analytics plugin.
Simple Traffic Stats WordPress Plugin is a lightweight, self-hosted analytics plugin for WordPress. It tracks page views, unique visitors, top pages, referrers, traffic sources, and optional visitor country stats directly inside the WordPress dashboard. The plugin stores analytics data locally in your WordPress database and does not rely on third-party analytics services for core tracking. It includes charts, summary tables, a dashboard widget, date filters, bot filtering, role... -
25
tirreno
Security framework for devs and product teams
tirreno is a security framework. tirreno [tir.ˈrɛ.no] helps understand, monitor, and protect your product from threats, fraud, and abuse. While classic cybersecurity focuses on infrastructure and network perimeter, most breaches occur through compromised accounts and application logic abuse that bypasses firewalls, SIEM, WAFs, and other defenses. tirreno detects threats where they actually happen: inside your product. tirreno is a few-dependency, "low-tech" PHP/PostgreSQL application....