Search Results for "web crawler source code"
Sort By:
Showing 115 open source projects for "web crawler source code"
View related business solutions-
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
Scope Sentry
Cyberspace asset mapping and vulnerability scanning platform
ScopeSentry is an open source cybersecurity tool designed for cyberspace asset mapping and automated security analysis. It helps security researchers and penetration testers discover, monitor, and analyze internet-facing assets belonging to a target scope. ScopeSentry combines multiple reconnaissance and vulnerability assessment capabilities such as subdomain enumeration, port scanning, directory scanning, and sensitive information detection. ScopeSentry can automatically identify assets and... -
2
ngx_waf
Handy, High performance, ModSecurity compatible Nginx firewall module
Handy, High-performance Nginx firewall module. Such as black and white list of IPs or IP range, uri black and white list, and request body black list, etc. Directives and rules are easy to write and readable. The IP detection is a constant-time operation. Most of the remaining inspections use caching to improve performance. Compatible with ModSecurity's rules, you can use OWASP ModSecurity Core Rule Set. Supports verifying Google, Bing, Baidu and Yandex crawlers and allowing them... -
3
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
4
ModSecurity
Cross platform web application firewall (WAF) engine for Apache
ModSecurity is an open-source, cross-platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language that provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis. Libmodsecurity is one component of the ModSecurity v3 project. The library codebase serves as an interface to ModSecurity Connectors taking in web... -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
SafeLine
Serve as a reverse proxy to protect your web services from attacks
SafeLine is a self-hosted WAF(Web Application Firewall) to protect your web apps from attacks and exploits. A web application firewall helps protect web apps by filtering and monitoring HTTP traffic between a web application and the Internet. It typically protects web apps from attacks such as SQL injection, XSS, code injection, os command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP-flood, bot abuse, among others. By... -
6
cargo-crev
A cryptographically verifiable code review system for the cargo
...Increase the trustworthiness of your own code. Build a web of trust of other reputable users to help verify the code you use. Static binaries are available from the releases page. Crev is a system for verifying the security and reliability of dependencies based on collaborative code reviews. Crev users review the source code of packages/libraries/crates and share their findings with others. -
7
Shannon
Fully autonomous AI hacker to find actual exploits in your web apps
Shannon is an autonomous AI penetration testing system built to find and prove real, exploitable vulnerabilities in web applications rather than stopping at static warnings or best-guess alerts. It focuses on “proof by exploitation,” meaning it actively hunts for attack vectors in your code and then attempts to execute end-to-end exploits to demonstrate impact. The project blends source-aware analysis with automated web interaction so it can validate issues like injection flaws, authentication bypasses, and other exploitable paths in a way that resembles an actual attacker’s workflow. ... -
8
Password Pusher
An application to securely communicate passwords over the web
Give your users the tools to be secure by default. Password Pusher is an open source application to communicate passwords over the web. Links to passwords expire after a certain number of views and/or time has passed. Only enter a password into the box. Other identifying information can compromise security. All passwords are encrypted prior to storage and are available to only those with the secret link. Once expired, encrypted passwords are unequivocally deleted from the database. -
9
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...Coraza runs the OWASP Core Rule Set (CRS) to protect your web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
10
PHP dotenv
Loads environment variables automatically
You should never store sensitive credentials in your code. Storing configuration in the environment is one of the tenets of a twelve-factor app. Anything that is likely to change between deployment environments, such as database credentials or credentials for 3rd party services, should be extracted from the code into environment variables. Basically, a .env file is an easy way to load custom configuration variables that your application needs without having to modify .htaccess files or... -
11
multiOTP open source
PHP strong authentication library, web interface & CLI, OATH certified
multiOTP is a PHP class, a powerful command line utility and a web interface developed by SysCo systèmes de communication sa in order to provide a completely free and easy operating system independent server side implementation for strong two factors authentication solution. multiOTP supports hardware and software tokens with different One-Time Password algorithms like OATH/HOTP, OATH/TOTP and mOTP (Mobile-OTP). QRcode generation is also embedded in order to support provisioning of Google... -
12
FlareSolverr
Proxy server to bypass Cloudflare protection
FlareSolverr is a proxy server to bypass Cloudflare and DDoS-GUARD protection. FlareSolverr starts a proxy server, and it waits for user requests in an idle state using few resources. When some request arrives, it uses puppeteer with the stealth plugin to create a headless browser (Firefox). It opens the URL with user parameters and waits until the Cloudflare challenge is solved (or timeout). The HTML code and the cookies are sent back to the user, and those cookies can be used to bypass... -
13
WAFW00F
WAFW00F allows one to identify and fingerprint Web App Firewall
...For further details, check out the source code on our main repository. -
14
Ente
End-to-end encrypted cloud for photos, videos and 2FA secrets
...On top of this platform, we have built two apps so far: Ente Photos (an alternative to Apple and Google Photos) and Ente Auth (a 2FA alternative to the deprecated Authy). This monorepo contains all our source code - the client apps (iOS / Android / F-Droid / Web / Linux / macOS / Windows) for both the products (and more planned future ones!), and the server that powers them. -
15
T3MP3ST
Autonomous red teaming platform
...It wraps an existing AI coding agent with orchestration for reconnaissance, exploit exploration, validation, and reporting. The project can work through a browser War Room or CLI while relying on agents such as Claude Code, Codex, Hermes, or locally hosted models. It emphasizes reproducible benchmark claims, keyless operation, and a clear status table that separates stable capabilities from experimental or roadmap work. Its stated target areas include web apps, CTF challenges, source-code review, OSS vulnerability hunting, smart contracts, and embedded or robotics research. ... -
16
privacy.sexy
Open-source tool to enforce privacy & security best-practices
Enforce privacy & security best practices on Windows, macOS, and Linux, because privacy is sexy. Regularly applying your configuration with privacy.sexy is recommended, especially after each new release and major operating system updates. Each version updates scripts to enhance stability, privacy, and security. -
17
Casdoor
An open-source Identity and Access Management (IAM)
A UI-first Identity Access Management (IAM) / Single-Sign-On (SSO) platform supporting OAuth 2.0, OIDC, SAML and CAS, integrated with Casbin RBAC and ABAC permission management. Within a few steps, we can setup a Casdoor app and realize our authorization management. Casdoor has a front-end back-end separation architecture, with maneuverable web UI and supporting high concurrency. Casdoor is supporting multi-languages, using i18n to support multi-languages UI. For more languages support,... -
18
ModSecurity Nginx Connector
ModSecurity v3 Nginx Connector
ModSecurity-nginx is the connector that embeds the ModSecurity v3 (libmodsecurity) web application firewall engine into NGINX. It integrates WAF processing into the NGINX request/response phases, allowing rules to inspect headers, bodies, and even streaming request data before it reaches upstream apps. Operators can load the OWASP Core Rule Set or custom rules to detect and block common attacks such as SQLi, XSS, RCE patterns, and protocol anomalies. The module exposes directives for... -
19
vRain
Chinese Ancient eBooks Generator
vRain is a small, focused automation tool aimed at simplifying routine tasks on education-oriented web platforms, particularly those that require periodic attendance or check-ins. It encapsulates login, session management, and request sequencing so repetitive actions can be executed reliably on a schedule. A configuration-driven approach lets users store multiple accounts, adjust intervals, and toggle features without modifying code. The program surfaces clear logs and result summaries so... -
20
React Native Auth0
React Native toolkit for Auth0 API
With a few lines of code, you can have Auth0 integrated into any app written in any language, and any framework. We provide 30+ SDKs & Quickstarts to help you succeed in your implementation. Enable user collaboration and granular access control in your applications with easy-to-use APIs. From improving customer experience through seamless sign-on to making MFA as easy as a click of a button, your login box must find the right balance between user convenience, privacy and security. That’s why... -
21
Nano ID
A secure, URL-friendly, unique string ID generator for JavaScript
Nano ID is a library for generating random IDs. Likewise UUID, there is a probability of duplicate IDs. However, this probability is extremely small. Meanwhile, a lot of projects generate IDs in small numbers. For those projects, the ID length could be reduced without risk. This calculator aims to help you realize the extent to which the ID length can be reduced. Instead of using the unsafe Math.random(), Nano ID uses the crypto module in Node.js and the Web Crypto API in browsers. These... -
22
IBM TPM Attestation Client Server
IBM's TPM 2.0 Attestation Client Server
This is sample code for a TCG attestation application. It supports TPM 2.0 and TPM 1.2. It includes 5 main pieces: 1 - An attestation server 2 - An attestation client that pushes quotes to the server 3 - An enrollment client that enrolls a signing key with the server 4 - A utility to provision an EK certificate on a SW TPM for testing 5 - Demo php code to display results The code is C, with json formatting at the client / server interface. Mysql is used for data storage. The... -
23
CacheGuard Gateway
Free UTM appliance: firewall, VPN, WAF and antivirus in one ISO.
...A fully custom network appliance operating system built from scratch over 20 years, now fully open source. One ISO includes: firewall, VPN, web antivirus, URL filtering, SSL inspection, WAF, reverse proxy, load balancer and QoS. Free for any number of users. Optional paid support available. Source code: https://github.com/cacheguard/CacheGuard-OS Website: https://www.cacheguard.com/ -
24
pH7 Social Dating CMS (pH7Builder)❤️
🚀 Professional Social Dating Web App Builder (formerly pH7CMS)
pH7Builder is a Professional, Free & Open Source PHP Social Dating Builder Software (primarily designed for developers ...). This Social Dating Web App is fully coded in object-oriented PHP (OOP) with the MVC pattern (Model-View-Controller). It is low resource-intensive, extremely powerful and highly secure. pH7Builder is included with over 42 native modules and is based on its homemade pH7 Framework which includes more than 52 packages To summarize, pH7Builder Social Dating Script gives you the perfect ingredients to create the best dating web app or social networking site on the World Wide Web! ... -
25
SentraX
Open source cybersecurity suite with 10 network scanners, API, JWT + 2
SENTRAX est une suite professionnelle open source de cybersécurité dédiée à l’audit réseau et à la détection de vulnérabilités. **Caractéristiques clés :** - 10 scanners réseau (TCP, SYN, UDP, ICMP, SNMP, DNS, OSINT, Passive, Time Machine, CVE) - API REST avec authentification JWT + 2FA + limitation de débit - Tableau de bord web moderne (thème sombre/clair, graphiques, histoire) - Windows EXE autonome (33 Mo, pas d’installation) - Code source pour Linux / macOS / Docker - Export PDF / JSON / CSV - Système de plug-inning **Cas d’utilisation :** - Audits internes du réseau - Formation et éducation en sécurité - Détection de vulnérabilité (EternalBlue, BlueKeep, Heartbleed, etc.) ...
