Search Results for "web crawler source code"
Sort By:
Showing 112 open source projects for "web crawler source code"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Enterprise-grade ITSM, for every businessFreshservice is an intuitive, AI-powered platform that helps IT, operations, and business teams deliver exceptional service without the usual complexity. Automate repetitive tasks, resolve issues faster, and provide seamless support across the organization. From managing incidents and assets to driving smarter decisions, Freshservice makes it easy to stay efficient and scale with confidence.
-
1
ngx_waf
Handy, High performance, ModSecurity compatible Nginx firewall module
Handy, High-performance Nginx firewall module. Such as black and white list of IPs or IP range, uri black and white list, and request body black list, etc. Directives and rules are easy to write and readable. The IP detection is a constant-time operation. Most of the remaining inspections use caching to improve performance. Compatible with ModSecurity's rules, you can use OWASP ModSecurity Core Rule Set. Supports verifying Google, Bing, Baidu and Yandex crawlers and allowing them... -
2
Scope Sentry
Cyberspace asset mapping and vulnerability scanning platform
ScopeSentry is an open source cybersecurity tool designed for cyberspace asset mapping and automated security analysis. It helps security researchers and penetration testers discover, monitor, and analyze internet-facing assets belonging to a target scope. ScopeSentry combines multiple reconnaissance and vulnerability assessment capabilities such as subdomain enumeration, port scanning, directory scanning, and sensitive information detection. ScopeSentry can automatically identify assets and... -
3
SafeLine
Serve as a reverse proxy to protect your web services from attacks
SafeLine is a self-hosted WAF(Web Application Firewall) to protect your web apps from attacks and exploits. A web application firewall helps protect web apps by filtering and monitoring HTTP traffic between a web application and the Internet. It typically protects web apps from attacks such as SQL injection, XSS, code injection, os command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP-flood, bot abuse, among others. By... -
4
cargo-crev
A cryptographically verifiable code review system for the cargo
...Increase the trustworthiness of your own code. Build a web of trust of other reputable users to help verify the code you use. Static binaries are available from the releases page. Crev is a system for verifying the security and reliability of dependencies based on collaborative code reviews. Crev users review the source code of packages/libraries/crates and share their findings with others. -
Try Google Cloud Risk-Free With $300 in CreditUse your credit across every product. Compute, storage, AI, analytics. When it runs out, 20+ products stay free. You only pay when you choose to.
-
5
Password Pusher
An application to securely communicate passwords over the web
Give your users the tools to be secure by default. Password Pusher is an open source application to communicate passwords over the web. Links to passwords expire after a certain number of views and/or time has passed. Only enter a password into the box. Other identifying information can compromise security. All passwords are encrypted prior to storage and are available to only those with the secret link. Once expired, encrypted passwords are unequivocally deleted from the database. -
6
ModSecurity
Cross platform web application firewall (WAF) engine for Apache
ModSecurity is an open-source, cross-platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language that provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis. Libmodsecurity is one component of the ModSecurity v3 project. The library codebase serves as an interface to ModSecurity Connectors taking in web... -
7
WAFW00F
WAFW00F allows one to identify and fingerprint Web App Firewall
...For further details, check out the source code on our main repository. -
8
PHP dotenv
Loads environment variables automatically
You should never store sensitive credentials in your code. Storing configuration in the environment is one of the tenets of a twelve-factor app. Anything that is likely to change between deployment environments, such as database credentials or credentials for 3rd party services, should be extracted from the code into environment variables. Basically, a .env file is an easy way to load custom configuration variables that your application needs without having to modify .htaccess files or... -
9
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...Coraza runs the OWASP Core Rule Set (CRS) to protect your web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
10
Shannon
Fully autonomous AI hacker to find actual exploits in your web apps
Shannon is an autonomous AI penetration testing system built to find and prove real, exploitable vulnerabilities in web applications rather than stopping at static warnings or best-guess alerts. It focuses on “proof by exploitation,” meaning it actively hunts for attack vectors in your code and then attempts to execute end-to-end exploits to demonstrate impact. The project blends source-aware analysis with automated web interaction so it can validate issues like injection flaws, authentication bypasses, and other exploitable paths in a way that resembles an actual attacker’s workflow. ... -
11
multiOTP open source
PHP strong authentication library, web interface & CLI, OATH certified
multiOTP is a PHP class, a powerful command line utility and a web interface developed by SysCo systèmes de communication sa in order to provide a completely free and easy operating system independent server side implementation for strong two factors authentication solution. multiOTP supports hardware and software tokens with different One-Time Password algorithms like OATH/HOTP, OATH/TOTP and mOTP (Mobile-OTP). QRcode generation is also embedded in order to support provisioning of Google... -
12
Ente
End-to-end encrypted cloud for photos, videos and 2FA secrets
...On top of this platform, we have built two apps so far: Ente Photos (an alternative to Apple and Google Photos) and Ente Auth (a 2FA alternative to the deprecated Authy). This monorepo contains all our source code - the client apps (iOS / Android / F-Droid / Web / Linux / macOS / Windows) for both the products (and more planned future ones!), and the server that powers them. -
13
FlareSolverr
Proxy server to bypass Cloudflare protection
FlareSolverr is a proxy server to bypass Cloudflare and DDoS-GUARD protection. FlareSolverr starts a proxy server, and it waits for user requests in an idle state using few resources. When some request arrives, it uses puppeteer with the stealth plugin to create a headless browser (Firefox). It opens the URL with user parameters and waits until the Cloudflare challenge is solved (or timeout). The HTML code and the cookies are sent back to the user, and those cookies can be used to bypass... -
14
privacy.sexy
Open-source tool to enforce privacy & security best-practices
Enforce privacy & security best practices on Windows, macOS, and Linux, because privacy is sexy. Regularly applying your configuration with privacy.sexy is recommended, especially after each new release and major operating system updates. Each version updates scripts to enhance stability, privacy, and security. -
15
Enlightn
Your performance & security consultant, an artisan command away
...The Enlightn OSS (open source software) version has 64 automated checks that scan your application code, web server configurations, and routes to identify performance bottlenecks, possible security vulnerabilities, and code reliability issues. Enlightn Pro (commercial) is available for purchase on the Enlightn website and has an additional 64 automated checks (a total of 128 checks). -
16
ModSecurity Nginx Connector
ModSecurity v3 Nginx Connector
ModSecurity-nginx is the connector that embeds the ModSecurity v3 (libmodsecurity) web application firewall engine into NGINX. It integrates WAF processing into the NGINX request/response phases, allowing rules to inspect headers, bodies, and even streaming request data before it reaches upstream apps. Operators can load the OWASP Core Rule Set or custom rules to detect and block common attacks such as SQLi, XSS, RCE patterns, and protocol anomalies. The module exposes directives for... -
17
Nano ID
A secure, URL-friendly, unique string ID generator for JavaScript
Nano ID is a library for generating random IDs. Likewise UUID, there is a probability of duplicate IDs. However, this probability is extremely small. Meanwhile, a lot of projects generate IDs in small numbers. For those projects, the ID length could be reduced without risk. This calculator aims to help you realize the extent to which the ID length can be reduced. Instead of using the unsafe Math.random(), Nano ID uses the crypto module in Node.js and the Web Crypto API in browsers. These... -
18
Casdoor
An open-source Identity and Access Management (IAM)
A UI-first Identity Access Management (IAM) / Single-Sign-On (SSO) platform supporting OAuth 2.0, OIDC, SAML and CAS, integrated with Casbin RBAC and ABAC permission management. Within a few steps, we can setup a Casdoor app and realize our authorization management. Casdoor has a front-end back-end separation architecture, with maneuverable web UI and supporting high concurrency. Casdoor is supporting multi-languages, using i18n to support multi-languages UI. For more languages support,... -
19
React Native Auth0
React Native toolkit for Auth0 API
With a few lines of code, you can have Auth0 integrated into any app written in any language, and any framework. We provide 30+ SDKs & Quickstarts to help you succeed in your implementation. Enable user collaboration and granular access control in your applications with easy-to-use APIs. From improving customer experience through seamless sign-on to making MFA as easy as a click of a button, your login box must find the right balance between user convenience, privacy and security. That’s why... -
20
vRain
Chinese Ancient eBooks Generator
vRain is a small, focused automation tool aimed at simplifying routine tasks on education-oriented web platforms, particularly those that require periodic attendance or check-ins. It encapsulates login, session management, and request sequencing so repetitive actions can be executed reliably on a schedule. A configuration-driven approach lets users store multiple accounts, adjust intervals, and toggle features without modifying code. The program surfaces clear logs and result summaries so... -
21
IBM TPM Attestation Client Server
IBM's TPM 2.0 Attestation Client Server
This is sample code for a TCG attestation application. It supports TPM 2.0 and TPM 1.2. It includes 5 main pieces: 1 - An attestation server 2 - An attestation client that pushes quotes to the server 3 - An enrollment client that enrolls a signing key with the server 4 - A utility to provision an EK certificate on a SW TPM for testing 5 - Demo php code to display results The code is C, with json formatting at the client / server interface. Mysql is used for data storage. The... -
22
pH7 Social Dating CMS (pH7Builder)❤️
🚀 Professional Social Dating Web App Builder (formerly pH7CMS)
pH7Builder is a Professional, Free & Open Source PHP Social Dating Builder Software (primarily designed for developers ...). This Social Dating Web App is fully coded in object-oriented PHP (OOP) with the MVC pattern (Model-View-Controller). It is low resource-intensive, extremely powerful and highly secure. pH7Builder is included with over 42 native modules and is based on its homemade pH7 Framework which includes more than 52 packages To summarize, pH7Builder Social Dating Script gives you the perfect ingredients to create the best dating web app or social networking site on the World Wide Web! ... -
23
SafeUtils
110+ developer tools as native MacOS, Linux & Windows desktop apps.
Tools: https://safeutils.com/barcode-generator https://safeutils.com/color-picker https://safeutils.com/qr-code-generator https://safeutils.com/qr-code-scanner https://safeutils.com/word-counter https://safeutils.com/base-64-decoder https://safeutils.com/diff-checker https://safeutils.com/hex-to-ascii https://safeutils.com/json-formatter https://safeutils.com/lorem-ipsum-generator https://safeutils.com/random-generator https://safeutils.com/time-converter https://safeutils.com/... -
24
In-Browser-File-Encrypter
The source code of the In-Browser-File-Encrypter web app
The In-Browser File Encrypter is a simple web application that enables you to securely encrypt your files directly in your browser using the AES-256 encryption algorithm in CBC mode. Check it out at: https://codepen.io/Northstrix/pen/xxvXvJL and https://northstrix.github.io/In-Browser-File-Encrypter/V1.0/web-app.html GitHub page: https://github.com/Northstrix/In-Browser-File-Encrypter The download shortcut:... -
25
A network access control (NAC) system featuring a captive-portal for registration and remediation, wired and wireless management, 802.1X support, isolation of devices, integration with IDS; it can be used to secure networks from small to large. Source code is hosted here: https://github.com/inverse-inc/packetfence Issue tracker is hosted here: https://github.com/inverse-inc/packetfence/issues
