Search Results for "python web crawler"
Sort By:
Showing 197 open source projects for "python web crawler"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
1
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
2
Wfuzz
Web application fuzzer
Wfuzz provides a framework to automate web applications security assessments and could help you to secure your web applications by finding and exploiting web application vulnerabilities. Wfuzz it is based on a simple concept: it replaces any reference to the FUZZ keyword by the value of a given payload. A payload in Wfuzz is a source of data. This simple concept allows any input to be injected in any field of an HTTP request, allowing to perform complex web security attacks in different web... -
3
ngx_waf
Handy, High performance, ModSecurity compatible Nginx firewall module
Handy, High-performance Nginx firewall module. Such as black and white list of IPs or IP range, uri black and white list, and request body black list, etc. Directives and rules are easy to write and readable. The IP detection is a constant-time operation. Most of the remaining inspections use caching to improve performance. Compatible with ModSecurity's rules, you can use OWASP ModSecurity Core Rule Set. Supports verifying Google, Bing, Baidu and Yandex crawlers and allowing them... -
4
Scope Sentry
Cyberspace asset mapping and vulnerability scanning platform
ScopeSentry is an open source cybersecurity tool designed for cyberspace asset mapping and automated security analysis. It helps security researchers and penetration testers discover, monitor, and analyze internet-facing assets belonging to a target scope. ScopeSentry combines multiple reconnaissance and vulnerability assessment capabilities such as subdomain enumeration, port scanning, directory scanning, and sensitive information detection. ScopeSentry can automatically identify assets and... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Robin
AI-powered tool for dark web OSINT search and investigation
Robin is an AI-powered open source tool designed to assist investigators and researchers in conducting dark web OSINT (Open Source Intelligence) investigations. It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven... -
6
H4X-Tools
A modular, terminal-based toolkit for OSINT, reconnaissance
H4X-Tools is a modular, terminal-based toolkit for OSINT, reconnaissance, and scraping workflows. It is built in Python and runs on Linux and Windows. The project organizes multiple research utilities behind an interactive menu and direct command-line options. Its tools cover areas such as Instagram profile research, web reconnaissance, phone lookup, IP lookup, username search, email search, leak intelligence, port scanning, WHOIS lookup, web scraping, directory discovery, Bluetooth scanning, and local user enumeration. ... -
7
IPRanges
Daily updated lists of cloud, bot, and service IP ranges
ipranges is an open source repository that provides continuously updated lists of IP address ranges associated with major cloud providers, search engine crawlers, and online services. ipranges collects IP ranges from publicly available sources and organizes them into structured files that can be easily used in security, networking, and automation workflows. It includes address ranges from providers such as Google Cloud, Amazon AWS, Microsoft, Oracle Cloud, and DigitalOcean, as well as well... -
8
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
WhatsApp Beacon is an open source OSINT tool designed to monitor and analyze the online activity status of WhatsApp users through WhatsApp Web. It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review... -
9
Hackingtool
ALL IN ONE Hacking Tool For Hackers
HackingTool by Z4nzu is a large collection (“all-in-one”) of tools and scripts for penetration testing / hacking / OSINT etc. It bundles many utilities (port scanners, payload injectors, web attack tools, phishing tools, wireless attack tools, reverse engineering, etc.) into a menu interface. Includes many individual tools, often wrappers or aggregations of existing well-known tools (e.g. port scanners, web attack tools, steganography, hash cracking etc.). A menu interface offering... -
Stop vibe-debugging.AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.
-
10
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
11
mitmproxy
A free and open source interactive HTTPS proxy
mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of... -
12
UFONet
UFONet - Denial of Service Toolkit
UFONet is a powerful and controversial Python-based toolkit for testing and conducting Distributed Denial of Service (DDoS) attacks using unconventional methods, such as leveraging third-party web applications as attack vectors. It automates the discovery of vulnerable targets and enables attackers or researchers to launch large-scale amplification attacks without directly using botnets. -
13
Django OAuth Toolkit
OAuth2 goodies for the Djangonauts!
...OAuth is an open standard for access delegation, commonly used as a way for Internet users to grant websites or applications access to their information on other websites but without giving them the passwords. Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. Your Django app exposes a web API you want to protect with OAuth2 authentication. You need to implement an OAuth2 authorization server to provide tokens management for your infrastructure. -
14
WAFW00F
WAFW00F allows one to identify and fingerprint Web App Firewall
The Web Application Firewall Fingerprinting Tool. Sends a normal HTTP request and analyses the response; this identifies a number of WAF solutions. If that is not successful, it sends a number of (potentially malicious) HTTP requests and uses simple logic to deduce which WAF it is. If that is also not successful, it analyses the responses previously returned and uses another simple algorithm to guess if a WAF or security solution is actively responding to our attacks. For further details,... -
15
OWASP Maryam
Modular OSINT framework for automated open-source intelligence gatheri
Maryam is an open source intelligence (OSINT) framework designed to automate the process of gathering and analyzing publicly available information from the internet. It provides a modular environment that enables users to collect data from search engines, open data sources, and various online services for reconnaissance and investigative purposes. Written in Python, Maryam is built to provide a flexible and extensible framework for harvesting information quickly and efficiently from open... -
16
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying... -
17
fsociety
Modular CLI framework for managing penetration testing tools
...Instead of implementing its own security scanners, the framework acts as a wrapper and orchestrator that helps users discover, install, and execute tools from various GitHub repositories. Its modular architecture organizes tools into categories such as information gathering, networking, web application security, and password testing. This structure allows users to quickly navigate through different security tasks while maintaining a consistent interface. fsociety can automatically clone and manage required tools, reducing the manual effort typically needed to set up a penetration testing toolkit. fsociety is distributed as a Python package. -
18
Payloads All The Things
A list of useful payloads and bypass for Web Application Security
A list of useful payloads and bypasses for Web Application Security. Feel free to improve with your payloads and techniques. The API key is a unique identifier that is used to authenticate requests associated with your project. Some developers might hardcode them or leave it on public shares. -
19
GTFOBins
GTFOBins is a curated list of Unix binaries
GTFOBins is a curated catalog of Unix / POSIX system binaries and how they can be misused to bypass restrictions, escalate privileges, exfiltrate data, spawn shells, or otherwise act as “living off the land” tools in a compromised environment. It collects documented techniques for how everyday binaries (e.g. awk, bash, tar, scp) can be abused under constrained conditions. Indexed list of Unix binaries and documented misuse techniques. Examples of command invocations to exploit... -
20
OAuthLib
A generic, spec-compliant, thorough implementation of the OAuth
A generic, spec-compliant, thorough implementation of the OAuth request-signing logic for Python 3.8+. OAuthLib is a framework which implements the logic of OAuth1 or OAuth2 without assuming a specific HTTP request object or web framework. Use it to graft OAuth client support onto your favorite HTTP library, or provide support onto your favourite web framework. If you're a maintainer of such a library, write a thin veneer on top of OAuthLib and get OAuth support for very little effort. -
21
Tookie-OSINT
Username OSINT tool for discovering accounts across many websites
Tookie-OSINT is an open source intelligence tool designed to help security researchers, ethical hackers, and investigators discover online accounts associated with a specific username. It automates the process of searching for usernames across multiple websites, making it easier to identify a person's presence on different platforms. By entering a target username, Tookie-OSINT scans a list of supported sites and checks whether the username exists on those platforms. This approach removes the... -
22
dnstwist
Detects phishing and lookalike domains using DNS fuzzing techniques
dnstwist is an open source cybersecurity tool designed to identify malicious or suspicious domain names that imitate legitimate websites. It works by generating a large set of domain name permutations based on a target domain and analyzing whether any of those variants are actively registered or used. These permutations simulate common techniques used in phishing attacks, typosquatting, and brand impersonation campaigns. Security teams can use the tool to discover potential threats where... -
23
Gobuster
Directory/File, DNS and VHost busting tool written in Go
Gobuster is a tool used to brute-force. This project is born out of the necessity to have something that didn't have a fat Java GUI (console FTW), something that did not do recursive brute force, something that allowed me to brute force folders and multiple extensions at once, something that compiled to native on multiple platforms, something that was faster than an interpreted script (such as Python), and something that didn't require a runtime. Provides several modes, like the classic... -
24
reNgine
Automated framework for web application reconnaissance and scanning
reNgine is an automated reconnaissance framework designed to simplify and enhance the process of gathering information about web applications during security assessments. It provides a streamlined workflow for penetration testers, bug bounty hunters, and security teams who need to perform reconnaissance efficiently and at scale. The platform integrates multiple open-source reconnaissance tools into a unified environment with a configurable scanning engine and an intuitive web interface.... -
25
discover
Automation framework for reconnaissance and penetration testing tasks
Discover is a collection of custom Bash scripts designed to automate many common tasks involved in penetration testing workflows. The project brings together a variety of security testing functions into a single framework that simplifies reconnaissance, scanning, and enumeration processes. It provides a menu-driven interface that allows security professionals to quickly launch different tools and scripts without manually executing each command. The framework helps streamline activities such...
