Search Results for "python web crawler"
Sort By:
Showing 176 open source projects for "python web crawler"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
2
Wfuzz
Web application fuzzer
Wfuzz provides a framework to automate web applications security assessments and could help you to secure your web applications by finding and exploiting web application vulnerabilities. Wfuzz it is based on a simple concept: it replaces any reference to the FUZZ keyword by the value of a given payload. A payload in Wfuzz is a source of data. This simple concept allows any input to be injected in any field of an HTTP request, allowing to perform complex web security attacks in different web... -
3
Robin
AI-powered tool for dark web OSINT search and investigation
Robin is an AI-powered open source tool designed to assist investigators and researchers in conducting dark web OSINT (Open Source Intelligence) investigations. It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven... -
4
H4X-Tools
A modular, terminal-based toolkit for OSINT, reconnaissance
H4X-Tools is a modular, terminal-based toolkit for OSINT, reconnaissance, and scraping workflows. It is built in Python and runs on Linux and Windows. The project organizes multiple research utilities behind an interactive menu and direct command-line options. Its tools cover areas such as Instagram profile research, web reconnaissance, phone lookup, IP lookup, username search, email search, leak intelligence, port scanning, WHOIS lookup, web scraping, directory discovery, Bluetooth scanning, and local user enumeration. ... -
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
5
IPRanges
Daily updated lists of cloud, bot, and service IP ranges
ipranges is an open source repository that provides continuously updated lists of IP address ranges associated with major cloud providers, search engine crawlers, and online services. ipranges collects IP ranges from publicly available sources and organizes them into structured files that can be easily used in security, networking, and automation workflows. It includes address ranges from providers such as Google Cloud, Amazon AWS, Microsoft, Oracle Cloud, and DigitalOcean, as well as well... -
6
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
WhatsApp Beacon is an open source OSINT tool designed to monitor and analyze the online activity status of WhatsApp users through WhatsApp Web. It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review... -
7
Hackingtool
ALL IN ONE Hacking Tool For Hackers
HackingTool by Z4nzu is a large collection (“all-in-one”) of tools and scripts for penetration testing / hacking / OSINT etc. It bundles many utilities (port scanners, payload injectors, web attack tools, phishing tools, wireless attack tools, reverse engineering, etc.) into a menu interface. Includes many individual tools, often wrappers or aggregations of existing well-known tools (e.g. port scanners, web attack tools, steganography, hash cracking etc.). A menu interface offering... -
8
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
9
mitmproxy
A free and open source interactive HTTPS proxy
mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of... -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
10
Django OAuth Toolkit
OAuth2 goodies for the Djangonauts!
...OAuth is an open standard for access delegation, commonly used as a way for Internet users to grant websites or applications access to their information on other websites but without giving them the passwords. Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. Your Django app exposes a web API you want to protect with OAuth2 authentication. You need to implement an OAuth2 authorization server to provide tokens management for your infrastructure. -
11
WAFW00F
WAFW00F allows one to identify and fingerprint Web App Firewall
The Web Application Firewall Fingerprinting Tool. Sends a normal HTTP request and analyses the response; this identifies a number of WAF solutions. If that is not successful, it sends a number of (potentially malicious) HTTP requests and uses simple logic to deduce which WAF it is. If that is also not successful, it analyses the responses previously returned and uses another simple algorithm to guess if a WAF or security solution is actively responding to our attacks. For further details,... -
12
OWASP Maryam
Modular OSINT framework for automated open-source intelligence gatheri
Maryam is an open source intelligence (OSINT) framework designed to automate the process of gathering and analyzing publicly available information from the internet. It provides a modular environment that enables users to collect data from search engines, open data sources, and various online services for reconnaissance and investigative purposes. Written in Python, Maryam is built to provide a flexible and extensible framework for harvesting information quickly and efficiently from open... -
13
fsociety
Modular CLI framework for managing penetration testing tools
...Instead of implementing its own security scanners, the framework acts as a wrapper and orchestrator that helps users discover, install, and execute tools from various GitHub repositories. Its modular architecture organizes tools into categories such as information gathering, networking, web application security, and password testing. This structure allows users to quickly navigate through different security tasks while maintaining a consistent interface. fsociety can automatically clone and manage required tools, reducing the manual effort typically needed to set up a penetration testing toolkit. fsociety is distributed as a Python package. -
14
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying... -
15
Payloads All The Things
A list of useful payloads and bypass for Web Application Security
A list of useful payloads and bypasses for Web Application Security. Feel free to improve with your payloads and techniques. The API key is a unique identifier that is used to authenticate requests associated with your project. Some developers might hardcode them or leave it on public shares. -
16
GTFOBins
GTFOBins is a curated list of Unix binaries
GTFOBins is a curated catalog of Unix / POSIX system binaries and how they can be misused to bypass restrictions, escalate privileges, exfiltrate data, spawn shells, or otherwise act as “living off the land” tools in a compromised environment. It collects documented techniques for how everyday binaries (e.g. awk, bash, tar, scp) can be abused under constrained conditions. Indexed list of Unix binaries and documented misuse techniques. Examples of command invocations to exploit... -
17
OAuthLib
A generic, spec-compliant, thorough implementation of the OAuth
A generic, spec-compliant, thorough implementation of the OAuth request-signing logic for Python 3.8+. OAuthLib is a framework which implements the logic of OAuth1 or OAuth2 without assuming a specific HTTP request object or web framework. Use it to graft OAuth client support onto your favorite HTTP library, or provide support onto your favourite web framework. If you're a maintainer of such a library, write a thin veneer on top of OAuthLib and get OAuth support for very little effort. -
18
Tookie-OSINT
Username OSINT tool for discovering accounts across many websites
Tookie-OSINT is an open source intelligence tool designed to help security researchers, ethical hackers, and investigators discover online accounts associated with a specific username. It automates the process of searching for usernames across multiple websites, making it easier to identify a person's presence on different platforms. By entering a target username, Tookie-OSINT scans a list of supported sites and checks whether the username exists on those platforms. This approach removes the... -
19
dnstwist
Detects phishing and lookalike domains using DNS fuzzing techniques
dnstwist is an open source cybersecurity tool designed to identify malicious or suspicious domain names that imitate legitimate websites. It works by generating a large set of domain name permutations based on a target domain and analyzing whether any of those variants are actively registered or used. These permutations simulate common techniques used in phishing attacks, typosquatting, and brand impersonation campaigns. Security teams can use the tool to discover potential threats where... -
20
reNgine
Automated framework for web application reconnaissance and scanning
reNgine is an automated reconnaissance framework designed to simplify and enhance the process of gathering information about web applications during security assessments. It provides a streamlined workflow for penetration testers, bug bounty hunters, and security teams who need to perform reconnaissance efficiently and at scale. The platform integrates multiple open-source reconnaissance tools into a unified environment with a configurable scanning engine and an intuitive web interface.... -
21
Maltrail
Malicious traffic detection system
Maltrail is a malicious traffic detection system, utilizing publicly available (black)lists containing malicious and/or generally suspicious trails, along with static trails compiled from various AV reports and custom user-defined lists, where trail can be anything from domain name, URL, IP address (e.g. 185.130.5.231 for the known attacker) or HTTP User-Agent header value (e.g. sqlmap for automatic SQL injection and database takeover tool). Also, it uses (optional) advanced heuristic... -
22
IVRE
Open source framework for large scale network reconnaissance and analy
...The framework integrates with well known security and scanning tools such as Nmap, Masscan, ZGrab2, ZDNS, and Zeek to gather large amounts of network intelligence. IVRE stores the collected data in a database and offers multiple ways to explore and analyze it, including a web interface, command line tools, and a Python API. This allows security professionals to query scan results, inspect network flows, and identify patterns across large datasets. The project can be used to build self hosted alternatives to internet scanning services such as Shodan or Censys, giving organizations full control over their own reconnaissance infrastructure. -
23
discover
Automation framework for reconnaissance and penetration testing tasks
Discover is a collection of custom Bash scripts designed to automate many common tasks involved in penetration testing workflows. The project brings together a variety of security testing functions into a single framework that simplifies reconnaissance, scanning, and enumeration processes. It provides a menu-driven interface that allows security professionals to quickly launch different tools and scripts without manually executing each command. The framework helps streamline activities such... -
24
ClusterFuzz
Scalable fuzzing infrastructure
ClusterFuzz is a scalable fuzzing infrastructure that finds security and stability issues in software. Google uses ClusterFuzz to fuzz all Google products and as the fuzzing backend for OSS-Fuzz. ClusterFuzz provides many features which help seamlessly integrate fuzzing into a software project's development process. Can run on any size cluster (e.g. OSS-Fuzz instance runs on 100,000 VMs). Fully automatic bug filing, triage and closing for various issue trackers (e.g. Monorail, Jira).... -
25
GRR
GRR Rapid Response, remote live forensics for incident response
GRR Rapid Response is an incident response framework focused on remote live forensics. It consists of a python client (agent) that is installed on target systems, and python server infrastructure that can manage and talk to clients. The goal of GRR is to support forensics and investigations in a fast, scalable manner to allow analysts to quickly triage attacks and perform analysis remotely. GRR client is deployed on systems that one might want to investigate.
