Search Results for "python web crawler"
Sort By:
Showing 34 open source projects for "python web crawler"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
1
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
2
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
WhatsApp Beacon is an open source OSINT tool designed to monitor and analyze the online activity status of WhatsApp users through WhatsApp Web. It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review... -
3
IPRanges
Daily updated lists of cloud, bot, and service IP ranges
ipranges is an open source repository that provides continuously updated lists of IP address ranges associated with major cloud providers, search engine crawlers, and online services. ipranges collects IP ranges from publicly available sources and organizes them into structured files that can be easily used in security, networking, and automation workflows. It includes address ranges from providers such as Google Cloud, Amazon AWS, Microsoft, Oracle Cloud, and DigitalOcean, as well as well... -
4
Robin
AI-powered tool for dark web OSINT search and investigation
Robin is an AI-powered open source tool designed to assist investigators and researchers in conducting dark web OSINT (Open Source Intelligence) investigations. It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven... -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
5
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
6
H4X-Tools
A modular, terminal-based toolkit for OSINT, reconnaissance
H4X-Tools is a modular, terminal-based toolkit for OSINT, reconnaissance, and scraping workflows. It is built in Python and runs on Linux and Windows. The project organizes multiple research utilities behind an interactive menu and direct command-line options. Its tools cover areas such as Instagram profile research, web reconnaissance, phone lookup, IP lookup, username search, email search, leak intelligence, port scanning, WHOIS lookup, web scraping, directory discovery, Bluetooth scanning, and local user enumeration. ... -
7
OWASP Maryam
Modular OSINT framework for automated open-source intelligence gatheri
Maryam is an open source intelligence (OSINT) framework designed to automate the process of gathering and analyzing publicly available information from the internet. It provides a modular environment that enables users to collect data from search engines, open data sources, and various online services for reconnaissance and investigative purposes. Written in Python, Maryam is built to provide a flexible and extensible framework for harvesting information quickly and efficiently from open... -
8
IVRE
Open source framework for large scale network reconnaissance and analy
...The framework integrates with well known security and scanning tools such as Nmap, Masscan, ZGrab2, ZDNS, and Zeek to gather large amounts of network intelligence. IVRE stores the collected data in a database and offers multiple ways to explore and analyze it, including a web interface, command line tools, and a Python API. This allows security professionals to query scan results, inspect network flows, and identify patterns across large datasets. The project can be used to build self hosted alternatives to internet scanning services such as Shodan or Censys, giving organizations full control over their own reconnaissance infrastructure. -
9
discover
Automation framework for reconnaissance and penetration testing tasks
Discover is a collection of custom Bash scripts designed to automate many common tasks involved in penetration testing workflows. The project brings together a variety of security testing functions into a single framework that simplifies reconnaissance, scanning, and enumeration processes. It provides a menu-driven interface that allows security professionals to quickly launch different tools and scripts without manually executing each command. The framework helps streamline activities such... -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
10
Tookie-OSINT
Username OSINT tool for discovering accounts across many websites
Tookie-OSINT is an open source intelligence tool designed to help security researchers, ethical hackers, and investigators discover online accounts associated with a specific username. It automates the process of searching for usernames across multiple websites, making it easier to identify a person's presence on different platforms. By entering a target username, Tookie-OSINT scans a list of supported sites and checks whether the username exists on those platforms. This approach removes the... -
11
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying... -
12
IntelOwl
Centralized platform for automated threat intelligence analysis
IntelOwl is an open source platform designed to manage and enrich threat intelligence data at scale. It provides a centralized environment where security analysts can gather information about suspicious files and observables such as IP addresses, domains, URLs, or hashes using a single API request. The platform integrates numerous online intelligence sources and advanced malware analysis tools, enabling users to obtain comprehensive threat intelligence without manually querying multiple... -
13
Raccoon
High-performance reconnaissance and vulnerability scanning tool
Raccoon is a high-performance offensive security tool designed to assist with reconnaissance and vulnerability scanning during penetration testing and security assessments. It automates several common reconnaissance tasks, allowing security professionals to quickly gather information about a target system or web application. The tool combines multiple scanning techniques into a single workflow, helping users identify potential weaknesses, exposed services, and accessible resources on a... -
14
dnstwist
Detects phishing and lookalike domains using DNS fuzzing techniques
dnstwist is an open source cybersecurity tool designed to identify malicious or suspicious domain names that imitate legitimate websites. It works by generating a large set of domain name permutations based on a target domain and analyzing whether any of those variants are actively registered or used. These permutations simulate common techniques used in phishing attacks, typosquatting, and brand impersonation campaigns. Security teams can use the tool to discover potential threats where... -
15
cloud_enum
Multi-cloud OSINT tool for discovering public cloud resources
cloud_enum is an open source reconnaissance and OSINT tool designed to discover publicly accessible cloud resources across major cloud providers. It focuses on enumerating assets in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform using keyword-based discovery techniques. It works by taking user-provided keywords and generating variations through mutation wordlists, then testing these combinations against common cloud service naming patterns. cloud_enum performs both... -
16
reNgine
Automated framework for web application reconnaissance and scanning
reNgine is an automated reconnaissance framework designed to simplify and enhance the process of gathering information about web applications during security assessments. It provides a streamlined workflow for penetration testers, bug bounty hunters, and security teams who need to perform reconnaissance efficiently and at scale. The platform integrates multiple open-source reconnaissance tools into a unified environment with a configurable scanning engine and an intuitive web interface.... -
17
linkedin2username
Generate probable usernames from LinkedIn company employee lists
linkedin2username is an open source OSINT (Open Source Intelligence) tool designed to generate lists of potential usernames by scraping employee information from a company’s LinkedIn page. It logs into LinkedIn using valid user credentials and collects publicly visible employee names associated with a specified organization. Using these names, it automatically generates multiple possible username formats that organizations commonly use for accounts or email addresses. This process helps... -
18
fsociety
Modular CLI framework for managing penetration testing tools
...Instead of implementing its own security scanners, the framework acts as a wrapper and orchestrator that helps users discover, install, and execute tools from various GitHub repositories. Its modular architecture organizes tools into categories such as information gathering, networking, web application security, and password testing. This structure allows users to quickly navigate through different security tasks while maintaining a consistent interface. fsociety can automatically clone and manage required tools, reducing the manual effort typically needed to set up a penetration testing toolkit. fsociety is distributed as a Python package. -
19
YoungerSibling
YoungerSibling: Cross-platform OSINT tool for quick data gathering.
YoungerSibling is a Python-based terminal utility script designed for educational purposes. It provides a set of useful tools to perform tasks like searching the web, performing lookups (Google search, IP lookup, username lookup, etc.), and extracting metadata from images, directly from the terminal. This project aims to help students, developers, and hobbyists learn about web scraping, API usage, and terminal interaction with Python. -
20
OnionSearch
Search multiple Tor .onion engines at once and collect hidden links.
...It also offers flexible command-line options that allow users to limit results, choose which engines to query, and export collected data. By automating searches across several dark web search engines, OnionSearch simplifies the process of discovering information on hidden services. -
21
NExfil
Fast OSINT tool for discovering web profiles by username
NExfil is an open source OSINT (Open Source Intelligence) tool designed to locate user profiles across the web based on a given username. Developed in Python, the tool automates the process of checking hundreds of websites to determine whether a specific username exists on those platforms. By performing automated queries across numerous services, NExfil helps investigators, researchers, and security professionals quickly identify potential accounts associated with a particular username. ... -
22
paramspider
Mine parameterized URLs from web archives for security testing
ParamSpider is an open source command-line tool designed to discover URLs that contain parameters by mining historical data from web archives such as the Wayback Machine. It helps security researchers, penetration testers, and bug bounty hunters collect potential attack surfaces by automatically gathering archived URLs related to a specific domain. Instead of returning every discovered URL, the tool intelligently filters results to highlight parameterized endpoints that are more useful for... -
23
Yark
Simple OSINT tool for archiving and browsing YouTube channels offline
Yark is an open source command-line tool designed to simplify the process of archiving YouTube channels for research, analysis, or personal preservation. The project focuses on OSINT (Open Source Intelligence) workflows by allowing users to collect and store videos, metadata, and thumbnails from a YouTube channel in a structured local archive. Instead of simply downloading individual videos, Yark creates a self-contained archive directory that includes metadata files and organized folders... -
24
Moriarty Project
Web-based OSINT tool for investigating phone number information
Moriarty Project is an open source web-based investigation tool designed to gather publicly available information about phone numbers. It allows users to input a phone number and analyze various details related to that number through multiple investigation features. It performs information gathering by scraping data from online sources to retrieve insights such as owner information, spam risk, and related web references. Users can select specific investigation features to run individually or... -
25
ReconSpider
Most Advanced Open Source Intelligence (OSINT) Framework
...Reconnaissance is a mission to obtain information by various detection methods, about the activities and resources of an enemy or potential enemy, or geographic characteristics of a particular area. A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).
