Search Results for "python web crawler" - Page 2
Sort By:
Showing 36 open source projects for "python web crawler"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
Hakrawler
Fast Go web crawler for discovering URLs and web app endpoints
hakrawler is a lightweight command-line web crawler built in Go that is designed to quickly discover URLs, endpoints, and assets within web applications. It is primarily used during the reconnaissance phase of security testing, bug bounty hunting, and penetration testing. It works by automatically crawling web pages and extracting links, JavaScript file locations, and other resources that may reveal additional attack surface or hidden functionality. hakrawler is implemented as a simple and efficient crawler using the Gocolly library, which allows it to perform fast and concurrent crawling of web pages. ... -
2
ReconSpider
Most Advanced Open Source Intelligence (OSINT) Framework
...Reconnaissance is a mission to obtain information by various detection methods, about the activities and resources of an enemy or potential enemy, or geographic characteristics of a particular area. A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). -
3
HostHunter
OSINT reconnaissance tool for discovering hostnames from IP addresses
HostHunter is an open source reconnaissance tool designed to discover and extract hostnames associated with a large set of IPv4 or IPv6 addresses. It helps security professionals map IP addresses to virtual hostnames using a combination of OSINT data sources and active reconnaissance techniques. This approach enables users to identify hidden or additional services that may be hosted behind a single IP address. By correlating hostname information from certificates, APIs, HTTP headers, and... -
4
Trape
OSINT tool for tracking users and analyzing browser data online
Trape is an open source OSINT analysis and research tool designed to track and analyze users on the internet in real time. The project focuses on demonstrating how web browsers can reveal sensitive information about users while interacting with websites and online services. It provides researchers, security professionals, and organizations with a platform for studying how attackers could gather intelligence through social engineering techniques. The tool can clone websites and monitor... -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
5
TIDoS
Python web penetration testing framework with modular security tools
TIDoS-Framework is an open source web application penetration testing framework designed to assist security researchers and ethical hackers in identifying vulnerabilities in web systems. It provides a comprehensive environment for performing multiple phases of security assessment, including reconnaissance, scanning, enumeration, vulnerability analysis, and exploitation. -
6
BlackWidow
Python web scanner for OSINT gathering and OWASP vulnerability fuzzing
BlackWidow is a Python-based web application scanning tool designed to crawl target websites and collect open-source intelligence (OSINT) while identifying potential security vulnerabilities. It functions as a web spider that systematically explores a site to gather valuable information such as URLs, dynamic parameters, subdomains, email addresses, and phone numbers associated with the target domain. -
7
FavFreak
Favicon hash–based reconnaissance tool for security research
FavFreak is an open source reconnaissance tool designed to assist security researchers, bug bounty hunters, and penetration testers in identifying web technologies using favicon hashes. It works by taking one or more URLs as input and automatically retrieving the favicon.ico file associated with each target website. After fetching the favicon, it calculates a hash value and organizes the scanned domains, subdomains, or IP addresses according to these hashes. FavFreak then compares the... -
8
AttackSurfaceMapper
Automated tool for mapping & expanding organization’s attack surface
AttackSurfaceMapper (ASM) is a reconnaissance and attack surface discovery tool designed to automate the process of mapping potential targets within an organization's infrastructure. It combines open source intelligence (OSINT) with selective active reconnaissance techniques to expand and analyze a target’s external attack surface. Users can supply domains, subdomains, or IP addresses as input, and applies multiple discovery methods to identify additional related assets such as new... -
9
Fav-up
Look up IP addresses using favicon hashes via Shodan
fav-up is an open source reconnaissance utility designed to identify servers and websites by analyzing favicon hashes and querying the Shodan search engine. It calculates the hash of a favicon image and then uses that value to search Shodan for other hosts that use the same icon. This technique is commonly used in security research and OSINT investigations to discover related infrastructure or services that may belong to the same organization. fav-up can retrieve favicon data from several... -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
Buster
OSINT tool for discovering information linked to email addresses
Buster is an open source OSINT tool designed for email reconnaissance and information gathering. It helps investigators, security researchers, and penetration testers discover publicly available information related to email addresses and usernames. It can analyze an email address to identify associated social media accounts, references across the web, and potential data breaches linked to that email. It also performs reverse WHOIS lookups to discover domains that may have been registered... -
11
Photon
Incredibly fast crawler designed for OSINT
Photon is an extremely fast web crawler built specifically for OSINT and reconnaissance use cases. It is designed to extract URLs, endpoints, files, and other intelligence artifacts from target websites with minimal overhead. The crawler prioritizes speed and breadth, making it suitable for mapping web attack surfaces and discovering hidden resources. Photon is commonly used during early reconnaissance phases to build a comprehensive inventory of reachable assets. ...