Search Results for "spider-man"
Sort By:
Showing 508 open source projects for "spider-man"
View related business solutions-
Forever Free Full-Stack Observability | Grafana CloudBuilt on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Spider
High-performance Rust web crawler and scraper for large-scale data
Spider can operate concurrently across many pages, allowing it to gather large datasets in a short period of time. Spider also provides mechanisms for subscribing to crawl events so developers can process page data such as URLs, status codes, or HTML content as it is discovered. It supports advanced capabilities such as headless browser rendering, background crawling tasks, and configurable rules that control crawl depth or ignored paths. -
2
xhs-spider
Desktop tool for collecting and exporting Xiaohongshu post data
XHS-Spider is a desktop data collection tool designed to gather content and metadata from the Xiaohongshu platform. It provides a graphical interface that allows users to explore posts, collect information, and download media such as images and videos from individual notes or search results. It was developed primarily as a learning project to demonstrate approaches to building web crawlers and experimenting with technologies such as WebView2 and WPF UI. -
3
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. -
4
spider_collection
Collection of Python web scraping scripts for data extraction tasks
spider_collection is a collection of Python web crawler scripts created primarily for experimentation, learning, and practical scraping tasks. spider_collection gathers multiple independent spiders designed to collect data from different platforms and services, demonstrating a variety of scraping techniques and workflows. These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages.... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
EasySpider
A visual no-code/code-free web crawler/spider
A visual code-free/no-code web crawler/spider, supporting both Chinese and English. -
6
libzip
A C library for reading, creating, and modifying zip archives.
...Files can be added from data buffers, files, or compressed data copied directly from other zip archives. Changes made without closing the archive can be reverted. The API is documented by man pages. libzip is fully documented via man pages. HTML versions of the man pages are on libzip.org and in the man directory. You can start with libzip(3), which lists all others. Example source code is in the examples and src subdirectories. -
7
FEAPDER
Powerful Python crawler framework for scalable web scraping tasks
...It focuses on providing a developer-friendly environment that makes it easier to create, run, and manage crawlers for a variety of data collection tasks. It includes several built-in spider types, such as AirSpider, Spider, TaskSpider, and BatchSpider, which address different crawling scenarios ranging from lightweight scraping to distributed and batch-based jobs. feapder supports features such as breakpoint resume, allowing crawlers to continue from where they stopped without losing progress. It also integrates monitoring and alerting capabilities to help developers track crawler performance and detect issues during execution. feapder includes browser rendering support for handling dynamic web pages and provides mechanisms for large-scale data deduplication during crawling. -
8
SSH-MITM
Server for security audits supporting public key authentication
ssh man-in-the-middle (ssh-mitm) server for security audits supporting publickey authentication, session hijacking and file manipulation. SSH-MITM is a man in the middle SSH Server for security audits and malware analysis. Password and publickey authentication are supported and SSH-MITM is able to detect, if a user is able to login with publickey authentication on the remote server. -
9
-
8 Monitoring Tools in One APM. Install in 5 Minutes.AppSignal works out of the box for Ruby, Elixir, Node.js, Python, and more. 30-day free trial, no credit card required.
-
10
Scrapy-Redis
Redis-based components for Scrapy
...Version 0.3 changed the requests serialization from marshal to cPickle, therefore persisted requests using version 0.2 will not able to work on 0.3. The class scrapy_redis.spiders.RedisSpider enables a spider to read the urls from redis. The urls in the redis queue will be processed one after another, if the first request yields more requests, the spider will process those requests before fetching another url from redis. -
11
tldr-pages
Collaborative cheatsheets for console commands
The tldr-pages project is a collection of community-maintained help pages for command-line tools, that aims to be a simpler, more approachable complement to traditional man pages. The tldr pages are a community effort to simplify the beloved man pages with practical examples. Maybe you are new to the command-line world? Or just a little rusty? Or perhaps you can't always remember the arguments to lsof, or tar? There seems to be room for simpler help pages, focused on practical examples. This repository is just that, an ever-growing collection of examples for the most common UNIX, Linux, macOS, SunOS and Windows command-line tools. ... -
12
bettercap
The Swiss Army knife for 802.11, BLE, IPv4 and IPv6 networks
bettercap is a powerful, easily extensible and portable framework written in Go which aims to offer to security researchers, red teamers and reverse engineers an easy to use, all-in-one solution with all the features they might possibly need for performing reconnaissance and attacking WiFi networks, Bluetooth Low Energy devices, wireless HID devices and Ethernet networks. -
13
apk-mitm
Application that automatically prepares Android APK files for HTTPS
Inspecting a mobile app's HTTPS traffic using a proxy is probably the easiest way to figure out how it works. However, with the Network Security Configuration introduced in Android 7 and app developers trying to prevent MITM attacks using certificate pinning, getting an app to work with an HTTPS proxy has become quite tedious. -
14
go-mitmproxy
mitmproxy implemented with golang
go-mitmproxy is a Golang implementation of mitmproxy that supports man-in-the-middle attacks and parsing, monitoring, and tampering with HTTP/HTTPS traffic. Parses HTTP/HTTPS traffic and displays traffic details via a web interface. Supports a plugin mechanism for easily extending functionality. Various event hooks can be found in the examples directory. HTTPS certificate handling is compatible with mitmproxy and stored in the ~/.mitmproxy folder. -
15
Proxify
A versatile and portable proxy for capturing HTTP/HTTPS traffic
Swiss Army Knife Proxy for rapid deployments. Supports multiple operations such as request/response dump, filtering and manipulation via DSL language, upstream HTTP/Socks5 proxy. Additionally, a replay utility allows to import the dumped traffic (request/responses with correct domain name) into BurpSuite or any other proxy by simply setting the upstream proxy to proxify. -
16
Proxyee
HTTP proxy server,support HTTPS & websocket
Proxyee is a JAVA-written HTTP proxy server library that supports HTTP, HTTPS, and WebSocket protocols, and supports MITM (Man-in-the-middle), which can capture and tamper with HTTP, and HTTPS packets. The CA certificate (src/resources/ca.crt) from the project needs to be imported to a trusted root certificate authority. The CA certificate (src/resources/ca.crt) from the project needs to be imported to a trusted root certificate authority. You can use the CertDownIntercept interceptor to enable the web certificate download feature. -
17
mitmproxy
A free and open source interactive HTTPS proxy
mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of... -
18
AWS Secrets Manager Python caching
Enables in-process caching of secrets for Python applications
The AWS Secrets Manager Python caching client enables in-process caching of secrets for Python applications. To use this client you must have Python 3.6 or newer. Use of Python versions 3.5 or older are not supported. An Amazon Web Services (AWS) account to access secrets stored in AWS Secrets Manager. To create an AWS account, go to Sign In or Create an AWS Account and then choose I am a new user. Follow the instructions to create an AWS account. To create a secret in AWS Secrets Manager,... -
19
AWS Secrets Manager JDBC Library
The AWS Secrets Manager JDBC Library for Java developers
The AWS Secrets Manager JDBC Library enables Java developers to easily connect to SQL databases using secrets stored in AWS Secrets Manager. We provide database drivers that intercept calls to real database drivers and swap out secret IDs for actual login credentials. This prevents hard-coding database credentials into your application code. This can be integrated into your app through a few configuration file changes. Change the jdbc url to one that our driver will intercept (starting with... -
20
SimpleX
The first messaging platform operating without user identifiers
Other apps have user IDs: Signal, Matrix, Session, Briar, Jami, Cwtch, etc. SimpleX does not, not even random numbers. This radically improves your privacy. The video shows how you connect to your friend via their 1-time QR-code, in person or via a video link. You can also connect by sharing an invitation link. Temporary anonymous pairwise identifiers SimpleX uses temporary anonymous pairwise addresses and credentials for each user contact or group member. It allows to deliver messages... -
21
Grab Framework Project
Web Scraping Framework
...The single request/response API that allows you to build network request, perform it and work with the received content. The API is built on top of urllib3 and lxml libraries. The Spider API to build asynchronous web crawlers. You write classes that define handlers for each type of network request. Each handler is able to spawn new network requests. Network requests are processed concurrently with a pool of asynchronous web sockets. Grab provides interface called Spider to develop multithreaded web-site scrapers. -
22
Scrapling
An adaptive Web Scraping framework
...The framework includes advanced fetchers capable of bypassing anti-bot protections such as Cloudflare Turnstile using stealth and browser automation techniques. Its powerful spider system supports multi-session crawling, pause and resume functionality, and real-time streaming of scraped data. Scrapling combines high performance, memory efficiency, and extensive async support to deliver blazing-fast scraping workflows. With a developer-friendly API, CLI tools, MCP server integration for AI-assisted extraction, and Docker support, it offers a complete solution for modern web scrapers. -
23
Atlantis iOS
A lightweight and powerful iOS framework for intercepting HTTP/HTTPS
...Intuitive, thoughtful, and built with meticulous attention to detail. Dive into the network level to diagnose and fix problems with reliable and powerful tools. Proxyman acts as a man-in-the-middle server that captures the traffic between your applications and SSL Web Server. With a built-in macOS setup, so you can inspect your HTTP/HTTPS Request and Responses in plain text with just one click. Narrow down your search with Proxyman's Multiple Filters. You can combine complex filtered criteria like Protocol, Content-Type, URL, Request Header, Response Header, Body, etc that find exact what you're looking for. -
24
Impacket
A collection of Python classes for working with network protocols
Impacket is a collection of Python classes designed for working with network protocols. It was primarily created in the hopes of alleviating some of the hindrances associated with the implementation of networking protocols and stacks, and aims to speed up research and educational activities. It provides low-level programmatic access to packets, and the protocol implementation itself for some of the protocols, like SMB1-3 and MSRPC. It features several protocols, including Ethernet, IP, TCP,... -
25
DotnetSpider
Lightweight .NET framework for fast web crawling and data scraping
DotnetSpider is a web crawling and data extraction framework built on the .NET Standard platform. It is designed to help developers create efficient and scalable crawlers for collecting structured data from websites. It provides a high-level API that simplifies the process of defining spiders, managing requests, and extracting content from web pages. Developers can create custom spiders by extending base classes and configuring pipelines that handle downloading, parsing, and storing...
