Search Results for "linux proxy scraper"
Sort By:
Showing 83 open source projects for "linux proxy scraper"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
Linkedin Scraper
A library that scrapes Linkedin for user data
Linkedin Scraper is a library that scrapes Linkedin for user data. Version 2.0.0 and before is called linkedin_user_scraper and can be installed via pip3 install --user linkedin_user_scraper. The reason is that LinkedIn has recently blocked people from viewing certain profiles without having previously signed in. So by setting scrape=False, it doesn't automatically scrape the profile, but Chrome will open the linkedin page anyways. You can login and logout, and the cookie will stay in the... -
2
nginx-proxy
Automated nginx proxy for Docker containers using docker-gen
nginx-proxy sets up a container running nginx and docker-gen. docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped. The containers being proxied must expose the port to be proxied, either by using the EXPOSE directive in their Dockerfile or by using the --expose flag to docker run or docker create and be in the same network. By default, if you don't pass the --net flag when your nginx-proxy container is created, it will only be... -
3
proxy.py
Utilize all available CPU cores for accepting new client connections
proxy.py is made with performance in mind. By default, proxy.py will try to utilize all available CPU cores to it for accepting new client connections. This is achieved by starting AcceptorPool which listens on configured server port. Then, AcceptorPool starts Acceptor processes (--num-acceptors) to accept incoming client connections. Alongside, if --threadless is enabled, ThreadlessPool is setup which starts Threadless processes (--num-workers) to handle the incoming client connections.... -
4
SMTP Tunnel Proxy
A high-speed covert tunnel that disguises TCP traffic as SMTP email
SMTP Tunnel Proxy is a high-speed covert tunneling proxy that disguises regular TCP traffic as legitimate SMTP email communication to evade deep packet inspection (DPI) firewalls and censorship systems. It implements a SOCKS5 proxy interface on the client that wraps outbound traffic into an SMTP-like handshake (EHLO, STARTTLS, AUTH) and encrypted payload, making the session appear to DPI systems as a normal email exchange. -
Stop Cyber Threats with VM-Series Next-Gen Firewall on AzureGain integrated visibility across all traffic in a single pass. Deploy Palo Alto Networks VM-Series to determine application identity and content while automating security policy updates via rich APIs.
-
5
XX-Net
A web proxy tool
XX-Net is an easy-to-use, anti-censorship web proxy tool from China. It includes GAE_proxy and X-Tunnel, with support for multiple platforms. -
6
AimiliVPN
aimili-vpngate is a proxy tool that allows Linux to use clean IPs
...Users can deploy it with a one-command installer on common Linux distributions such as Debian, Ubuntu, CentOS, and Alpine. It is useful for users who want a lightweight, self-hosted gateway for managing VPNGate node selection and local proxy access. -
7
JobFunnel
Scrape job websites into a single spreadsheet with no duplicates.
Scrape job websites into a single spreadsheet with no duplicates. Automated tool for scraping job postings into a .csv file. You can search for jobs with YAML configuration files or by passing command arguments. By performing regular scraping and reviewing, you can cut through the noise of even the busiest job markets. Run funnel with your settings YAML to populate your master CSV file with jobs from available providers. JobFunnel can be easily automated to run nightly with crontab. If you... -
8
Mihomo
A simple Python Pydantic model for Honkai
Mihomo is a Python client library leveraging Pydantic to model parsed Honkai: Star Rail user data from the Mihomo public API. It provides structured types, type hints, and convenience methods to fetch and transform player profiles, daily stats, and character details efficiently. -
9
CyberScraper 2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
CyberScraper 2077 is not just another web scraping tool – it's a glimpse into the future of data extraction. Born from the neon-lit streets of a cyberpunk world, this AI-powered scraper uses OpenAI, Gemini and LocalLLM Models to slice through the web's defenses, extracting the data you need with unparalleled precision and style. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
paqctl
Unified proxy manager for bypassing firewalls
paqctl is a unified command-line manager for deploying and controlling bypass proxy setups across client and server machines. It is designed for users who run a server component on a VPS and a local client on Windows, macOS, or Linux. The tool supports two different approaches: Paqet, which uses KCP over raw sockets for simpler cases, and GFW-Knocker, which combines TCP and QUIC tunneling for heavier censorship scenarios. -
11
mitmproxy
A free and open source interactive HTTPS proxy
mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of... -
12
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper
Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations. -
13
ScrapeGraphAI
Python scraper based on AI
Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you. -
14
tumblr-crawler
Python crawler to download photos and videos from Tumblr blogs
tumblr-crawler is an open source Python-based utility designed to download media content from Tumblr blogs. It provides a script that automatically retrieves photos and videos from specified Tumblr sites and saves them locally for offline access. Users can specify one or multiple blogs to crawl by editing a configuration file or by passing parameters through the command line. Once executed, the script fetches media from the Tumblr API and stores the downloaded files in folders named after... -
15
MDCx
Movie metadata scraper and organizer for media libraries and NFO
MDCx is an open source media metadata scraping and organization tool designed to automate the process of collecting detailed information for movie files. It retrieves metadata from multiple online sources and applies it to local media collections, helping users maintain structured and well-organized libraries. MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. It also supports... -
16
urllib3
Python HTTP library with thread-safe connection pooling
urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. Thread safety, connection pooling. Client-side TLS/SSL verification. File uploads with multipart encoding. Helpers for retrying requests and dealing with HTTP redirects. Support for gzip, deflate, brotli, and zstd encoding. Proxy support for HTTP and SOCKS. 100% test coverage. Professional support for urllib3 is available as part of the Tidelift Subscription.... -
17
rnet
Python HTTP client with TLS and HTTP/2 fingerprint emulation support
rnet is an ergonomic and modular Python HTTP client designed for developers who need advanced control over network requests and protocol behavior. It provides a flexible API for making HTTP requests while supporting both asynchronous and blocking workflows, allowing it to integrate easily into different Python applications and runtimes. rnet focuses on low-level protocol customization, giving users fine-grained control over TLS and HTTP/2 configuration in order to emulate specific browser... -
18
gain
Asyncio-based Python framework for building fast web crawling spiders
Gain is a Python web crawling framework designed to simplify the process of building efficient and scalable web scrapers. It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and... -
19
MasterHttpRelayVPN
Domain-fronted HTTP/SOCKS5 proxy tunneling traffic through Google Apps
MasterHttpRelayVPN is a networking tool designed to route traffic through HTTP-based relay systems, enabling VPN-like functionality over standard web protocols. It focuses on bypassing network restrictions by tunneling data through HTTP requests, making it effective in constrained or filtered environments. The system supports secure communication and can be configured to relay traffic across multiple endpoints. It is particularly useful for users who need to access restricted networks or... -
20
HTTPX
A next generation HTTP client for Python
HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. HTTPX should currently be considered in beta. A 1.0 release is expected to be issued sometime in 2021. International domains and URLs, keep-alive and connection pooling, sessions with cookie persistence, browser-style SSL verification. Basic/digest authentication, elegant key/value cookies, automatic decompression. Automatic content decoding, unicode response... -
21
Nitter
Alternative Twitter front-end
Nitter is an open-source alternative frontend for Twitter designed to provide a privacy-focused and lightweight way to browse content without interacting directly with the official platform. It acts as a proxy between the user and Twitter, ensuring that requests are handled by the backend server rather than exposing the user’s IP address or browser fingerprint. The interface is intentionally minimalistic and removes elements such as advertisements, tracking scripts, and algorithmic... -
22
Microsoft Azure CLI
Azure command-line interface
A great cloud needs great tools; we're excited to introduce Azure CLI, our next-generation multi-platform command-line experience for Azure. Take a test run now from Azure Cloud Shell! We support tab completion for groups, commands, and some parameters. You can use the --query parameter and the JMESPath query syntax to customize your output. With the Azure CLI Tools Visual Studio Code extension, you can create .azcli files and use these features. IntelliSense for commands and their... -
23
new-pac
Scientific Internet access
This repository aggregates tools, guides, and configuration files aimed at enabling network access in restrictive environments across desktop and mobile platforms. It collects client applications, one-click browser bundles, configuration examples, and references for widely used proxy and tunneling technologies. The emphasis is on approachability: instructions, packaged builds, and links are organized so non-experts can find a workable setup for their device. Because endpoint reliability and... -
24
Grab Framework Project
Web Scraping Framework
Grab is a python framework for building web scrapers. With Grab you can build web scrapers of various complexity, from simple 5-line scripts to complex asynchronous website crawlers processing millions of web pages. Grab provides an API for performing network requests and for handling the received content e.g. interacting with DOM tree of the HTML document. The single request/response API that allows you to build network request, perform it and work with the received content. The API is... -
25
Scweet
Scrape tweets, profiles, followers and following from Twitter/X
Scweet is a Python-based Twitter/X scraping library and CLI designed to collect tweets, profile timelines, followers, following lists, and user profile data without requiring the official Twitter/X API or a developer account. Instead of depending on deprecated unauthenticated scraping methods, it works by using X’s web GraphQL API together with authenticated browser cookies, which gives it a more current and practical approach for data extraction. The project supports a broad set of...
