Search Results for "linux proxy scraper"
Sort By:
Showing 189 open source projects for "linux proxy scraper"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Linkedin Scraper
A library that scrapes Linkedin for user data
Linkedin Scraper is a library that scrapes Linkedin for user data. Version 2.0.0 and before is called linkedin_user_scraper and can be installed via pip3 install --user linkedin_user_scraper. The reason is that LinkedIn has recently blocked people from viewing certain profiles without having previously signed in. So by setting scrape=False, it doesn't automatically scrape the profile, but Chrome will open the linkedin page anyways. You can login and logout, and the cookie will stay in the... -
2
shot-scraper
A command-line utility for taking automated screenshots of websites
shot-scraper is a command-line utility for taking automated screenshots of web pages using a headless browser engine. After installation, a single command can capture a full-page screenshot of a URL and save it to a file, making it ideal for documentation, monitoring, and visual regression tasks. Under the hood it uses a modern browser (installed via a one-time shot-scraper install step) and exposes options for viewport size, full-page versus clipped screenshots, and device emulation. Beyond... -
3
proxy.py
Utilize all available CPU cores for accepting new client connections
proxy.py is made with performance in mind. By default, proxy.py will try to utilize all available CPU cores to it for accepting new client connections. This is achieved by starting AcceptorPool which listens on configured server port. Then, AcceptorPool starts Acceptor processes (--num-acceptors) to accept incoming client connections. Alongside, if --threadless is enabled, ThreadlessPool is setup which starts Threadless processes (--num-workers) to handle the incoming client connections.... -
4
nginx-proxy
Automated nginx proxy for Docker containers using docker-gen
nginx-proxy sets up a container running nginx and docker-gen. docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped. The containers being proxied must expose the port to be proxied, either by using the EXPOSE directive in their Dockerfile or by using the --expose flag to docker run or docker create and be in the same network. By default, if you don't pass the --net flag when your nginx-proxy container is created, it will only be... -
Build Data Resilience - Take the Assessment TodayIs your recovery strategy as strong as you think? Take this quick self-assessment to check your recovery readiness and gain tailored insights. In only 2 minutes, you'll learn where you fall on the recovery readiness scale.
-
5
SMTP Tunnel Proxy
A high-speed covert tunnel that disguises TCP traffic as SMTP email
SMTP Tunnel Proxy is a high-speed covert tunneling proxy that disguises regular TCP traffic as legitimate SMTP email communication to evade deep packet inspection (DPI) firewalls and censorship systems. It implements a SOCKS5 proxy interface on the client that wraps outbound traffic into an SMTP-like handshake (EHLO, STARTTLS, AUTH) and encrypted payload, making the session appear to DPI systems as a normal email exchange. -
6
XX-Net
A web proxy tool
XX-Net is an easy-to-use, anti-censorship web proxy tool from China. It includes GAE_proxy and X-Tunnel, with support for multiple platforms. -
7
AimiliVPN
aimili-vpngate is a proxy tool that allows Linux to use clean IPs
...Users can deploy it with a one-command installer on common Linux distributions such as Debian, Ubuntu, CentOS, and Alpine. It is useful for users who want a lightweight, self-hosted gateway for managing VPNGate node selection and local proxy access. -
8
JobFunnel
Scrape job websites into a single spreadsheet with no duplicates.
Scrape job websites into a single spreadsheet with no duplicates. Automated tool for scraping job postings into a .csv file. You can search for jobs with YAML configuration files or by passing command arguments. By performing regular scraping and reviewing, you can cut through the noise of even the busiest job markets. Run funnel with your settings YAML to populate your master CSV file with jobs from available providers. JobFunnel can be easily automated to run nightly with crontab. If you... -
9
CyberScraper 2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
CyberScraper 2077 is not just another web scraping tool – it's a glimpse into the future of data extraction. Born from the neon-lit streets of a cyberpunk world, this AI-powered scraper uses OpenAI, Gemini and LocalLLM Models to slice through the web's defenses, extracting the data you need with unparalleled precision and style. -
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
10
Mihomo
A simple Python Pydantic model for Honkai
Mihomo is a Python client library leveraging Pydantic to model parsed Honkai: Star Rail user data from the Mihomo public API. It provides structured types, type hints, and convenience methods to fetch and transform player profiles, daily stats, and character details efficiently. -
11
X4G
Fast and modern gateway for VLESS tunneling on WebSocket and XHTTP
X4G is a self-hosted gateway for operating VLESS tunnels over WebSocket and two XHTTP transport modes. It also includes an internal HTTP proxy for clients that need a conventional proxy endpoint. A web dashboard displays service statistics, hourly traffic charts, live connections, activity records, and error logs. Administrators can create unlimited connection links with individual traffic quotas, bandwidth limits, expiration dates, ports, and simultaneous IP restrictions. Subscription... -
12
mitmproxy
A free and open source interactive HTTPS proxy
mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of... -
13
paqctl
Unified proxy manager for bypassing firewalls
paqctl is a unified command-line manager for deploying and controlling bypass proxy setups across client and server machines. It is designed for users who run a server component on a VPS and a local client on Windows, macOS, or Linux. The tool supports two different approaches: Paqet, which uses KCP over raw sockets for simpler cases, and GFW-Knocker, which combines TCP and QUIC tunneling for heavier censorship scenarios. -
14
optillm
Optimizing inference proxy for LLMs
OptiLLM is an optimizing inference proxy for Large Language Models (LLMs) that implements state-of-the-art techniques to enhance performance and efficiency. It serves as an OpenAI API-compatible proxy, allowing for seamless integration into existing workflows while optimizing inference processes. OptiLLM aims to reduce latency and resource consumption during LLM inference. -
15
XHS-Downloader
GUI/CLI tool for downloading Xiaohongshu
XHS-Downloader is a GUI/CLI tool for downloading Xiaohongshu (Little Red Book) content without watermarks, supporting both graphics and video posts. Prebuilt packages for Windows and macOS are available from Releases and GitHub Actions artifacts, so most users can run it by unzipping and launching the included executable. The project offers two execution paths—run the compiled app or run from source—and documents default download and configuration paths to simplify first use. Recent releases... -
16
Dispatcharr
Your Ultimate IPTV & Stream Management Companion
Dispatcharr is a self-hosted IPTV and streaming management platform built for people who want full control over their playlists, streams, and program guide (EPG) data. It lets you import M3U/EPG sources, curate and organize them with filtering and grouping, and serve them reliably to clients like Plex, Jellyfin, and Emby with advanced proxying and failover support. Designed with both simple personal use and larger self-hosted setups in mind, Dispatcharr offers real-time streaming statistics,... -
17
MediaCrawler
Self-media platform crawler
MediaCrawler is an open-source tool for collecting public content from major Chinese social media platforms. It supports Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, Zhihu, and other services. The project uses Playwright to automate browser login and preserve authenticated sessions. It obtains required request signatures from the active browser context, avoiding complex JavaScript reverse engineering. Users can search by keyword, crawl specific posts, collect nested comments, and... -
18
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper
Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations. -
19
ScrapeGraphAI
Python scraper based on AI
Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you. -
20
tumblr-crawler
Python crawler to download photos and videos from Tumblr blogs
tumblr-crawler is an open source Python-based utility designed to download media content from Tumblr blogs. It provides a script that automatically retrieves photos and videos from specified Tumblr sites and saves them locally for offline access. Users can specify one or multiple blogs to crawl by editing a configuration file or by passing parameters through the command line. Once executed, the script fetches media from the Tumblr API and stores the downloaded files in folders named after... -
21
Emissary-ingress
open source Kubernetes-native API gateway for microservices
Emissary-Ingress is the most popular API Gateway Kubernetes-native - open-source, that delivers scalability, flexibility, and simplicity for some of the world's largest Kubernetes installations. Emissary-Ingress is an open source CNCF incubating project, and it uses the ubiquitous Envoy Proxy at its core. Once your organization hits a certain scale, a reverse proxy or load balancer isn’t enough for traffic management. If you’re using Kubernetes, you’re probably already at that scale and need... -
22
Browser Use
Make websites accessible for AI agents
Browser Use is an AI-powered browser automation framework designed to let agents interact with websites just like humans do. It enables developers and AI systems to perform complex online tasks such as form filling, data extraction, and navigation through natural language instructions. Built with Python and compatible with modern LLMs, it integrates seamlessly with tools like ChatBrowserUse, Google Gemini, and Anthropic models. The platform supports both open-source deployment and a fully... -
23
MDCx
Movie metadata scraper and organizer for media libraries and NFO
MDCx is an open source media metadata scraping and organization tool designed to automate the process of collecting detailed information for movie files. It retrieves metadata from multiple online sources and applies it to local media collections, helping users maintain structured and well-organized libraries. MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. It also supports... -
24
gdown
Google Drive public file downloader when curl/wget fails
gdown is a Python downloader designed for public Google Drive files and folders that ordinary curl or wget commands may not retrieve correctly. It handles Drive confirmation pages and extracts file identifiers from several common share-link formats. Users can download individual files, recursively copy folders, choose output paths, or inspect resolved names as JSON. Google Docs, Sheets, and Slides can be exported to formats such as DOCX, XLSX, PPTX, PDF, and CSV. Interrupted transfers can... -
25
mcpo
A simple, secure MCP-to-OpenAPI proxy server
mcpo is a minimal bridge that exposes any MCP tool as an OpenAPI-compatible HTTP server. Instead of writing glue code, you point mcpo at an MCP server command and it generates REST endpoints and an OpenAPI spec that other systems (or LLM agent frameworks) can call immediately. This design lets you reuse a growing library of MCP servers with platforms that only understand HTTP+OpenAPI, unifying tool access across ecosystems. The project emphasizes “dead-simple” setup and pairs with Open WebUI...
