Search Results for "python web crawler"

XX-Net is an easy-to-use, anti-censorship web proxy tool from China. It includes GAE_proxy and X-Tunnel, with support for multiple platforms.

mitmproxy is an open source, interactive SSL/TLS-capable intercepting HTTP proxy, with a console interface fit for HTTP/1, HTTP/2, and WebSockets. It's the ideal tool for penetration testers and software developers, able to debug, test, and make privacy measurements. It can intercept, inspect, modify and replay web traffic, and can even prettify and decode a variety of message types. Its web-based interface mitmweb gives you a similar experience as Chrome's DevTools, with the addition of...

X4G is a self-hosted gateway for operating VLESS tunnels over WebSocket and two XHTTP transport modes. It also includes an internal HTTP proxy for clients that need a conventional proxy endpoint. A web dashboard displays service statistics, hourly traffic charts, live connections, activity records, and error logs. Administrators can create unlimited connection links with individual traffic quotas, bandwidth limits, expiration dates, ports, and simultaneous IP restrictions. Subscription...

The main function of the crawler agent IP pool project is to regularly collect free agents published on the Internet for verification and storage, and to regularly verify and store agents to ensure the availability of agents, and to provide API and CLI. At the same time, you can also expand the proxy source to increase the quality and quantity of the proxy pool IP.

Jupyter Server Proxy lets you run arbitrary external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc) alongside your notebook server and provide authenticated web access to them using a path like /rstudio next to others like /lab. Alongside the Python package that provides the main functionality, the JupyterLab extension (@jupyterhub/jupyter-server-proxy) provides buttons in the JupyterLab launcher window to get to RStudio for example.

Rendora is a dynamic renderer to provide zero-configuration server-side rendering mainly to web crawlers in order to effortlessly improve SEO for websites developed in modern Javascript frameworks such as React.js, Vue.js, Angular.js, etc. Rendora works totally independently of your frontend and backend stacks. Rendora can be seen as a reverse HTTP proxy server sitting between your backend server (e.g. Node.js/Express.js, Python/Django, etc...) and potentially your frontend proxy server (e.g. nginx, traefik, apache, etc...) or even directly to the outside world that does actually nothing but transporting requests and responses as they are except when it detects whitelisted requests according to the config. ...

Proxenet is a hacker-friendly, DIY web proxy designed for web application penetration testing. Written in C, it allows interaction with higher-level languages like Python and Ruby for on-the-fly modification of HTTP requests and responses.

webStraktor is a programmable World Wide Web data extraction client. Its purpose is to scrape HTML based content via the HTTP protocol and extract relevant information. webStraktor features a scripting language to facilitate the collection, the extraction and the storage of information available on the web, including images. The scripting language uses elements of the Regular Expression and xPath syntax. The webStraktor scripting language has a small instruction set and its syntax is easy...

HTTP Replicator is a general purpose caching proxy server written in python. It reduces bandwidth by merging concurrent downloads and building a local 'replicated' file hierarchy, similar to wget -r. The cache is also accessible through a web interface

Open Source Software Tool for installing and configuring Squid: Proxy Server and Web Cache Daemon. Developed By: Ashwin Tumma

Proxy server that performs proof of concept session hijacking by receiving cookies from one or more NomNom Agents. It then offers the user an option to switch to any intercepted session. More features and site specific options later.

Shweby is a new ICAP enabled proxy server. This proxy passes HTTP requests and responses through one or many ICAP servers in order to be modified in their way to the origin server or to the client.

This project is a Python-based HTTP web proxy server that hooks into MySQL to store a full history of your browsing. Allows you to check out statistics about your browsing habits. Creates a personal portal page, has search features, multi-user, filters.