Open Source Python Internet Software - Page 67

Python app used to download (torrent) files from various RSS feeds. Designed for use with Transmission client...

Ruia is an asynchronous web scraping micro-framework built for Python that focuses on simplicity, speed, and flexibility when creating web crawlers. Ruia is powered by Python’s asyncio library along with aiohttp, enabling developers to perform concurrent network requests efficiently and scrape data from websites with minimal overhead. Ruia follows a “write less, run faster” philosophy, emphasizing concise code and streamlined spider development. It provides a structured approach to building scraping projects through components such as data items, spiders, middleware, and plugins. Developers can define structured fields to extract information from HTML content and process responses asynchronously to improve crawling performance. It also supports middleware and plugin systems that allow customization of request handling, response processing, and additional functionality.

Example Python script to work with S3.

It is a powerful data scraping tool that helps you extract information from various online sources. Easily collect data from Google SERP, Maps, Shopify, Zillow, and more. With a user-friendly interface, you can scrape and save data in JSON or Excel formats. Unlock insights from the web effortlessly with scrape-it.cloud API.

The scrapy-proxy-headers package is designed for adding proxy headers to HTTPS requests in Scrapy. In normal usage, custom headers put in request.headers cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at Proxy Server Requests over HTTPS. Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at ProxyMesh made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.

This project produces software to convert the Moby Shakespeare texts to XML and HTML. It also provides miscellaneous software for maintaing literature Web sites. The software is currently used to produce the Web site http://tech-two.mit.edu/Shakespeare/

An open collaborative translation system built with Zope. Translation projects are broken down into fragments. Users can either add new translations to fragments or vote on previous translations. The fragment translation with the most votes is used.

simple Python Server Page is a simple server-side scripting facility which is similar to ASP/JSP/PHP, and which uses Python as the scripting language. sPSP allows you to easily create dynamic content and implement Web-applications.

This project aims to help a user manage his/hers social networks across multiple communities keeping track with their activities in his/hers domains of interest

Socketify.py is a reliable, high-performance Python web framework for building large-scale app backends and microservices. With no precedents websocket performance and a really fast HTTP server that can delivery encrypted TLS 1.3 quicker than most alternative servers can do even unencrypted, cleartext messaging.

Solrscan is a tool for posting Solr format Xml documents to a Solr Index. It has support for full or incremental mode and maintains a cache of the current state of an index.

Command line interface for testing internet bandwidth using speedtest.net. It is not a goal of this application to be a reliable latency reporting tool. Latency reported by this tool should not be relied on as a value indicative of ICMP style latency. It is a relative value used for determining the lowest latency server for performing the actual speed test against. Speedtest CLI brings the trusted technology and global server network behind Speedtest to the command line. Measure internet connection performance metrics like download, upload, latency and packet loss natively without relying on a web browser. Test the internet connection of your Linux desktop, a remote server or even lower-powered devices such as the Raspberry Pi with the Speedtest Server Network. Set up automated scripts to collect connection performance data, including trends over time.

spider_collection is a collection of Python web crawler scripts created primarily for experimentation, learning, and practical scraping tasks. spider_collection gathers multiple independent spiders designed to collect data from different platforms and services, demonstrating a variety of scraping techniques and workflows. These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages. Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. It also contains examples of proxy pool integration and encapsulation to support more reliable crawling when working with sites that enforce request limits.

sqlit is a keyboard-first terminal UI that lets you connect to, browse, and query SQL databases quickly without relying on heavyweight GUI clients. It positions itself as a “lazygit-style” experience for databases, aiming for fast startup, intuitive navigation, and developer-friendly workflows directly inside your terminal. The tool supports a wide range of database providers, so you can use one interface across local databases, remote servers, and cloud-hosted instances rather than juggling multiple clients. It includes a connection manager so you can save connections and switch between them without constantly retyping CLI arguments or connection strings. For querying, it emphasizes productivity features like syntax highlighting, searchable query history, and vim-style keybindings so power users can move fast. For exploring data at scale, it can load and inspect very large result sets and provides filtering and fuzzy search to find rows and values efficiently.

Little SSH configuration Manager. Manages hosts in SSH client configuration file. (See man ssh_config) Written in Python.

StatiCMS is a content management system that is purely static: all output is generated offline. Thanks to a plugin system, all kinds of static data can be generated and a dependency checker makes building fast.

A Python agent and a web interface aiming to help the analysis and investigation of a legacy application. A set of machines, processes, databases, programs etc ... all communicating with each other, manipulating your data, and whose software architecture has become, with time, complicated, difficult to understand, and undocumented. Data are aggregated with an RDF inference engine, creating a global vision of the business information processing.

Synchro is a desktop ping.fm, hellotxt-like project thought to be useful as coordinator for each important service on the net (statuses).

tagin! is a WiFi-based point-of-interest (POI) service that allows users to tag outdoor and indoor locations.

This is the Open Source RESTful client for the take.io platform.

TamTam is collaborative web application. It is mostly based on the concept of WikiWiki, but has many features that are not a part of standard Wiki implementations & clones.

Apple has a utility in /usr/bin named tccutil, but it only supports one command, which is to reset the entire database. It has been like this for many versions of macOS. I wanted a command-line utility that would be able to add, remove, list, and take other actions. This tool needs SIP disabled in order to function. The risk of doing so is up to you. tccutil.py can be installed without any additional software. Depending how you have your $PATH variable setup, you can simply type tccutil (instead of the full path) and it will run this utility instead of Apple's. This utility needs super-user priveleges for most operations. It is important that you either run this as root or use sudo, otherwise it won't work and you will end up with “permission denied” errors.