Open Source Python Internet Software - Page 44

Named after a well know product from Microsoft, SandStorm is a framework for creating modular middle-end web products.

Build fast, run fast with Sanic! Sanic is a Python 3.6+ web server and web framework designed to go fast. It provides a way to get a highly performant HTTP server up and running fast, while also making it easy to build, expand, and eventually scale. Sanic aspires to be as simple as possible while delivering the performance that you require. It allows the usage of the async/await syntax added in Python 3.5, so your code is guaranteed to be non-blocking and speedy. It's also ASGI compliant, so it's possible to deploy with an alternative ASGI webserver.

Sarfe stands for "Send And Receive Files Easily". It aims to provide an easy and efficient interface for file transfers.

Satrap is an online address management system. Users can change their own records via a web-based interface, but there are also special administration groups.

The OO component based XML processing framework.The goal is to make the XML developer's life easier.It's going ot achieve that by providing reusable components for all stages of XML processsing with easy language and application binding.

SchemaDoc is a XML-based markup language for documenting XML schemas. The work products include both the vocabulary and a set of tools for combining it with the schema source (e.g. a DTD) to produce documentation in HTML, XML DocBook, LaTeX, etc.

Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you.

Web spider and SERP scrapper

You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3 changed the requests serialization from marshal to cPickle, therefore persisted requests using version 0.2 will not able to work on 0.3. The class scrapy_redis.spiders.RedisSpider enables a spider to read the urls from redis. The urls in the redis queue will be processed one after another, if the first request yields more requests, the spider will process those requests before fetching another url from redis.

Scrapyd can manage multiple projects and each project can have multiple versions uploaded, but only the latest one will be used for launching new spiders. A common (and useful) convention to use for the version name is the revision number of the version control tool you’re using to track your Scrapy project code. For example: r23. The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd is an application (typically run as a daemon) that listens to requests for spiders to run and spawns a process for each one. Scrapyd also runs multiple processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load.

Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization. Make sure that Scrapyd has been installed and started on all of your hosts. Start ScrapydWeb via command scrapydweb. (a config file would be generated for customizing settings on the first startup.) Add your Scrapyd servers, both formats of string and tuple are supported, you can attach basic auth for accessing the Scrapyd server, as well as a string for grouping or labeling. You can select any number of Scrapyd servers by grouping and filtering, and then invoke the HTTP JSON API of Scrapyd on the cluster with just a few clicks.

A Trac plugin to allow pasting screenshots or other images captured or copied in the clipboard directly as attachements to tickets, Wiki pages, etc., without the need to first saving as images and then uploading them. Once the plugin is installed in Trac, you can easily attach a screenshot or any image you have in the clipboard to a Ticket or Wiki page, with one click.

ScrollServer is a Python based HTTP server that provides an html interface to the ScrollKeeper documentation database. It allows you to browse all documentation that is registered in your ScrollKeeper database using any web browser.

Scweet is a Python-based Twitter/X scraping library and CLI designed to collect tweets, profile timelines, followers, following lists, and user profile data without requiring the official Twitter/X API or a developer account. Instead of depending on deprecated unauthenticated scraping methods, it works by using X’s web GraphQL API together with authenticated browser cookies, which gives it a more current and practical approach for data extraction. The project supports a broad set of collection patterns, including searches by keyword, hashtag, user, date range, engagement thresholds, language, and location, making it useful for research, monitoring, and data gathering workflows. It is built for both local use and higher-volume runs, with support for proxies, dedicated accounts, and multi-account cookie handling to improve reliability at scale. Scweet also includes asynchronous method variants, a command-line interface, automatic credential persistence in a local database, etc.

A web based application that can search newsgroup names very easy. Use can search use Regular Expression for names and search by popularity.

The Secure Remote Log Monitor (SRLM) project provides client and server utilities that collect application or system log files from multiple systems over an untrusted network onto a central server for analysis and action.

Generates logs of typical formats that would often be found in a SOC. As of 31st January 2023, it supports IDS, Web Access and Endpoint log formats. Can generate a specific number of events in a linear fashion or use a waveform to add 'bumpiness' to your data. The code is modular and extensible, adding additional formats can be done with relative ease.

Security Monkey monitors AWS, GCP, OpenStack, and GitHub orgs for assets and their changes over time. Security Monkey monitors your AWS and GCP accounts for policy changes and alerts on insecure configurations. Support is available for OpenStack public and private clouds. Security Monkey can also watch and monitor your GitHub organizations, teams, and repositories. It provides a single UI to browse and search through all of your accounts, regions, and cloud services. The monkey remembers previous states and can show you exactly what changed, and when. Security Monkey can be extended with custom account types, custom watchers, custom auditors, and custom alerters. It works on CPython 2.7. It is known to work on Ubuntu Linux and OS X. Security Monkey accesses accounts to scan via credentials it is provided.

This is a Firewall and IDS front-end, the main purpose is simplify configuration of Iptables and Fwsnort and Psad management. The Firewall provides an experimental feature to decide in real-time about each outgoing connection.

A fast HTML5 parser with CSS selectors using Modest and Lexbor engines. Selectolax supports two backends: Modest and Lexbor. By default, all examples use the Modest backend. Most of the features between backends are almost identical, but there are still some differences. Currently, the Lexbor backend is in beta and missing some of the features. To use lexbor, just import the parser and use it in the similar way to the HTMLParser.

calendaring/event scheduling python product for zope

ServicemarX allows you to easily discover, bookmark, and share network services. It extends Bonjour (formerly known as Rendezvous) by expanding its scope Internet-wide. ServicemarX also enables Tiger's Wide Area capabilities. (full source in CVS below)

Shadow Watcher is a Peer to Peer log sharing tool for the security analyst. Consider it a "community watch" program which allows you to keep your system secure from the threats on the internet and help others do the same.

Shadow0049's Micro-Server is an attempt to make it easier for webpage developers to test PHP/MySQL pages. The entire thing is about 20MB when installed and can run independantly off of a flash drive or any other removable media. (CD's not recommended)

Shapley is a compilation of free PyGTK networking tools distributed under the BSD license.