Search Results for "filter"

Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization. Make sure that Scrapyd has been installed and started on all of your hosts. Start ScrapydWeb via command scrapydweb. (a config file would be generated for customizing settings on the first startup.) Add your Scrapyd servers, both formats of string and tuple are supported, you can attach basic auth for accessing the Scrapyd server, as well as a string for grouping or labeling. You can select any...

A minimal, fast GUI for quickly launching SSH sessions from your ~/.ssh/config. SSH keychain 🔑🔗 Type to filter hosts Arrow Up/Down: Navigate through the list of hosts Enter: Connect to the selected host Esc: Close the application Delete: Clear the filter

...Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3 changed the requests serialization from marshal to cPickle, therefore persisted requests using version 0.2 will not able to work on 0.3. ...

...Tasks are managed through queues, allowing different parts of the crawler to process work asynchronously and efficiently. PSpider defines a set of modules and utility classes that help developers manage crawling tasks, filter URLs, and process scraped content. By organizing crawling tasks into structured stages, PSpider allows developers to build scalable spiders while keeping the codebase relatively compact and readable. Its modular design also makes it easier to extend the framework with additional features or integrate it into existing Python projects.

...It supports several proxy protocols, including HTTP, HTTPS, SOCKS4, and SOCKS5, making it flexible for different networking and scraping scenarios. ProxyBroker can filter proxies based on criteria such as anonymity level, response time, country of origin, and DNS blacklist status. In addition to discovering and validating proxies, it can also function as a proxy server that distributes incoming requests across a rotating pool of working proxies. This capability allows users to route traffic through multiple proxies automatically, which can help with tasks.

Charm is a full-featured, cross-platfom blogging client for LiveJournal, Atom (Movable Type, Blogger), and MetaWeb (WordPress). It is console-based, all-text, and can be used entirely from the command line. It is written in Python.

Feedisto is your personal newspaper. Your newspaper is collocted from rss feeds, parsed by a bayesian filter to rate its relevance and served as a static html page or on a webserver to train the filter. Writing plugins is very easy.

Paniagua is a filter designed to replace certain marked critical parts in HTML code with encrypted JavaScript code. The generated code is portable and accessible. It can be used to keep email and other addresses from being harvested, and keep certain info

This project enables you to easily build a custom HTML whitelist filter that you can use to sanitize user-submitted HTML so that it can be safely displayed in various browsers.

Python scripts that filter, parse, mask, and analyze FileZilla Server Log files. Works on Windows, Linux, and Mac systems. WARNING: NEW VERSIONS ARE BEING POSTED TO GITHUB HERE: https://github.com/Stunner/FileZilla-Log-Analyzer

a small collection of python 3000 scripts/modules used to automate searching craigslist.org cities and categories for interesting stuff; these scripts currently use html screen scraping, since craigslist currently has no api

Postfix Log Parser in Python is a log analysis tool written in python language to get statistic reports and detect anomalous behaviours on a mail server based on Postfix and Cyrus

...A platform for the integration and mining of complex and distributed data. Represents cross-linked semantic web documents as a network of software objects and offers easy ways to filter, and sort them.

Auto Proxy Filter Test (APFT) automates the testing of safe and unsafe URLs against a content filtering proxy (such as Dansguardian) and helps prevent regressions. APFT is useful to people who are designing filter rules.

Upcow is a Python program that passes text input through a 'cowsay'-like filter, and may automatically upload the result to a given FTP location.

PyEsp - Enhanced/Evolving/Extensible Semantic Profiling. This Python program will sort and filter search results by applying semantic profiling on web pages. The program will learn the user preferences and profiling will be done on the client computer.