Showing 54 open source projects for "processing"

View related business solutions
  • Paessler: Easy to Use With Enterprise Power. Free Trial Icon
    Paessler: Easy to Use With Enterprise Power. Free Trial

    A low-code dashboard makes monitoring intuitive for any admin, while scripting and custom sensors give experts full control.

    You shouldn't have to choose between a monitoring tool that's easy to use and one that's powerful enough for a complex environment. PRTG's low-code interface lets any admin build dashboards, set alerts and monitor devices without scripting, while custom sensors and full API access are there when your team needs deeper control. One platform, no compromise. Download a free 30-day trial now.
    Get Free Download
  • Already Paying for an RMM? Get 50% Off Splashtop AEM Icon
    Already Paying for an RMM? Get 50% Off Splashtop AEM

    Switch to Splashtop AEM and save up to 50% in your first year

    Automated OS and third-party patching for Windows and macOS, CVE prioritization with AI patch summaries, and built-in remote control that also supports Linux. Qualifying organizations switching from another RMM get up to 50% off year one.
    Get Started
  • 1
    Python API for JMComic

    Python API for JMComic

    Python crawler and API for downloading JMComic albums and images

    ...It includes command-line functionality and configuration files so users can customize download behavior, directory structures, and performance settings without modifying code. It also supports plugin-based extensions that allow additional processing.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 2
    Parser-Chitai-Gorod

    Parser-Chitai-Gorod

    Intelligent parser for catalog «Read-city»

    ...Its workflow first discovers individual book pages across catalog sections and pagination. A second stage processes those pages and extracts fields such as title, author, ISBN, price, binding, description, page count, publication year, and image URL. Processing can use multiple threads to improve throughput across larger catalogs. Completed URLs are tracked so interrupted jobs can resume without unnecessarily processing the same books again. Results are exported to Excel workbooks, with separate sheets organized by book genre for easier analysis.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 3
    Trafilatura

    Trafilatura

    Python & command-line tool to gather text on the Web

    Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by avoiding the noise caused by recurring elements (headers, footers, links/blogroll etc.) and second by including information such as author and date in order to make sense of the data. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 4
    newspaper4k

    newspaper4k

    Python library for scraping and analyzing online news articles easily

    ...Newspaper4k also includes natural language processing capabilities that can generate summaries and identify keywords from extracted article text. Newspaper4k supports both single-article extraction and full news site processing, allowing users to build sources representing entire publications and iterate through their articles. It maintains compatibility with the original project so that existing code written for newspaper3k can continue working with minimal changes.
    Downloads: 0 This Week
    Last Update:
    See Project
  • Train ML Models With SQL You Already Know Icon
    Train ML Models With SQL You Already Know

    BigQuery automates data prep, analysis, and predictions with built-in AI assistance.

    Build and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
    Start Free
  • 5
    MDCx

    MDCx

    Movie metadata scraper and organizer for media libraries and NFO

    ...MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. It also supports image processing tasks such as downloading and cropping artwork used by media centers. It includes several interfaces, allowing users to operate it through a graphical desktop application, a browser-based web interface, or command-line utilities depending on their workflow. Its architecture separates core scraping logic from the user interfaces, allowing the same metadata processing system to be reused across different modes.
    Downloads: 3 This Week
    Last Update:
    See Project
  • 6
    mitmproxy2swagger

    mitmproxy2swagger

    Automagically reverse-engineer REST APIs via capturing traffic

    mitmproxy2swagger is a Python tool that converts captured HTTP traffic into OpenAPI 3.0 specifications. It can analyze mitmproxy flow files or HAR exports from browser developer tools. The first processing pass discovers candidate endpoints and path structures from observed requests. Users can review those paths before a second pass generates parameters, request bodies, responses, and endpoint descriptions. Existing schemas can be extended across multiple captures instead of being recreated from scratch. Optional flags can include examples and headers, although captured sensitive data must be handled carefully. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 7
    gain

    gain

    Asyncio-based Python framework for building fast web crawling spiders

    Gain is a Python web crawling framework designed to simplify the process of building efficient and scalable web scrapers. It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and...
    Downloads: 11 This Week
    Last Update:
    See Project
  • 8
    Snoop Project

    Snoop Project

    This is the most powerful software taking into account CIS location

    ...Snoop project is developed without taking into account the opinions of the NSA and their friends, that is, it is available to the average user. Snoop is a research work (own database / closed bugbounty) in the field of searching and processing public data on the Internet. In terms of specialized search, Snoop is able to compete with traditional search engines.
    Downloads: 5 This Week
    Last Update:
    See Project
  • 9
    Any Auto Register

    Any Auto Register

    Auto-register & manage accounts for ChatGPT, Cursor, Kiro, Grok

    ...Users can configure email providers, verification services, and proxy pools for automated workflows. Its dashboard monitors registration success rates, account validity, token expiration, and trial status. The application also supports batch processing, account exporting, and integration with the Any2API gateway. Available as a Windows and macOS desktop application or through Docker, it uses a plugin-based architecture for extending platform compatibility.
    Downloads: 1 This Week
    Last Update:
    See Project
  • Our Free Plans just got better! | Auth0 Icon
    Our Free Plans just got better! | Auth0

    With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

    You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
    Try free now
  • 10
    diskover-community

    diskover-community

    Open source file indexing & storage analytics powered by Elasticsearch

    Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across...
    Downloads: 3 This Week
    Last Update:
    See Project
  • 11
    watercrawl

    watercrawl

    AI-ready web crawler that extracts and structures website content

    WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website....
    Downloads: 3 This Week
    Last Update:
    See Project
  • 12
    AIOHTTP

    AIOHTTP

    Asynchronous HTTP client/server framework for asyncio and Python

    ...A long awaited new feature is tracing client request life cycle to figure out when and why client request spends a time waiting for connection establishment, getting server response headers etc. Now it is possible by registering special signal handlers on every request processing stage. The main change is dropping yield from support and using async/await everywhere. Farewell, Python 3.4. You often want to send some sort of data in the URL’s query string. If you were constructing the URL by hand, this data would be given as key/value pairs in the URL after a question mark, e.g. httpbin.org/get?key=val. Requests allows you to provide these arguments as a dict, using the params keyword argument. aiohttp internally performs URL canonicalization before sending request.
    5.0 out of 5 stars 1 Review
    Downloads: 3 This Week
    Last Update:
    See Project
  • 13
    Grab Framework Project

    Grab Framework Project

    Web Scraping Framework

    Grab is a python framework for building web scrapers. With Grab you can build web scrapers of various complexity, from simple 5-line scripts to complex asynchronous website crawlers processing millions of web pages. Grab provides an API for performing network requests and for handling the received content e.g. interacting with DOM tree of the HTML document. The single request/response API that allows you to build network request, perform it and work with the received content. The API is built on top of urllib3 and lxml libraries. ...
    Downloads: 1 This Week
    Last Update:
    See Project
  • 14
    douyin

    douyin

    Open source Douyin crawler for collecting and downloading public data

    DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. DouyinCrawler supports both automated scraping and batch operations to process multiple targets efficiently. It...
    Downloads: 2 This Week
    Last Update:
    See Project
  • 15
    furl

    furl

    The easiest way to parse and modify URLs in Python

    ...It also handles encoding automatically, including percent-encoding, Unicode domains, Unicode paths, and query strings. furl supports Python 3 and PyPy3 and is designed to be well tested and practical for everyday backend, scripting, and data-processing workflows. It is a good fit for developers who frequently need to clean, transform, generate, or normalize URLs in Python applications.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 16
    news-please

    news-please

    Python tool for crawling and extracting structured data from news site

    ...It provides an integrated pipeline that crawls news sites, retrieves article pages, and extracts structured information such as headlines, authors, publication dates, and article text. news-please can recursively follow internal links and read RSS feeds to gather both recent and archived articles from a news outlet when given only the root URL of a site. It combines several established technologies and libraries to perform web crawling and content extraction, enabling reliable processing across a wide range of news sources. Developers can use the software either as a standalone command line application or integrate it into their own Python applications through its library interface. Extracted article data can be stored in different formats and systems, including JSON files or database-backed storage solutions.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 17
    Tweepy

    Tweepy

    Twitter for Python

    An easy-to-use Python library for accessing the Twitter API. You can also use Git to clone the repository from GitHub to install the latest development version. The easiest way to install the latest version from PyPI is by using pip. Twitter requires all requests to use OAuth for authentication. The API class provides access to the entire twitter RESTful API methods. Each method can accept various parameters and return responses. When we invoke an API method most of the time returned back to...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 18

    apache-logs-to-mysql

    Apache Log Parser and Data Normalization Application

    Apache Log Parser and Data Normalization Application Python handles File Processing & MySQL handles Data Processing ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis. Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0. 4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution. ...
    Downloads: 5 This Week
    Last Update:
    See Project
  • 19
    OmniPull

    OmniPull

    Just pull anything

    OmniPull is a powerful, cross-platform download manager built with Python and PySide6. It provides a modern, intuitive interface for managing downloads with advanced features like multi-threading, queue management, and media extraction.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 20
    DNS Crawler

    DNS Crawler

    A Bulk Domain Assessment Tool

    DNS Crawler is a lightweight, Python-based utility designed for efficient batch processing and assessment of internet domain names. It reads from a list of domains formatted as: domain_name <tab> or ; optional_comment and generates a detailed, Excel-compatible CSV report with columns including: DOMAIN: Domain name REG: Registrar SOA, NS, MX, TXT, SPF, DMARC, MS, A, PTR: Common DNS records for comprehensive domain analysis NOTE: Optional comments from the original input file Whether you're managing multiple domains or auditing for cross-platform DNS consistency, DNS Crawler simplifies the process, offering a clear, structured output for easy review and reporting.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 21
    Scrapy-Redis

    Scrapy-Redis

    Redis-based components for Scrapy

    You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3 changed the requests serialization from marshal to cPickle, therefore persisted requests using version 0.2 will not able to work on 0.3. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 22
    spider_collection

    spider_collection

    Collection of Python web scraping scripts for data extraction tasks

    ...Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. It also contains examples of proxy pool integration and encapsulation to support more reliable crawling when working with sites that enforce request limits.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 23
    mlscraper

    mlscraper

    ML-based HTML scraper that learns extraction rules from examples

    mlscraper is a Python library designed to automatically extract structured data from HTML pages without requiring developers to manually write CSS selectors or XPath rules. Instead of defining extraction logic by hand, users provide a few examples of the data they want to retrieve from a webpage. It analyzes those examples within the HTML document and determines patterns or rules that can be used to extract the same type of information from similar pages. Once trained, the generated scraper...
    Downloads: 9 This Week
    Last Update:
    See Project
  • 24
    pspider

    pspider

    Simple Python framework for building multithreaded web crawlers

    PSpider is a lightweight web crawling framework written in Python designed to simplify the development of custom web spiders. It focuses on providing an easy-to-understand architecture while still supporting concurrent crawling for improved performance. It uses a multithreaded model that separates the crawling workflow into several components responsible for fetching, parsing, and saving data. Tasks are managed through queues, allowing different parts of the crawler to process work...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 25
    ruia

    ruia

    Async Python framework for fast and flexible web scraping spiders

    ...Developers can define structured fields to extract information from HTML content and process responses asynchronously to improve crawling performance. It also supports middleware and plugin systems that allow customization of request handling, response processing, and additional functionality.
    Downloads: 0 This Week
    Last Update:
    See Project
  • Previous
  • You're on page 1
  • 2
  • 3
  • Next