Search Results for "parallel"

Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations.

...Designed as an alternative to tools like Ansible, pyinfra prioritizes speed, scalability, and developer flexibility while maintaining a declarative operational model. It supports ad-hoc command execution, reusable operations, inventory management, and parallel deployments across thousands of hosts. The framework integrates naturally with existing DevOps ecosystems and allows users to create highly customizable deployment logic using native Python syntax. Its architecture combines infrastructure-as-code concepts with efficient remote execution, making it suitable for modern cloud and server automation workflows.

Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by...

OmniPull is a powerful, cross-platform download manager built with Python and PySide6. It provides a modern, intuitive interface for managing downloads with advanced features like multi-threading, queue management, and media extraction.

...Scrapyd is an application (typically run as a daemon) that listens to requests for spiders to run and spawns a process for each one. Scrapyd also runs multiple processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load.

Manage node sets, node groups and execute commands on cluster nodes in parallel. Provides an event-based Python library to improve administration of large compute clusters or server farms. Command line tools: clush and nodeset included.

SOURCE CODE MOVED TO https://github.com/kowalpy/Robot-Framework-FTP-Library . NEW RELEASES WILL APPEAR ONLY AT GITHUB AND PYPI. This Python library makes it possible to test or use FTP server using Robot Framework keywords. Together with the library, there are also available project example and keywords documentation created by libdoc. This sourceforge webpage is linked from http://robotframework.org/#test-libraries as external library so it can be considered as the official Robot...

Are you maitaining many machines and bored with the tedious operation of logging to those machines one by one just for checking the machine's status? Or are you losing patient for your simple script that have to run command in those machines in sequence? The corunner is just for those senarios. It's designed to speed up execution in your machines. It can executes command in handreds of or thousands of machines concurrently. Provide it with your files and command and the machines' location,...