Search Results for "process open"
Sort By:
Showing 51 open source projects for "process open"
View related business solutions-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
1
proxy.py
Utilize all available CPU cores for accepting new client connections
proxy.py is made with performance in mind. By default, proxy.py will try to utilize all available CPU cores to it for accepting new client connections. This is achieved by starting AcceptorPool which listens on configured server port. Then, AcceptorPool starts Acceptor processes (--num-acceptors) to accept incoming client connections. Alongside, if --threadless is enabled, ThreadlessPool is setup which starts Threadless processes (--num-workers) to handle the incoming client connections.... -
2
MDCx
Movie metadata scraper and organizer for media libraries and NFO
MDCx is an open source media metadata scraping and organization tool designed to automate the process of collecting detailed information for movie files. It retrieves metadata from multiple online sources and applies it to local media collections, helping users maintain structured and well-organized libraries. MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. ... -
3
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
4
gain
Asyncio-based Python framework for building fast web crawling spiders
Gain is a Python web crawling framework designed to simplify the process of building efficient and scalable web scrapers. It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and... -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
5
hosts
Consolidate and extend hosts files from several well-curated sources
Consolidating and extending hosts files from several well-curated sources. You can optionally pick extensions to block pornography, social media, and other categories. The unified hosts file is optionally extensible. Extensions are used to include domains by category. Currently, we offer the following categories: fakenews, social, gambling, and porn. Extensions are optional, and can be combined in various ways with the base hosts file. The combined products are stored in the alternates... -
6
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website. ... -
7
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
8
Factorio
Factorio headless server in a Docker container
Factorio is an open-source project that provides a Docker container for running a headless Factorio game server. Factorio is a factory-building simulation game in which players automate production lines, research technologies, and manage complex industrial systems, and the repository focuses specifically on hosting the game server in a containerized environment. -
9
Scrapy-Redis
Redis-based components for Scrapy
You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version... -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
news-please
Python tool for crawling and extracting structured data from news site
news-please is an open source news crawler and information extraction tool designed to collect and structure articles from online news websites. It provides an integrated pipeline that crawls news sites, retrieves article pages, and extracts structured information such as headlines, authors, publication dates, and article text. news-please can recursively follow internal links and read RSS feeds to gather both recent and archived articles from a news outlet when given only the root URL of a... -
11
FEAPDER
Powerful Python crawler framework for scalable web scraping tasks
feapder is a Python-based web crawling framework designed to simplify the process of building scalable and efficient web scrapers. It focuses on providing a developer-friendly environment that makes it easier to create, run, and manage crawlers for a variety of data collection tasks. It includes several built-in spider types, such as AirSpider, Spider, TaskSpider, and BatchSpider, which address different crawling scenarios ranging from lightweight scraping to distributed and batch-based... -
12
Toapi
Convert websites into structured APIs automatically with Python tool
Toapi is a Python library designed to transform ordinary websites into usable API services. Instead of building a traditional web crawler that collects and stores data before exposing it through an API, Toapi simplifies the process by allowing developers to define data structures that automatically generate an API layer from existing web pages. It works by parsing HTML content from a source site and mapping selected elements into structured data that can be returned as JSON through API... -
13
tumblr-crawler
Python crawler to download photos and videos from Tumblr blogs
tumblr-crawler is an open source Python-based utility designed to download media content from Tumblr blogs. It provides a script that automatically retrieves photos and videos from specified Tumblr sites and saves them locally for offline access. Users can specify one or multiple blogs to crawl by editing a configuration file or by passing parameters through the command line. Once executed, the script fetches media from the Tumblr API and stores the downloaded files in folders named after... -
14
YehDown
Yeahdown: Easy-to-use video downloader for Windows
Yeahdown is a straightforward, user-friendly Windows-based application designed to simplify the process of downloading videos and audio from popular websites like YouTube and Vimeo. Perfect for non-technical users, it offers an intuitive interface and fast, reliable downloads. Key features include improved download speeds, support for multiple major video platforms, and real-time updates for new features. Tested on windows 11. -
15
DNS Crawler
A Bulk Domain Assessment Tool
DNS Crawler is a lightweight, Python-based utility designed for efficient batch processing and assessment of internet domain names. It reads from a list of domains formatted as: domain_name <tab> or ; optional_comment and generates a detailed, Excel-compatible CSV report with columns including: DOMAIN: Domain name REG: Registrar SOA, NS, MX, TXT, SPF, DMARC, MS, A, PTR: Common DNS records for comprehensive domain analysis NOTE: Optional comments from the original input... -
16
VirtualHostAdd
Command-line tool for creating Apache virtual hosts on Debian 12.
VirtualHostAdd is a command-line tool for Debian 12—and derived Linux distributions—designed to simplify the creation and configuration of virtual hosts on the Apache HTTP Server, for PHP and CGI-based applications. It automates the entire process, handling directory structure, Apache configuration, MPM and Apache modules handling, and file/directory permissions, reducing manual setup time and errors for developers and sysadmins. -
17
apache-logs-to-mysql
Apache Log Parser and Data Normalization Application
Apache Log Parser and Data Normalization Application Python handles File Processing & MySQL handles Data Processing ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis. Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0. 4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed... -
18
RPA for Python
Python package for doing RPA
Python package for doing RPA. RPA for Python's simple and powerful API makes robotic process automation fun! You can use it to quickly automate away repetitive time-consuming tasks on websites, desktop applications, or the command line. See sample Python script, the RPA Challenge solution, and RedMart groceries example. To send a Telegram app notification, simply look up @rpapybot to allow receiving messages. To automate Chrome browser invisibly, use headless mode. -
19
Scrapyd
A service daemon to run Scrapy spiders
Scrapyd can manage multiple projects and each project can have multiple versions uploaded, but only the latest one will be used for launching new spiders. A common (and useful) convention to use for the version name is the revision number of the version control tool you’re using to track your Scrapy project code. For example: r23. The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd is an... -
20
AWS GameKit Plugin for Unreal
The AWS GameKit Plugin for Unreal
AWS GameKit is an open-source engine integration toolkit for game developers who want to create high-quality, cloud-connected game features without the potentially steep learning curve for cloud service architecture and AWS. With AWS GameKit, you can deliver core functionality to your players with minimal effort and time by using pre-built AWS solutions for your game backend and integrating the features into your game. You can use the AWS GameKit game features as designed or as a launch pad... -
21
grab-site
Web crawler for archiving and backing up sites into WARC archives
grab-site is an open source web crawling tool designed to archive and back up websites by recursively downloading their content. It works by taking a starting URL and systematically following links across the site, capturing pages and resources and saving them into WARC archive files for long-term preservation. Internally, the crawler uses a fork of the wpull engine to fetch and process web pages efficiently during large-scale crawls. grab-site includes a built-in dashboard that displays real-time crawl activity, including which URLs are currently being processed and how many remain in the queue. ... -
22
pspider
Simple Python framework for building multithreaded web crawlers
PSpider is a lightweight web crawling framework written in Python designed to simplify the development of custom web spiders. It focuses on providing an easy-to-understand architecture while still supporting concurrent crawling for improved performance. It uses a multithreaded model that separates the crawling workflow into several components responsible for fetching, parsing, and saving data. Tasks are managed through queues, allowing different parts of the crawler to process work... -
23
ruia
Async Python framework for fast and flexible web scraping spiders
Ruia is an asynchronous web scraping micro-framework built for Python that focuses on simplicity, speed, and flexibility when creating web crawlers. Ruia is powered by Python’s asyncio library along with aiohttp, enabling developers to perform concurrent network requests efficiently and scrape data from websites with minimal overhead. Ruia follows a “write less, run faster” philosophy, emphasizing concise code and streamlined spider development. It provides a structured approach to building... -
24
GoogleScraper
Python tool for scraping search engine results from many providers
GoogleScraper is a Python-based tool designed to automatically collect and process search engine results from multiple providers. It enables developers and researchers to programmatically query search engines and extract useful information such as links, titles, and result descriptions. GoogleScraper supports several major search engines and can be used to gather structured datasets from search result pages for further analysis. It provides two different scraping approaches: sending direct... -
25
ECommerceCrawlers
Collection of Python ecommerce and website crawler examples projects
ECommerceCrawlers is a collection of practical Python web crawler projects designed to gather data from a variety of ecommerce platforms, websites, and online services. It aggregates many independent crawler examples created by contributors and organized into separate subprojects that target specific sites or data sources. These examples demonstrate how to build and operate web scrapers capable of collecting structured information such as product listings, news content, job postings, social...
