Search Results for "data collection algorithm"
Sort By:
Showing 108 open source projects for "data collection algorithm"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
1
skycaiji
Open source web scraping system for automated data collection tasks
SkyCaiji is an open source web scraping and data collection system designed to gather information from websites through configurable extraction rules. It focuses on simplifying the process of building crawlers by allowing users to visually define scraping rules rather than writing complex code. It can collect structured or unstructured data from many types of webpages and automate the extraction process for large datasets. -
2
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
3
Immutable.js
Immutable collections for JavaScript
Immutable.js offers a collection of Persistent Immutable data structures for JavaScript. Immutable data is unchangeable once created, which makes application development so much simpler. There’s no defensive copying, and you get advanced memoization and change detection techniques with simple logic. Persistent data gives you a mutative API, one that doesn’t update data in-place but always produces new and updated data. -
4
POI POI POI
Scalable KanColle browser and tool
poi is a scalable browser and tool set for Kantai Collection(KanColle). Key features include proxy,HTTP, Socks5 and PAC (Experimental). Cache, including custom cache. Data synthesis and analysis. Notification and plugin support for extensive functionalities. -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
Helium Browser
Private, fast, and honest web browser
...Helium blocks ads and trackers by default through an integrated, unbiased uBlock Origin extension prepackaged as a native browser component. Its UI and feature set emphasize minimalism, no “smart” recommendations, account sync, or background data collection, resulting in a distraction-free browsing experience that respects user autonomy. The browser is available across macOS, Linux, and Windows, each version built from a fully open source pipeline for reproducibility and trust. Development focuses on maintaining compatibility with modern web standards while decoupling Chromium from its Google dependencies and services. -
6
Dataproc Templates
Dataproc templates and pipelines for solving simple in-cloud data task
Dataproc templates are designed to address various in-cloud data tasks, including data import/export/backup/restore and bulk API operations. These templates leverage the power of Google Cloud's Dataproc, supporting both Dataproc Serverless and Dataproc clusters. Google provides this collection of pre-implemented Dataproc templates as a reference and for easy customization. -
7
FEAPDER
Powerful Python crawler framework for scalable web scraping tasks
feapder is a Python-based web crawling framework designed to simplify the process of building scalable and efficient web scrapers. It focuses on providing a developer-friendly environment that makes it easier to create, run, and manage crawlers for a variety of data collection tasks. It includes several built-in spider types, such as AirSpider, Spider, TaskSpider, and BatchSpider, which address different crawling scenarios ranging from lightweight scraping to distributed and batch-based jobs. feapder supports features such as breakpoint resume, allowing crawlers to continue from where they stopped without losing progress. ... -
8
Weibo Crawler
Python crawler for collecting and downloading Sina Weibo user data
weibo-crawler is a Python-based data collection tool designed to retrieve information from Sina Weibo user accounts. It automates the process of gathering posts, user profile details, and engagement metrics from one or more target accounts. weibo-crawler can extract comprehensive information about users, including profile attributes such as nickname, follower count, following count, and account metadata. -
9
crawler
Collection of JS reverse engineering examples for web scraping study
crawler is a collection of web scraping and JavaScript reverse engineering examples designed for learning how modern websites protect their data and how those protections can be analyzed. It contains many case studies that demonstrate how to analyze and replicate request parameters, cookies, and encryption logic used by real websites. Each directory in the project focuses on a specific target service or scenario, showing how browser network requests and JavaScript code can be studied to reproduce API calls programmatically. ... -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
10
Scweet
Scrape tweets, profiles, followers and following from Twitter/X
Scweet is a Python-based Twitter/X scraping library and CLI designed to collect tweets, profile timelines, followers, following lists, and user profile data without requiring the official Twitter/X API or a developer account. Instead of depending on deprecated unauthenticated scraping methods, it works by using X’s web GraphQL API together with authenticated browser cookies, which gives it a more current and practical approach for data extraction. The project supports a broad set of collection patterns, including searches by keyword, hashtag, user, date range, engagement thresholds, language, and location, making it useful for research, monitoring, and data gathering workflows. ... -
11
OpenSearch
Open source distributed and RESTful search engine
...It offers excellent performance and can scale up and down as the needs of the application grow or shrink. Its distributed design means that you interact with OpenSearch clusters. Each cluster is a collection of one or more nodes, servers that store your data and process search requests. You can run OpenSearch locally on a laptop, its system requirements are minimal, but you can also scale a single cluster to hundreds of powerful machines in a data center. -
12
syslog-ng
Log management solution that improves the performance of SIEM
...Instead of deploying multiple agents on hosts, organizations can unify their log data collection and management. syslog-ng Store Box provides automated archiving, tamper-proof encrypted storage, granular access controls to protect log data. The largest appliance can store up to 10TB of raw logs. -
13
Geziyor
Blazing fast Go framework for web crawling and data scraping tasks
...With built-in tools for caching, metrics collection, and proxy management, it enables developers to build robust and customizable scraping systems using Go. -
14
OpenSEO
Open source alternative to Semrush and Ahrefs
...It provides focused workflows for keyword research, rank tracking, competitor analysis, backlink research, site auditing, and AI visibility monitoring. A modern web interface keeps these tasks accessible without hiding them inside a large collection of unrelated reports. The platform exposes an MCP server so AI agents can work directly with SEO data and reusable agent skills. Users can run a hosted edition or self-host the application with Docker or Cloudflare. Self-hosted installations connect to DataForSEO with the user's own API key and use pay-as-you-go data rather than a mandatory software subscription. ... -
15
SwifterSwift
A collection of more than 500 native Swift extensions
SwifterSwift is a collection of over 500 native Swift extensions, with handy methods, syntactic sugar, and performance improvements for a wide range of primitive data types, UIKit and Cocoa classes, over 500 in 1– for iOS, macOS, tvOS, watchOS, and Linux. CocoaPods is a dependency manager for Cocoa projects. To integrate SwifterSwift into your Xcode project using CocoaPods, specify it in your Podfile. -
16
Docker-ELK
The Elastic stack (ELK) powered by Docker and Compose
A turnkey Docker Compose stack to spin up the ELK stack (Elasticsearch, Logstash, Kibana) for log collection, analysis, and visualization. Based on official Elastic images and enhanced with configuration defaults optimized for local development and testing. -
17
Aeron
Efficient UDP unicast, UDP multicast, and IPC message transport
...Message streams can be recorded by the Archive module to persistent storage for later, or real-time, replay. Aeron Cluster provides support for fault-tolerant services as replicated state machines based on the Raft consensus algorithm. Performance is the key focus. A design goal for Aeron is to be the highest throughput with the lowest and most predictable latency of any messaging system. Aeron integrates with Simple Binary Encoding (SBE) for the best possible message encoding and decoding performance. Many of the data structures used in the creation of Aeron have been factored out to the Agrona project. -
18
watercrawl
AI-ready web crawler that extracts and structures website content
...WaterCrawl supports customizable extraction rules so users can focus only on relevant elements while ignoring unnecessary page components. WaterCrawl also offers real-time monitoring capabilities, allowing users to track crawling progress, performance metrics, and errors during large data collection jobs. Developers can integrate the tool into applications through a REST API and multiple client SDKs, enabling automated data pipelines and AI data preparation workflows. -
19
Rybbit
Open-source and privacy-friendly alternative to Google Analytics
Rybbit is an open-source, privacy-friendly web and product analytics platform positioned as a modern alternative to heavyweight tracking suites. It focuses on a clean UI and intuitive metrics so non-analysts can answer everyday questions without wading through dozens of reports. The data model is event-driven, enabling funnels, retention views, and simple user journeys without complex configuration. Because privacy is a first-class goal, it aims to minimize personal data collection and offers a self-host path so you can keep analytics in your own infrastructure. The project ships with a lightweight snippet and a Docker-based deployment, making it quick to trial or roll out. ... -
20
Waterfox
Privacy browser built for users who value customization & control
...It began as a 64-bit-focused fork and has grown into a full-featured browser available on Windows, macOS, Linux, and Android, making it a versatile alternative to mainstream browsers. Waterfox removes or disables much of Firefox’s telemetry and unnecessary data collection by default, giving users greater control over what is shared with third parties while still supporting classic and modern add-ons. The project emphasizes speed and efficiency through optimizations and choice, letting users tweak settings or install favored extensions without being locked into ecosystem constraints. Because it’s open source and actively developed by the community, new features, security updates, and performance improvements arrive regularly to align with web standards. -
21
KCP
A fast and reliable ARQ protocol
KCP is a fast and reliable protocol that can achieve the transmission effect of a reduction of the average latency by 30% to 40% and reduction of the maximum delay by a factor of three, at the cost of 10% to 20% more bandwidth wasted than TCP. It is implemented by using the pure algorithm, and is not responsible for the sending and receiving of the underlying protocol (such as UDP), requiring the users to define their own transmission mode for the underlying data packet, and provide it to KCP in the way of callback. Even the clock needs to be passed in from the outside, without any internal system calls. The entire protocol has only two source files of ikcp.h, ikcp.c, which can be easily integrated into the user's own protocol stack. ... -
22
OpenWPM
A web privacy measurement framework
OpenWPM is a web privacy measurement framework that makes it easy to collect data for privacy studies on a scale of thousands to millions of websites. OpenWPM is built on top of Firefox, with automation provided by Selenium. It includes several hooks for data collection. Check out the instrumentation section below for more details. OpenWPM is tested on Ubuntu 18.04 via TravisCI and is commonly used via the docker container that this repo builds, which is also based on Ubuntu. ... -
23
Browserless
Deploy headless browsers in Docker
...The project also provides REST APIs for common automation tasks such as screenshots, PDF generation, scraping, crawling, and content export. Browserless is useful for teams that need scalable browser execution for testing, data collection, rendering, or AI-agent browsing workflows. Its deployment model supports self-hosting, private infrastructure, queues, concurrency controls, and enterprise-oriented configuration. The project’s main value is turning browser automation into a managed service layer that can be reused across applications and workflows. -
24
Jimp
An image processing library written entirely in JavaScript for Node
An image processing library for Node written entirely in JavaScript, with zero native dependencies. If you're using this library with TypeScript the method of importing slightly differs from JavaScript. Instead of using require, you must import it with ES6 default import scheme. If you're using a web bundles (webpack, rollup, parcel) you can benefit from using the module build of jimp. Using the module build will allow your bundler to understand your code better and exclude things you aren't... -
25
DotnetSpider
Lightweight .NET framework for fast web crawling and data scraping
...DotnetSpider is modular, allowing different components such as request schedulers, downloaders, and storage systems to work together in a flexible workflow. DotnetSpider also supports distributed crawling environments, making it possible to scale data collection across multiple agents and machines. With support for various storage backends and extensible parsing mechanisms, it is suitable for building complex scraping systems or automated data gathering pipelines.