Search Results for "data collection algorithm" - Page 2
Sort By:
Showing 126 open source projects for "data collection algorithm"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
1
OpenWPM
A web privacy measurement framework
OpenWPM is a web privacy measurement framework that makes it easy to collect data for privacy studies on a scale of thousands to millions of websites. OpenWPM is built on top of Firefox, with automation provided by Selenium. It includes several hooks for data collection. Check out the instrumentation section below for more details. OpenWPM is tested on Ubuntu 18.04 via TravisCI and is commonly used via the docker container that this repo builds, which is also based on Ubuntu. ... -
2
Jimp
An image processing library written entirely in JavaScript for Node
An image processing library for Node written entirely in JavaScript, with zero native dependencies. If you're using this library with TypeScript the method of importing slightly differs from JavaScript. Instead of using require, you must import it with ES6 default import scheme. If you're using a web bundles (webpack, rollup, parcel) you can benefit from using the module build of jimp. Using the module build will allow your bundler to understand your code better and exclude things you aren't... -
3
Browserless
Deploy headless browsers in Docker
...The project also provides REST APIs for common automation tasks such as screenshots, PDF generation, scraping, crawling, and content export. Browserless is useful for teams that need scalable browser execution for testing, data collection, rendering, or AI-agent browsing workflows. Its deployment model supports self-hosting, private infrastructure, queues, concurrency controls, and enterprise-oriented configuration. The project’s main value is turning browser automation into a managed service layer that can be reused across applications and workflows. -
4
DotnetSpider
Lightweight .NET framework for fast web crawling and data scraping
...DotnetSpider is modular, allowing different components such as request schedulers, downloaders, and storage systems to work together in a flexible workflow. DotnetSpider also supports distributed crawling environments, making it possible to scale data collection across multiple agents and machines. With support for various storage backends and extensible parsing mechanisms, it is suitable for building complex scraping systems or automated data gathering pipelines. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
5
spider_collection
Collection of Python web scraping scripts for data extraction tasks
...Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. It also contains examples of proxy pool integration and encapsulation to support more reliable crawling when working with sites that enforce request limits. -
6
ArchiSteamFarm
A multi-account able app for farming cards in Steam
ASF is a robust C# application designed to farm Steam cards from multiple accounts simultaneously. Unlike other tools like Idle Master, which manage only one account at a time and require the Steam client to be running, ASF operates without the Steam client or additional processes. This allows users to farm Steam cards from unlimited accounts efficiently. ASF is optimized for use on servers or headless machines and supports all major operating systems, including Windows, Linux, and macOS,... -
7
DavUtils
Powerful webdav client and sync tools with client-side AES encryption
...It is very flexible and configurable and has advanced data safety features to prevent data loss. The integrated client-side AES encryption makes it secure to use public clouds. -
8
Hirtius
A Historical Research Assistant
Hirtius is a web-based historical research assistant created by Les Ateliers du Héron. Its goal is to replace the historian's notebook and box of cards, while providing a collaborative work environment for a whole team of researchers, that can be thought of as a purpose-built CMS cross-bred with a digital pictures management system, a genealogy application and a Wiki. Eventually, based on the information you stored in the application, and the links you created between the various data... -
9
fojecleno_downloader
Free video downloader, simple, fast, bilingual and ad-free
fojecleno_downloader is a simple, fast, and completely free Windows application for downloading royalty-free videos from numerous platforms. No ads, no data collection, no complications—just a clear, efficient, and accessible tool for everyone. Main features: - Responsible one-click video downloading - Bilingual interface (French/English) - Automatic pasting from the clipboard - Choice of download folder - Activity log - Light/dark mode - Zoom and smooth scrolling - Collapsible sections - Portable version (no installation required) - Installable version with shortcuts and clean uninstallation - Compatible with Windows 10 and 11 -
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
10
Ballast is a tool for balancing user load across SSH servers that supports user-specific policies as well as traditional policies based on metrics such as CPU load. It includes a simple client, a lightweight data server, and a data collection agent.
-
11
python-fxxk-spider
Collection of 100+ Python web scraping projects and crawler examples
python-fxxk-spider is a curated collection of Python web scraping and crawler projects gathered in a single repository for reference and learning. It aggregates many independent scraping examples that target a wide range of websites, online services, and public data sources. Instead of being a single crawler tool, it functions as a catalog of ready-made Python spider implementations that demonstrate different scraping techniques. python-fxxk-spider includes scrapers for social media, e-commerce platforms, job listings, music services, video platforms, and various content sites. ... -
12
LibreWolf
Linux-specific build and associated scripts for LibreWolf
LibreWolf’s browser-linux repository hosts the Linux-specific build and associated scripts for LibreWolf, a free and open-source web browser that’s a community-driven fork of Firefox with a strong emphasis on privacy, security, and user freedom. Unlike mainstream browsers that collect telemetry or nudge users toward proprietary features, LibreWolf removes telemetry and many “data collection” mechanisms by default and ships with hardened privacy configurations geared toward minimizing tracking and unwanted online profiling. It also disables certain features like automatic updates or cloud synchronization out of the box, putting control back in the hands of the user and making it suitable for privacy-conscious individuals and organizations. ... -
13
Front-end development blog
Front-end Development Thoughts
Front-end development blog is a front-end development article repository focused on JavaScript, ES6, React, Redux, and related web engineering topics. It works as a curated index of essays rather than a traditional software package. The repository collects technical thinking on subjects such as AI in development, VS Code history, React Conf, JavaScript number behavior, server-side versus client-side rendering, and React best practices. It also includes articles about CSS Modules, Immutable... -
14
crawlergo
Headless Chrome crawler for collecting URLs for vulnerability scans
crawlergo is a browser-based web crawler designed to collect URLs and request data that can be used by web vulnerability scanning tools. It uses a Chrome headless environment to render web pages and observe behavior during the DOM rendering stage in order to capture as many accessible endpoints as possible. By monitoring the page lifecycle and interacting with web elements, the crawler automatically triggers JavaScript events and navigational actions that would normally occur during real... -
15
DecryptLogin
Python library providing APIs for automated website login workflows
...DecryptLogin supports a wide variety of online services and platforms, including social media sites, developer platforms, cloud services, and other web portals. Developers can integrate these login routines into automation scripts, crawlers, or data collection tools that require authenticated sessions. It also provides example utilities and automation scripts demonstrating how the login APIs can be used in practical scenarios. -
16
mlscraper
ML-based HTML scraper that learns extraction rules from examples
mlscraper is a Python library designed to automatically extract structured data from HTML pages without requiring developers to manually write CSS selectors or XPath rules. Instead of defining extraction logic by hand, users provide a few examples of the data they want to retrieve from a webpage. It analyzes those examples within the HTML document and determines patterns or rules that can be used to extract the same type of information from similar pages. Once trained, the generated scraper... -
17
Scylla
Intelligent proxy pool for collecting and managing public proxies
Scylla is an open source proxy pool system designed to collect, validate, and manage large numbers of public proxy servers for use in web scraping and data extraction workflows. It automatically crawls the internet to discover proxy IP addresses and evaluates their availability and reliability before adding them to a usable pool. It includes a JSON API that allows developers and applications to retrieve proxy information programmatically, making it easier to integrate proxy rotation into... -
18
JAMon API
Monitor Java applications - SQL, HTTP, Methods, Exceptions and more.
JAMon API is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor the performance and scalability of production applications. JAMon tracks hits, execution times (total, avg, min, max, std dev), and more. * JAMon Users Manual: For more on the JAMon, including installing, configuring, and using it, see http://jamonapi.sourceforge.net/. * Support: If you have any questions about usage please post a question on the forum at ... -
19
Zenoss Community Edition
Zenoss - Intelligent IT Operations Management
Zenoss provides software-defined IT operations for the world’s largest organizations. We deliver the ultimate level of IT service health with simplicity by providing the most granular and intelligent IT service modeling possible, at any scale, and sharing these unique insights with other IT operations management (ITOM) tools to make them more efficient. Zenoss Community Edition is not a “demo” or trial version of Zenoss Enterprise or Zenoss Cloud! Before You install Zenoss Community... -
20
StreamAlert
StreamAlert is a serverless, realtime data analysis framework
StreamAlert is a serverless, real-time data analysis framework that empowers you to ingest, analyze, and alert on data from any environment, using data sources and alerting logic you define. Computer security teams use StreamAlert to scan terabytes of log data every day for incident detection and response. Incoming log data will be classified and processed by the rules engine. Alerts are then sent to one or more outputs. Rules are written in Python; they can utilize any Python libraries or... -
21
CloudMapper
CloudMapper analyzes your Amazon Web Services (AWS) environments
...You must have AWS credentials configured that can be used by the CLI with reading permissions for the different metadata to collect. Cloudmapper needs to make IAM calls and cannot use session credentials for collection, so you cannot use the AWS-vault server if you want to collect data, and must pass role credentials indirectly or configure AWS credentials manually inside the container. Generate HTML report. Includes a summary of the accounts and audit findings. Generate an HTML report for the IAM information of an account. -
22
lxspider
Educational Python web scraping case collection for many sites
lxSpider is a collection of web scraping examples designed primarily for learning and experimentation with data extraction techniques. It gathers numerous crawler implementations that demonstrate how to collect data from a wide range of websites and online services. It focuses heavily on practical cases that illustrate how different platforms handle requests, authentication parameters, and anti-scraping protections. lxSpider includes examples targeting areas such as e-commerce platforms, social media services, content sites, research databases, and information portals. ... -
23
Collectral
Free Mobile Data Collection Software, Dynamic Mobile Forms App
Collectral allow you to collect data from from Android Mobile Applications directly into your servers. Collectral Consist from 3 components 1. Main Collectral Site where you need to register and set key 2. Collectral ERP - it is WordPress plugin 3. Android Application - Which you need to get from google play In this page you can find only WordPress Plugin as download Android component located in :... -
24
Base62 Encode/Decode Utility
A tool for encoding/decoding to base62
the package chewingum contains two main programs: - base62 - chew They have been implemented in perl. Base62 has been coded in c++ too. chew has been written to shorten long decimal integer numbers into base62 strings. It's not a mere base conversion, because it doesn't use arithmetic inside its logic. In this way it can convert very long numbers into strings quickly, without overflows. For example if I want to share an id using an url, this will make the url shorter. the... -
25
GoogleScraper
Python tool for scraping search engine results from many providers
GoogleScraper is a Python-based tool designed to automatically collect and process search engine results from multiple providers. It enables developers and researchers to programmatically query search engines and extract useful information such as links, titles, and result descriptions. GoogleScraper supports several major search engines and can be used to gather structured datasets from search result pages for further analysis. It provides two different scraping approaches: sending direct...