Search Results for "data collection algorithm" - Page 2
Sort By:
Showing 91 open source projects for "data collection algorithm"
View related business solutions-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Geziyor
Blazing fast Go framework for web crawling and data scraping tasks
...With built-in tools for caching, metrics collection, and proxy management, it enables developers to build robust and customizable scraping systems using Go. -
2
DavUtils
Powerful webdav client and sync tools with client-side AES encryption
...It is very flexible and configurable and has advanced data safety features to prevent data loss. The integrated client-side AES encryption makes it secure to use public clouds. -
3
DotnetSpider
Lightweight .NET framework for fast web crawling and data scraping
...DotnetSpider is modular, allowing different components such as request schedulers, downloaders, and storage systems to work together in a flexible workflow. DotnetSpider also supports distributed crawling environments, making it possible to scale data collection across multiple agents and machines. With support for various storage backends and extensible parsing mechanisms, it is suitable for building complex scraping systems or automated data gathering pipelines. -
4
spider_collection
Collection of Python web scraping scripts for data extraction tasks
...Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. It also contains examples of proxy pool integration and encapsulation to support more reliable crawling when working with sites that enforce request limits. -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
5
Ballast is a tool for balancing user load across SSH servers that supports user-specific policies as well as traditional policies based on metrics such as CPU load. It includes a simple client, a lightweight data server, and a data collection agent.
-
6
python-fxxk-spider
Collection of 100+ Python web scraping projects and crawler examples
python-fxxk-spider is a curated collection of Python web scraping and crawler projects gathered in a single repository for reference and learning. It aggregates many independent scraping examples that target a wide range of websites, online services, and public data sources. Instead of being a single crawler tool, it functions as a catalog of ready-made Python spider implementations that demonstrate different scraping techniques. python-fxxk-spider includes scrapers for social media, e-commerce platforms, job listings, music services, video platforms, and various content sites. ... -
7
Front-end development blog
Front-end Development Thoughts
Front-end development blog is a front-end development article repository focused on JavaScript, ES6, React, Redux, and related web engineering topics. It works as a curated index of essays rather than a traditional software package. The repository collects technical thinking on subjects such as AI in development, VS Code history, React Conf, JavaScript number behavior, server-side versus client-side rendering, and React best practices. It also includes articles about CSS Modules, Immutable... -
8
crawlergo
Headless Chrome crawler for collecting URLs for vulnerability scans
crawlergo is a browser-based web crawler designed to collect URLs and request data that can be used by web vulnerability scanning tools. It uses a Chrome headless environment to render web pages and observe behavior during the DOM rendering stage in order to capture as many accessible endpoints as possible. By monitoring the page lifecycle and interacting with web elements, the crawler automatically triggers JavaScript events and navigational actions that would normally occur during real... -
9
DecryptLogin
Python library providing APIs for automated website login workflows
...DecryptLogin supports a wide variety of online services and platforms, including social media sites, developer platforms, cloud services, and other web portals. Developers can integrate these login routines into automation scripts, crawlers, or data collection tools that require authenticated sessions. It also provides example utilities and automation scripts demonstrating how the login APIs can be used in practical scenarios. -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
10
mlscraper
ML-based HTML scraper that learns extraction rules from examples
mlscraper is a Python library designed to automatically extract structured data from HTML pages without requiring developers to manually write CSS selectors or XPath rules. Instead of defining extraction logic by hand, users provide a few examples of the data they want to retrieve from a webpage. It analyzes those examples within the HTML document and determines patterns or rules that can be used to extract the same type of information from similar pages. Once trained, the generated scraper... -
11
Scylla
Intelligent proxy pool for collecting and managing public proxies
Scylla is an open source proxy pool system designed to collect, validate, and manage large numbers of public proxy servers for use in web scraping and data extraction workflows. It automatically crawls the internet to discover proxy IP addresses and evaluates their availability and reliability before adding them to a usable pool. It includes a JSON API that allows developers and applications to retrieve proxy information programmatically, making it easier to integrate proxy rotation into... -
12
JAMon API
Monitor Java applications - SQL, HTTP, Methods, Exceptions and more.
JAMon API is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor the performance and scalability of production applications. JAMon tracks hits, execution times (total, avg, min, max, std dev), and more. * JAMon Users Manual: For more on the JAMon, including installing, configuring, and using it, see http://jamonapi.sourceforge.net/. * Support: If you have any questions about usage please post a question on the forum at ... -
13
Zenoss Community Edition
Zenoss - Intelligent IT Operations Management
Zenoss provides software-defined IT operations for the world’s largest organizations. We deliver the ultimate level of IT service health with simplicity by providing the most granular and intelligent IT service modeling possible, at any scale, and sharing these unique insights with other IT operations management (ITOM) tools to make them more efficient. Zenoss Community Edition is not a “demo” or trial version of Zenoss Enterprise or Zenoss Cloud! Before You install Zenoss Community... -
14
StreamAlert
StreamAlert is a serverless, realtime data analysis framework
StreamAlert is a serverless, real-time data analysis framework that empowers you to ingest, analyze, and alert on data from any environment, using data sources and alerting logic you define. Computer security teams use StreamAlert to scan terabytes of log data every day for incident detection and response. Incoming log data will be classified and processed by the rules engine. Alerts are then sent to one or more outputs. Rules are written in Python; they can utilize any Python libraries or... -
15
CloudMapper
CloudMapper analyzes your Amazon Web Services (AWS) environments
...You must have AWS credentials configured that can be used by the CLI with reading permissions for the different metadata to collect. Cloudmapper needs to make IAM calls and cannot use session credentials for collection, so you cannot use the AWS-vault server if you want to collect data, and must pass role credentials indirectly or configure AWS credentials manually inside the container. Generate HTML report. Includes a summary of the accounts and audit findings. Generate an HTML report for the IAM information of an account. -
16
lxspider
Educational Python web scraping case collection for many sites
lxSpider is a collection of web scraping examples designed primarily for learning and experimentation with data extraction techniques. It gathers numerous crawler implementations that demonstrate how to collect data from a wide range of websites and online services. It focuses heavily on practical cases that illustrate how different platforms handle requests, authentication parameters, and anti-scraping protections. lxSpider includes examples targeting areas such as e-commerce platforms, social media services, content sites, research databases, and information portals. ... -
17
Collectral
Free Mobile Data Collection Software, Dynamic Mobile Forms App
Collectral allow you to collect data from from Android Mobile Applications directly into your servers. Collectral Consist from 3 components 1. Main Collectral Site where you need to register and set key 2. Collectral ERP - it is WordPress plugin 3. Android Application - Which you need to get from google play In this page you can find only WordPress Plugin as download Android component located in :... -
18
Base62 Encode/Decode Utility
A tool for encoding/decoding to base62
the package chewingum contains two main programs: - base62 - chew They have been implemented in perl. Base62 has been coded in c++ too. chew has been written to shorten long decimal integer numbers into base62 strings. It's not a mere base conversion, because it doesn't use arithmetic inside its logic. In this way it can convert very long numbers into strings quickly, without overflows. For example if I want to share an id using an url, this will make the url shorter. the... -
19
GoogleScraper
Python tool for scraping search engine results from many providers
GoogleScraper is a Python-based tool designed to automatically collect and process search engine results from multiple providers. It enables developers and researchers to programmatically query search engines and extract useful information such as links, titles, and result descriptions. GoogleScraper supports several major search engines and can be used to gather structured datasets from search result pages for further analysis. It provides two different scraping approaches: sending direct... -
20
LymPHOS2
LymPHOS2 Web-App
...DOI: 10.1021/pr800500r - Ovelleiro, D., Carrascal, M., Casas, V., Abian, J., LymPHOS: design of a phosphosite database of primary human T cells. Proteomics 2009, 9, 3741–3751. DOI: 10.1002/pmic.200800701 - Gallardo, Ó., Ovelleiro, D., Gay, M., Carrascal, M., Abian, J., A collection of open source applications for mass spectrometry data mining. Proteomics 2014, 20, 2275-2279. DOI: 10.1002/pmic.20140012 -
21
ECommerceCrawlers
Collection of Python ecommerce and website crawler examples projects
ECommerceCrawlers is a collection of practical Python web crawler projects designed to gather data from a variety of ecommerce platforms, websites, and online services. It aggregates many independent crawler examples created by contributors and organized into separate subprojects that target specific sites or data sources. These examples demonstrate how to build and operate web scrapers capable of collecting structured information such as product listings, news content, job postings, social media data, and other publicly available web data. ... -
22
React Performance Devtool
A browser developer tool extension to inspect performance
...With every re-render, measures are updated and logged to the console. To use this devtool extension, you'll need to register an observer in your app which will observe a collection of data (performance measures) over time. Registering an observer hooks an object containing information about the events and performance measures of React components to the window object, which can then be accessed inside the inspected window using eval(). -
23
ATTENTION: For more info and latest version, refer to https://github.com/estigma88/jdhtuq. jDHTUQ is a peer-to-peer DHT system based in Chord algorithm, but built like API to generalize the implementation of peer-to-peer DHT systems. jDHTUQ have two fundamental services , put and get of resource.
-
24
mzitu
Python crawler that downloads image galleries and analyzes titles
...This makes the repository both a scraping example and a small data analysis experiment built around the collected content. Overall, mzitu serves as a learning-oriented implementation of Python web scraping, data processing, and visualization techniques. -
25
WeChatSogou
Python library to crawl and retrieve data from WeChat accounts
...These components work together to process HTTP requests, handle verification mechanisms, and transform HTML or JSON responses into usable objects. Developers can integrate the library into scripts or larger data collection systems to automate gathering content from public accounts.
