Search Results for "data collect" - Page 2
Sort By:
Showing 297 open source projects for "data collect"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
fess
Open source enterprise search server for websites, files, and data
Fess is an open source enterprise search server designed to provide powerful full-text search capabilities across multiple data sources. It enables organizations to quickly deploy a scalable search environment without requiring deep knowledge of underlying search technologies. Fess is built on top of OpenSearch and offers an integrated solution for crawling, indexing, and searching documents from websites, file systems, and various data stores. Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. ... -
2
RisingWave
RisingWave: the next-generation streaming database in the cloud
RisingWave is a cloud-native streaming database that uses SQL as the interface. It is designed to reduce the complexity and cost of building real-time applications. RisingWave consumes streaming data, performs continuous queries, and updates results dynamically. As a database system, RisingWave maintains results in its own storage so that users can access data efficiently. RisingWave accepts data from sources like Apache Kafka, Apache Pulsar, Amazon Kinesis, Redpanda, and materialized CDC... -
3
Spider
High-performance Rust web crawler and scraper for large-scale data
...It focuses on speed, concurrency, and reliability by using asynchronous and multi-threaded processing to handle large volumes of web pages. It can rapidly crawl websites to collect links, retrieve page content, and extract structured information from HTML documents. Spider can operate concurrently across many pages, allowing it to gather large datasets in a short period of time. Spider also provides mechanisms for subscribing to crawl events so developers can process page data such as URLs, status codes, or HTML content as it is discovered. ... -
4
OpenCTI
Open Cyber Threat Intelligence Platform
OpenCTI is a comprehensive open-source cyber threat intelligence platform designed to help organizations collect, structure, analyze, and share information about cyber threats. It provides a modern web application backed by a GraphQL API and a data model aligned with the STIX2 standard to ensure interoperability across the threat intelligence ecosystem. The platform enables teams to correlate technical indicators such as observables and TTPs with higher-level context like attribution and victimology, creating a unified intelligence knowledge base. ... -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
Laminar
Open-source all-in-one platform for engineering AI products
Laminar is an open source all-in-one platform for engineering best-in-class LLM products. Data governs the quality of your LLM application. Laminar helps you collect it, understand it, and use it. When you trace your LLM application, you get a clear picture of every step of execution and simultaneously collect invaluable data. You can use it to set up better evaluations, as dynamic few-shot examples, and for fine-tuning. All traces are sent in the background via gRPC with minimal overhead. ... -
6
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website. ... -
7
Grafana Agent
Vendor-neutral programmable observability pipelines
Grafana Agent is an OpenTelemetry Collector distribution with a configuration inspired by Terraform. It is designed to be flexible, performant, and compatible with multiple ecosystems such as Prometheus and OpenTelemetry. Grafana Agent is based on components. Components are wired together to form programmable observability pipelines for telemetry collection, processing, and delivery. -
8
FinMind
Open Data, more than 50 financial data
In the era of big data, data is the foundation of everything. We collect more than 50 kinds of Taiwan stock related information and provide download, online analysis, and backtesting. Regardless of the program, you can download data through the api provided by FinMind, or you can download data directly from the website. After data is available, statistical analysis, regression analysis, time series analysis, machine learning, and deep learning can be performed. ... -
9
fluentbit
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX
...No more OOM errors! Integration with all your technology, cloud-native services, containers, streaming processors, and data backends. Fully event-driven design leverages the operating system API for performance and reliability. All operations to collect and deliver data are asynchronous. -
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
10
ArchiveBox
Open source self-hosted web archiving
ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view websites offline. Without active preservation effort, everything on the internet eventually disappears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content. ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. ... -
11
OpenTelemetry Collector distributions
OpenTelemetry Collector Official Releases
High-quality, ubiquitous, and portable telemetry to enable effective observability. OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Create and collect telemetry from your services and software, then forward it to a variety of analysis tools. OpenTelemetry integrates with many popular libraries and frameworks, and supports code-based and zero-code instrumentation. -
12
dxy-covid-19-crawler
Realtime crawler for COVID-19 outbreak statistics from DXY data
DXY-COVID-19-Crawler is a Python-based project designed to collect real-time COVID-19 infection data from the public dataset provided by Ding Xiang Yuan (DXY). The crawler periodically retrieves pandemic statistics and stores them in a database so that historical changes in the outbreak can be preserved and analyzed later. It was created to make up-to-date infection data more accessible for developers, researchers, and analysts who wanted to build visualizations or conduct data analysis during the early stages of the pandemic. ... -
13
Vector
A high-performance observability data pipeline
Vector is a Rust‑based, high‑performance observability data pipeline tool (agent + aggregator) designed to collect, transform, and route logs and metrics at scale. Created by Datadog, it aims to be the only tool needed from ingestion to vendor output, providing cost-efficient, safe, and flexible telemetry processing. -
14
Floorp Browser
All of source code of version 10 or later of Floorp Browser
...In addition to the usual Firefox theme customization, Floorp allows you to switch between five different interface designs, including OS-specific themes. Floorp is based on Firefox ESR. Floorp will be updated every 4 weeks, with security updates provided before each Firefox release. We don't collect personal information from users. We don't track users. We don't sell user data. We have no affiliation with any advertising companies. Floorp's source code is entirely open, allowing anyone to view it and contribute to the project. Not only is the browser itself open source, but the build environment is as well. -
15
Alibaba iLogtail
Fast and Lightweight Observability Data Collector
iLogtail was born for observable scenarios and has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of external Alibaba Cloud customers. You can deploy it in physical machines, Kubernetes and other environments to collect telemetry data, such as logs, traces and metrics. Supports a variety of Logs, Traces, and Metrics data collection, and is friendly to container and Kubernetes environment support. The resource cost of data collection is quite low, 5-20 times better than similar telemetry data collection Agent performance. High stability, used in the production of Alibaba and tens of thousands of Alibaba Cloud customers, and collecting dozens of petabytes of observable data every day with nearly tens of millions deployments. -
16
SPX
A simple & straight-to-the-point PHP profiling extension
...Multi metrics capable: 22 are currently supported (various time & memory metrics, included files, objects in use, I/O...). Able to collect data without losing context. For example Xhprof (and potentially its forks) aggregates data per caller / callee pairs, which implies the loss of the full call stack and forbids timeline or Flamegraph based analysis. -
17
FinalRecon
All-in-one Python web reconnaissance tool for fast target analysis
FinalRecon is an all-in-one web reconnaissance tool written in Python that helps security professionals gather information about a target website quickly and efficiently. It combines multiple reconnaissance techniques into a single command-line utility so users do not need to run several separate tools to collect similar data. FinalRecon focuses on providing a fast overview of a web target while maintaining accuracy in the collected results. It includes modules for gathering server information, analyzing SSL certificates, performing WHOIS lookups, and crawling website resources. FinalRecon can also enumerate DNS records, discover subdomains, search for directories and files, and scan common network ports. ... -
18
JSON Schema Validator
A fast Java JSON schema validator that supports draft V4, V6, V7, V201
This is a Java implementation of the JSON Schema Core Draft v4, v6, v7, v2019-09 and v2020-12(partial) specification for JSON schema validation. In addition, it also works for OpenAPI 3.0 request/response validation with some configuration flags. For users who want to collect information from a JSON node based on the schema, the walkers can help. The default JSON parser is the Jackson which is the most popular one. As it is a key component in our light-4j microservices framework to validate... -
19
Profile Summary
Tool for visualizing GitHub profiles
Profile Summary for GitHub is a web application for turning a public GitHub account into a more visual overview of its development activity. It uses the GitHub API to collect profile and repository information and present it as summary graphics and statistics. Its visualizations include repository distribution by programming language and commit distribution by language. Users can generate summaries for GitHub usernames through the web interface. The project can also be self-hosted and... -
20
skycaiji
Open source web scraping system for automated data collection tasks
SkyCaiji is an open source web scraping and data collection system designed to gather information from websites through configurable extraction rules. It focuses on simplifying the process of building crawlers by allowing users to visually define scraping rules rather than writing complex code. It can collect structured or unstructured data from many types of webpages and automate the extraction process for large datasets. -
21
litlyx
Analytics for developers, setup Analytics in 30 seconds
The easiest, developer-centric analytics tool. Litlyxis an open-source, self-hostable analytics solution for the modern framework. Litlyx offers a unique eyewear cleaning system that includes a special cleaning solution and reusable microfiber swabs. This system is designed to provide a more thorough and eco-friendly way to clean glasses, lenses, and screens. The brand emphasizes sustainability by reducing single-use plastics and promoting long-term use of their products. Their cleaning kit... -
22
PHP Debug Bar
Debug bar for PHP
...The DebugBar uses DataCollectors to collect data from your PHP code. Some of them are automated but others are manual. Use the DebugBar like an array where keys are the collector names. Includes generic data collectors and collectors for well-known libraries. Easily create your own collectors and their associated view in the bar. DebugBar is very easy to use and you can add it to any of your projects in no time. -
23
Robin
AI-powered tool for dark web OSINT search and investigation
...The project provides a modular architecture separating search, scraping, and AI processing components so it can be extended with new data sources. -
24
Open Wearables
Self-hosted platform to unify wearable health data
Open Wearables is an open-source initiative that aims to provide a community-driven ecosystem for wearable device software and interoperability by connecting sensor data, activity tracking, and health insights across multiple platforms and devices. Instead of relying on closed vendor ecosystems, the project provides standardized data models and APIs that let developers and hobbyists collect, sync, and analyze biometric and environmental data from wearables, DIY sensors, and open hardware projects. ... -
25
Komari
A simple server monitor tool
Komari is a lightweight, self-hosted server monitoring solution designed to deliver simple and efficient infrastructure visibility with minimal resource overhead. The platform uses a lightweight agent to collect performance data from monitored nodes and presents the results through a clean web interface. Its architecture emphasizes ease of deployment and full data ownership, making it attractive for users who prefer private monitoring environments over managed services. Komari is designed to scale across servers of different sizes while maintaining low CPU and memory consumption. ...
