Best Time Series Databases for Apache Kafka

InfluxDB is a purpose-built data platform designed to handle all time series data, from users, sensors, applications and infrastructure — seamlessly collecting, storing, visualizing, and turning insight into action. With a library of more than 250 open source Telegraf plugins, importing and monitoring data from any system is easy. InfluxDB empowers developers to build transformative IoT, monitoring and analytics services and applications. InfluxDB’s flexible architecture fits any implementation — whether in the cloud, at the edge or on-premises — and its versatility, accessibility and supporting tools (client libraries, APIs, etc.) make it easy for developers at any level to quickly build applications and services with time series data. Optimized for developer efficiency and productivity, the InfluxDB platform gives builders time to focus on the features and functionalities that give their internal projects value and their applications a competitive edge.

Telegraf is the open source server agent to help you collect metrics from your stacks, sensors and systems. Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. Telegraf is written in Go and compiles into a single binary with no external dependencies, and requires a very minimal memory footprint. Telegraf can collect metrics from a wide array of inputs and write them into a wide array of outputs. It is plugin-driven for both collection and output of data so it is easily extendable. It is written in Go, which means that it is a compiled and standalone binary that can be executed on any system with no need for external dependencies, no npm, pip, gem, or other package management tools required. With 300+ plugins already written by subject matter experts on the data in the community, it is easy to start collecting metrics from your end-points.

Instaclustr is the Open Source-as-a-Service company, delivering reliability at scale. We operate an automated, proven, and trusted managed environment, providing database, analytics, search, and messaging. We enable companies to focus internal development and operational resources on building cutting edge customer-facing applications. Instaclustr works with cloud providers including AWS, Heroku, Azure, IBM Cloud, and Google Cloud Platform. The company has SOC 2 certification and provides 24/7 customer support.

Power your metrics and alerting with a leading open-source monitoring solution. Prometheus fundamentally stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Besides stored time series, Prometheus may generate temporary derived time series as the result of queries. Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus's expression browser, or consumed by external systems via the HTTP API. Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc.). Download: https://sourceforge.net/projects/prometheus.mirror/

Apache Druid is an open source distributed data store. Druid’s core design combines ideas from data warehouses, timeseries databases, and search systems to create a high performance real-time analytics database for a broad range of use cases. Druid merges key characteristics of each of the 3 systems into its ingestion layer, storage format, querying layer, and core architecture. Druid stores and compresses each column individually, and only needs to read the ones needed for a particular query, which supports fast scans, rankings, and groupBys. Druid creates inverted indexes for string values for fast search and filter. Out-of-the-box connectors for Apache Kafka, HDFS, AWS S3, stream processors, and more. Druid intelligently partitions data based on time and time-based queries are significantly faster than traditional databases. Scale up or down by just adding or removing servers, and Druid automatically rebalances. Fault-tolerant architecture routes around server failures.

The enterprise database for time series, documents, and vectors. Store any type of data and combine the simplicity of SQL with the scalability of NoSQL. CrateDB is an open source distributed database running queries in milliseconds, whatever the complexity, volume and velocity of data.

Warp 10 is a modular open source platform that collects, stores, and analyzes data from sensors. Shaped for the IoT with a flexible data model, Warp 10 provides a unique and powerful framework to simplify your processes from data collection to analysis and visualization, with the support of geolocated data in its core model (called Geo Time Series). Warp 10 is both a time series database and a powerful analytics environment, allowing you to make: statistics, extraction of characteristics for training models, filtering and cleaning of data, detection of patterns and anomalies, synchronization or even forecasts. The analysis environment can be implemented within a large ecosystem of software components such as Spark, Kafka Streams, Hadoop, Jupyter, Zeppelin and many more. It can also access data stored in many existing solutions, relational or NoSQL databases, search engines and S3 type object storage system.

Heroic is an open-source monitoring system originally built at Spotify to address problems faced with large scale gathering and near real-time analysis of metrics. Heroic uses a small set of components which are responsible for very specific things. Indefinite retention, as long as you have the hardware spend. Federation support to connect multiple Heroic clusters into a global interface. Heroic uses a small set of components which are responsible for very specific things. Consumers are the component responsible for consuming metrics. When building Heroic it was quickly realized that navigating hundreds of millions of time series without context is hard. Heroic has support for federating requests, which allows multiple independent Heroic clusters to serve clients through a single global interface. This can be used to reduce the amount of geographical traffic by allowing one cluster to operate completely isolated within its zone.

Circonus IRONdb makes it easy to handle and store unlimited volumes of telemetry data, easily handling billions of metric streams. Circonus IRONdb enables users to identify areas of opportunity and challenge in real time, providing forensic, predictive, and automated analytics capabilities that no other product can match. Rely on machine learning to automatically set a “new normal” as your data and operations dynamically change. Circonus IRONdb integrates with Grafana, which has native support for our analytics query language. We are also compatible with other visualization apps, such as Graphite-web. Circonus IRONdb keeps your data safe by storing multiple copies of your data in a cluster of IRONdb nodes. System administrators typically manage clustering, often spending significant time maintaining it and keeping it working. Circonus IRONdb allows operators to set and forget their cluster, and stop wasting resources manually managing their time series data store.

QuestDB is a relational column-oriented database designed for time series and event data. It uses SQL with extensions for time series to assist with real-time analytics. These pages cover core concepts of QuestDB, including setup steps, usage guides, and reference documentation for syntax, APIs and configuration. This section describes the architecture of QuestDB, how it stores and queries data, and introduces features and capabilities unique to the system. Designated timestamp is a core feature that enables time-oriented language capabilities and partitioning. Symbol type makes storing and retrieving repetitive strings efficient. Storage model describes how QuestDB stores records and partitions within tables. Indexes can be used for faster read access on specific columns. Partitions can be used for significant performance benefits on calculations and queries. SQL extensions allow performant time series analysis with a concise syntax.