Open Source Go Logging Software

Extract internal monitoring data from application logs for collection in a time-series database. mtail is a tool for extracting metrics from application logs to be exported into a timeseries database or timeseries calculator for alerting and dashboarding. It fills a monitoring niche by being the glue between applications that do not export their own internal state (other than via logs) and existing monitoring systems, such that system operators do not need to patch those applications to instrument them or writing custom extraction code for every such application. The extraction is controlled by mtail programs which define patterns and actions. Metrics are exported for scraping by a collector as JSON or Prometheus format over HTTP, or can be periodically sent to a collectd, StatsD, or Graphite collector socket. Precompiled binaries for released versions are available in the Releases page on Github. Using the latest production release binary is the recommended way of installing mtail.

Beats is a platform of lightweight, single-use data shippers written in Go. It captures all sorts of operational data from hundreds or thousands of machines and systems and sends them to Logstash or Elasticsearch. Beats is lightweight in that it has a small installation footprint, uses limited system resources and has no runtime dependencies. It consists of: Auditbeat (for audit data), Filebeat (for log files), Functionbeat (serverless shipper), Heartbeat (for uptime monitoring), Metricbeat (for metrics), Packetbeat (for monitoring network data), and Winlogbeat (for Windows Event logs), with many other Beats created by the community.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki. Loki supports multi-tenancy so that data between tenants is completely separated. Multi-tenancy is achieved through a tenant ID (which is represented as an alphanumeric string). When multi-tenancy mode is disabled, all requests are internally given a tenant ID of “fake”.

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color-coded for quicker debugging. The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed. When a pod contains multiple containers Stern can tail all of them too without having to do this manually for each one. Simply specify the container flag to limit what containers to show. By default all containers are listened to. If you don't want to build from source go grab a binary release. Govendor is required to install vendored dependencies. On macOS, you can also install Stern using Homebrew.

Kafka/distributed commit log service in Go. Protocol compatible with Kafka so Kafka clients and services work with Jocko. Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper). Able to use percentages of disk space for retention policies rather than only bytes and time kept. Handling size configs when you change the number of partitions or add topics. API versioning [more API versions to implement]. Replication [first draft done - testing heavily now]. Partition consensus and distribution.

The Logging operator solves your logging-related problems in Kubernetes environments by automating the deployment and configuration of a Kubernetes logging pipeline. The Logging operator manages the log collectors and log forwarders of your logging infrastructure, and the routing rules that specify where you want to send your different log messages. You can filter and process the incoming log messages using the flow custom resource of the log forwarder to route them to the appropriate output. The outputs are the destinations where you want to send your log messages, for example, Elasticsearch, or an Amazon S3 bucket. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users can reference but cannot modify.

Logrus is a structured, pluggable logger for Go (golang) that is completely API compatible with the standard library logger. It encourages careful, structured logging through much more discoverable logging fields rather than long, unparseable error messages. This produces much more useful logging messages. Logrus is currently in maintenance mode, which means that new features will no longer be introduced. This does not mean however, that it is dead. It continues to be maintained for security, backwards compatibility and performance, so you can still rely on it for structured logging in Golang. Please take note however, that everything using Logrus will need to use lower-case so as to avoid casing issues.

Logspout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system. It's a mostly stateless log appliance. It's not meant for managing log files or looking at history. It is just a means to get your logs out to live somewhere else, where they belong. For now it only captures stdout and stderr, but a module to collect container syslog is planned. Logspout is a very small Docker container (15.2MB virtual, based on Alpine). The simplest way to use logspout is to just take all logs and ship to a remote syslog. Just pass a syslog URI (or several comma separated URIs) as the command. Here we show use of the tls encrypted transport option in the URI. You can tell logspout to ignore specific containers by setting an environment variable when starting your container.