DownloadLoggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang. Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation. Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources. Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3. With sufficient resources for the Agent specification, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism), and observe their respective CPU, memory, and pod network card transmission rates.
Features
- Supports multiple pipeline and pluggable components, including data transfer, filtering, parsing, and alerting
- Documentation available
- Uses native Kubernetes CRD for operation and management
- Offers a range of observability, reliability, and automation features suitable for production environments
- Based on Loggie, we can build a cloud-native scalable log data platform
- Next-generation cloud-native log collection and transmission
- Building pipelines based on CRD
Project Samples
