A Fluent Bit output plugin for Amazon Kinesis Data Streams. A partition key is used to group data by shard within a stream. A Kinesis Data Stream uses the partition key that is associated with each data record to determine which shard a given data record belongs to. For example, if your logs come from Docker containers, you can use container_id as the partition key, and the logs will be grouped and stored on different shards depending upon the id of the container they were generated from. As the data within a shard are coarsely ordered, you will get all your logs from one container in one shard roughly in order. Nested partition key is supported and you can use -> to point to your target key which is nested under another key. For example, your partition_key could be kubernetes->pod_name. If you don't set a partition key or put an invalid one, a random key will be generated, and the logs will be directed to random shards.
Features
- The plugin requires kinesis:PutRecords permissions
- This plugin uses the AWS SDK Go
- This plugin has been tested with Fluent Bit 1.2.0+
- We distribute a container image with Fluent Bit and this plugin
- Our images are available in Amazon ECR Public Gallery
- You can use our SSM Public Parameters to find the Amazon ECR image URI in your region