librdkafka Files

KIP-932 Queues for Kafka – Now in Preview

• Added a preview implementation of the share consumer (Queues for Kafka, KIP-932). Members of a share group cooperatively consume from the same partitions with per-record acquire/acknowledge semantics and redelivery, providing queue-like consumption on top of Kafka.
• New rd_kafka_share_* public API, with a dedicated rd_kafka_share_t handle created via rd_kafka_share_consumer_new():
• Subscription: rd_kafka_share_subscribe(), rd_kafka_share_unsubscribe(), rd_kafka_share_subscription().
• Batch polling: rd_kafka_share_poll() returns an rd_kafka_messages_t batch (rd_kafka_messages_count() / rd_kafka_messages_get() / rd_kafka_messages_destroy()).
• Acknowledgement: rd_kafka_share_acknowledge(), rd_kafka_share_acknowledge_type(), rd_kafka_share_acknowledge_offset() with ACCEPT / RELEASE / REJECT types, and rd_kafka_message_delivery_count().
• Commit: rd_kafka_share_commit_sync(), rd_kafka_share_commit_async() and the acknowledgement-commit callback (rd_kafka_share_set_acknowledgement_commit_cb()).
• Lifecycle: rd_kafka_share_consumer_close(), rd_kafka_share_consumer_close_queue(), rd_kafka_share_destroy().
• Two acknowledgement modes selected by share.acknowledgement.mode (default implicit; explicit requires the application to acknowledge every record before the next poll).
• New max.poll.records property (default 500) and adjusted defaults for several network properties for share consumers (receive.message.max.bytes, connections.max.idle.ms, reconnect.backoff.ms, reconnect.backoff.max.ms).
• See the Share consumers (Queues for Kafka) section of INTRODUCTION.md, the Share consumer section in rdkafka.h, and the examples/share_consumer* programs.

[!Note] The KIP-932 share consumer is currently in Preview and should not be used in production environments. The public interfaces may change before General Availability, and known limitations apply (see INTRODUCTION.md). The share consumer is single-threaded and not thread-safe by design. It requires a broker with share groups enabled (generally available in Apache Kafka 4.2.0).

Enhancements

• Add aws_iam option to sasl.oauthbearer.metadata.authentication.type, with a defensive stub that fails when no token-refresh callback is registered.

Fixes

General fixes

• Issues: [#5135]. Fix compilation with CMake when CURL is disabled. The OAuthBearer OIDC code included <curl/curl.h> under #ifdef WITH_OAUTHBEARER_OIDC, but CMake always defines that macro (to 0 or 1), so CURL was required even when it was turned off. Happening since 2.11.0 (#5136).
• Issues: [#5282]. Fix rd_atomic32_set/rd_atomic64_set returning the new value instead of the previous one in CMake builds. CMake never defined HAVE_ATOMICS_{32,64}_ATOMIC, so the setters used a non-atomic fallback that returned the new value, which prevented the ALL_BROKERS_DOWN event from being raised under CMake. Happening since 2.11.1 (#5136).

Consumer fixes

• Issues: [#5541]. Improve error handling in the KIP-848 consumer group protocol:
  • Defer the leave heartbeat until the assignment is revoked, so a member that exceeds max.poll.interval.ms rejoins cleanly instead of being rejected with a fatal INVALID_REQUEST.
  • Treat GROUP_ID_NOT_FOUND in the ConsumerGroupHeartbeat response as fatal, except while the member is already leaving.
  • Surface unexpected permanent broker-level heartbeat errors instead of retrying them in a loop; internal transport/timeout codes keep their existing retry/reconnect handling. Happening since 2.12.0 (#5488)

## Checksums Release asset checksums:

• v2.15.0.zip SHA256 d0655c770f7edc5699855f65b391d1e20dc24fc03ecf2dd680665934dd4d1cc1
• v2.15.0.tar.gz SHA256 259015220cdca708afe838b5aa79ebf1a5fb710fb4179cf918d390aed85d5dbc