Apache Pinot 1.6.0

Changes since Apache Pinot 1.5.1.

Highlights

• SQL DDL: manage tables with CREATE TABLE, DROP TABLE, SHOW TABLES, and SHOW CREATE TABLE through the controller's /sql/ddl endpoint. Materialized views gain CREATE/SHOW/DROP DDL and single-stage query-rewrite support. https://github.com/apache/pinot/pull/18241 https://github.com/apache/pinot/pull/18544 https://github.com/apache/pinot/pull/18529
• Analytics: GROUPING SETS, ROLLUP, and CUBE are supported in both query engines. Multi-stage execution also gains streaming distinct execution and lookup-join planning improvements. https://github.com/apache/pinot/pull/18662 https://github.com/apache/pinot/pull/18817 https://github.com/apache/pinot/pull/19419 https://github.com/apache/pinot/pull/18158
• Vector search: add IVF_FLAT, IVF_PQ, and IVF_ON_DISK backends alongside HNSW, with filter-aware search, radius queries, and backend tuning controls. https://github.com/apache/pinot/pull/17994 https://github.com/apache/pinot/pull/18090 https://github.com/apache/pinot/pull/18119
• Data handling: introduce OPEN_STRUCT for evolving nested data, a logical UUID type, Arrow IPC batch ingestion, and BSON input support. https://github.com/apache/pinot/pull/18368 https://github.com/apache/pinot/pull/19040 https://github.com/apache/pinot/pull/18869 https://github.com/apache/pinot/pull/17871 https://github.com/apache/pinot/pull/18950
• Reliability: improve null-aware filtering, multi-stage buffer cleanup during failure/cancellation, and transactional Kafka offset handling. https://github.com/apache/pinot/pull/19440 https://github.com/apache/pinot/pull/19462 https://github.com/apache/pinot/pull/19161

Fixes and improvements

• Lucene rollback recovery: rebuild incompatible text and HNSW indexes from retained original forward-index values during segment loading. Compatible indexes are reused; missing recovery sources fail before replacing the index. Lucene remains 9.12.0. Install this recovery release on the rollback target before a later Lucene upgrade, and retain the original forward indexes. Rebuilding adds CPU/I/O and loading time. #19976, #19900.
• Legacy table-config notices: record deprecation versions, replacement guidance and removal intent for 16 legacy properties. The changes are limited to annotations/Javadoc; runtime acceptance and JSON compatibility remain unchanged. #19978, #18411.
• Approximation reporting: the approximatedResult response flag reports when a SMART aggregation actually produces an approximate result, including explicit SMART queries, rather than whether a rewrite occurred. The corresponding broker meter is BROKER_RESPONSES_WITH_APPROXIMATED_RESULT, and the query-log field is approximatedResult. Always-approximate functions do not set this flag. #19972.
• Conditional aggregate correctness: run the multi-stage approximate rewrite after CASE-to-FILTER conversion to preserve NULL handling for conditional distinct counts when approximation parameters are configured. Automatic approximate rewriting remains disabled by default. #19977, #19974.

Upgrade notes and removals

• Pinot services now require Java 25 or newer to build and run. Java/JDBC clients and the explicitly pinned SPI/common artifacts continue to target Java 11 bytecode. Update service JVMs and check custom service plugins before upgrading. https://github.com/apache/pinot/pull/19014
• The default Scala binary version changes to 2.13. Official Spark 3 artifacts, including pinot-spark-3-connector, are compiled for Scala 2.13; their Maven artifact IDs have no Scala-version suffix. These artifacts require Java 25 and Spark built for Scala 2.13. Scala 2.12 requires a custom source build using the scala-2.12 profile. https://github.com/apache/pinot/pull/18100
• Native text/FST indexes and TEXT_CONTAINS are removed. Migrate affected configurations and queries to supported Lucene text/FST facilities and TEXT_MATCH, reviewing the query semantics before upgrading. https://github.com/apache/pinot/pull/17998
• Spark 2 and Presto integrations, the deprecated Kafka high-level consumer, and several APIs deprecated in releases through 1.3.0 are removed. Recompile custom integrations against 1.6.0 and replace removed API calls. https://github.com/apache/pinot/pull/17950 https://github.com/apache/pinot/pull/17947 https://github.com/apache/pinot/pull/18931 https://github.com/apache/pinot/pull/19139 https://github.com/apache/pinot/pull/19140 https://github.com/apache/pinot/pull/19141 https://github.com/apache/pinot/pull/19142 https://github.com/apache/pinot/pull/19143
• New and updated schemas submitted through the controller reject deprecated TimeFieldSpec (fieldType=TIME); existing stored schemas can still load. Use DateTimeFieldSpec (fieldType=DATE_TIME). https://github.com/apache/pinot/pull/18502
• Legacy controller periodic-task aliases are removed. For example, replace controller.retention.frequencyInSeconds with controller.retention.frequencyPeriod and a duration value such as 6h. https://github.com/apache/pinot/commit/8dec9686d783b76e2899ac76a91df31acc8e1765
• Kafka's raw auto.offset.reset property now defaults to earliest when omitted. This controls recovery from unavailable Kafka offsets and is separate from Pinot's stream.kafka.consumer.prop.auto.offset.reset initial partition-offset setting. Set the raw property explicitly if another recovery behavior is required. https://github.com/apache/pinot/pull/19359
• Single-stage queries no longer silently ignore HAVING; unsupported QUALIFY clauses are rejected. Review queries that relied on the earlier behavior. https://github.com/apache/pinot/pull/19554 https://github.com/apache/pinot/pull/19553

SMART aggregation mixed-version restrictions

The approximation-reporting change adds DataTable metadata and StatMap keys that older readers cannot safely skip. The risk occurs when a SMART aggregation actually turns approximate, including a SMART function explicitly written in a query. Leaving automatic rewriting disabled does not protect explicit SMART queries.

• Upgrade brokers before servers for the DataTable path. An old broker receiving new final-result metadata can misread subsequent metadata or fail with BufferUnderflowException.
• Broker-first ordering alone does not make arbitrary server canaries or downgrades safe. In the multi-stage path, an older StatMap reader can drop that query's statistics, including numGroupsLimitReached, on encountering a new ordinal. Avoid approximation-producing SMART execution on paths where older components would receive the new metadata; do not assume arbitrary canary/downgrade combinations are supported.
• Older servers returning final results do not emit the new metadata. During that transition, a newer broker can report approximatedResult=false for a result produced from a sketch. Treat the flag as incomplete until relevant servers are upgraded.

These are known compatibility limits of #19972, not fixed by the CASE-ordering fix. Plan the rollout and query controls accordingly.

Release artifacts

Release commit: affe120ac8e858ace266961931f0c4dabfe21704.

• Source distribution, signature, SHA-512.
• Binary distribution, signature, SHA-512.
• Maven Central artifacts.

Signing key: F6C06B50BFB7A502B1F95901FE4489E8BC6B342D (Apache Pinot KEYS).