Apache Hudi Files

Trino packaging changes

The Trino-Hudi connector now lives in the Hudi repo (RFC-105, [#18837]). This changes which Trino artifacts are published:

• org.apache.hudi:hudi-trino-bundle is no longer published. 1.2.0 is its last version.
• The connector is now published as org.apache.hudi:hudi-trino, a regular (non-shaded) jar built against Trino 483 and JDK 25.
• The hudi-hadoop-trinobase-docker, hudi-hadoop-trinocoordinator-docker and hudi-hadoop-trinoworker-docker test images are no longer published.
• The connector config hudi.table.resolve-column-name-casing.enabled defaults to false, where the connector in the Trino repo defaulted to true. Set it to true for tables with mixed-case column names.

Behavior changes

• hoodie.client.heartbeat.tolerable.misses now defaults to 10 instead of 2 (#18904). With the default 60-second heartbeat interval, a writer is deemed failed after 10 minutes without a heartbeat, up from 2. A healthy writer's commit is no longer aborted after a transient pause such as a GC or a slow storage write, but pending writes from a crashed writer are detected and rolled back later. Set it back to 2 to keep the previous behavior.

What's Changed

• chore(docker): reduce base_java17 and spark_base image size by @kartikeyaagrawal in https://github.com/apache/hudi/pull/18542
• test(spark): Add date logical type test to TestAvroConversionUtils by @ashokkumar-allu in https://github.com/apache/hudi/pull/18584
• feat(common): roll over commit metadata to clean by @kbuci in https://github.com/apache/hudi/pull/18590
• refactor: move checkpoint metadata lookup helper to hudi-common by @suryaprasanna in https://github.com/apache/hudi/pull/18489
• feat(flink): Backport Flink 2.1 nested Parquet column readers and INT64 timestamp dispatch (FLINK-35702) by @skywalker0618 in https://github.com/apache/hudi/pull/18636
• feat(flink): Wire Flink 2.1 nested Parquet readers into the Hudi read path (FLINK-35702) by @skywalker0618 in https://github.com/apache/hudi/pull/18700
• refactor(flink): Remove legacy Parquet nested readers superseded by Flink 2.1 Dremel path (FLINK-35702) by @skywalker0618 in https://github.com/apache/hudi/pull/18701
• docs: Document muttley package as internal/optional for OSS users by @shangxinli in https://github.com/apache/hudi/pull/18394
• perf: Reduce unnecessary FSDataOutputStream#hsync to enhance append performance by @TheR1sing3un in https://github.com/apache/hudi/pull/17517
• fix(flink): add Apache license header to muttley/README.md by @shangxinli in https://github.com/apache/hudi/pull/18713
• feat: add variant type adapter for Flink by @danny0405 in https://github.com/apache/hudi/pull/18702
• chore: migrate the flink ITs run to flink2.1 by @danny0405 in https://github.com/apache/hudi/pull/18717
• feat(utilities): add Spark/HoodieStreamer validators for pre-commit validation - Phase 3 by @shangxinli in https://github.com/apache/hudi/pull/18405
• [MINOR] Fix typos in comments and assertion messages by @shangxinli in https://github.com/apache/hudi/pull/18763
• fix(flink): enforce Parquet VARIANT annotation in Flink schema conversion for unshredded variant by @kbuci in https://github.com/apache/hudi/pull/18539
• feat(spark): add restore_to_instant stored procedure by @mahsoodebrahim in https://github.com/apache/hudi/pull/18696
• feat(spark): add show_inflight_commits and cleanup_stale_inflight_com... by @mahsoodebrahim in https://github.com/apache/hudi/pull/18709
• chore: add spark4.1 and flink2.1 profile entries to RC bundle validation by @yihua in https://github.com/apache/hudi/pull/18796
• fix(flink): fix disable table service not effective in hudi-flink by @fhan688 in https://github.com/apache/hudi/pull/13875
• feat(flink): Backport Flink 2.1 Dremel nested Parquet reader rewrite to hudi-flink1.19.x (FLINK-35702) by @skywalker0618 in https://github.com/apache/hudi/pull/18809
• chore: Fix stale zip file for variant backward compatibility test by @cshuo in https://github.com/apache/hudi/pull/18815
• chore: Add release information for Apache Hudi 1.2.0 by @yihua in https://github.com/apache/hudi/pull/18831
• fix(spark): Add options for archive procedure by @fhan688 in https://github.com/apache/hudi/pull/18437
• fix: RLI bootstrap fails due to NPE with cleaner table service by @lokeshj1703 in https://github.com/apache/hudi/pull/18836
• chore: Fix Flink CI Maven profile arguments by @cshuo in https://github.com/apache/hudi/pull/18845
• [MINOR] Handle cancellation error with HoodieMetadataTableValidator by @lokeshj1703 in https://github.com/apache/hudi/pull/18371
• docs: Update DOAP file to include 0.15.1 by @voonhous in https://github.com/apache/hudi/pull/18838
• feat(trino): [RFC-105] Trino Hudi Connector -- Shim/Bundle Refactor by @yihua in https://github.com/apache/hudi/pull/18782
• feat(flink): add lance format for Flink append only table by @danny0405 in https://github.com/apache/hudi/pull/18741
• refactor(flink): Refactor RowData writer factory to use HoodieSchema by @cshuo in https://github.com/apache/hudi/pull/18873
• fix(cli): Fix the typo in show-inflight CLI command by @yihua in https://github.com/apache/hudi/pull/18868
• perf(streamer): fold validate() error-table WriteStatus sums into one pass by @Davis-Zhang-Onehouse in https://github.com/apache/hudi/pull/18871
• fix(hudi-sync): Fix Hive test temp directory cleanup with JUnit TempDir by @fhan688 in https://github.com/apache/hudi/pull/18883
• fix(flink): set canonical base path in Hive sync config by @fhan688 in https://github.com/apache/hudi/pull/18884
• fix(flink): fix data loss in stream read from earliest by @fhan688 in https://github.com/apache/hudi/pull/18848
• fix(spark): fix MOR bulk insert commit operation error by @fhan688 in https://github.com/apache/hudi/pull/18878
• feat(utilities): migrate HoodieStreamerWriteStatusValidator into pre-commit validator framework by @shangxinli in https://github.com/apache/hudi/pull/18765
• fix: Fix NPE due to race condition while handling rocksdb handles by @lokeshj1703 in https://github.com/apache/hudi/pull/18834
• fix(flink): Trigger a failover after pending instants recommitted for... by @cshuo in https://github.com/apache/hudi/pull/18793
• refactor: Add Lombok annotations to hudi-utilities (Part 1) by @voonhous in https://github.com/apache/hudi/pull/17823
• refactor: Add Lombok Builders to HoodieFileGroupReader, InputSplit, ReaderParameters by @voonhous in https://github.com/apache/hudi/pull/17781
• fix(spark): align CTAS partition fields by table partition order by @fhan688 in https://github.com/apache/hudi/pull/18899
• fix(ci): Flink version corresponds to used in Docker image, resolution for Everit JSON schema by @geserdugarov in https://github.com/apache/hudi/pull/18905
• refactor: Add Lombok annotations to hudi-utilities (Part 3) by @voonhous in https://github.com/apache/hudi/pull/17877
• refactor: Add Lombok annotations to hudi-common module (part 3) by @voonhous in https://github.com/apache/hudi/pull/17825
• refactor: Add Lombok Builder annotation to HoodieLogFormat by @voonhous in https://github.com/apache/hudi/pull/17785
• chore(docker): silence Dockerfile lint warnings across image set by @voonhous in https://github.com/apache/hudi/pull/18664
• refactor(core): Unify record key/index key splitting and extraction by @cshuo in https://github.com/apache/hudi/pull/18842
• fix(streamer): Use checkpoint V1 for non-incremental streamer sources by @yihua in https://github.com/apache/hudi/pull/18896
• fix(flink): Use the execution mode without rocksdb cache by default ... by @cshuo in https://github.com/apache/hudi/pull/18894
• [MINOR] Update DOAP with 0.14.2 Release by @danny0405 in https://github.com/apache/hudi/pull/18924
• fix(flink): avoid repeated timeline reload for unchanged lookup table commits by @fhan688 in https://github.com/apache/hudi/pull/18930
• fix(common): Close log writer output stream on append failure by @fhan688 in https://github.com/apache/hudi/pull/18909
• fix(metrics): NPE handling when hudi metrics is disabled by @HuangZhenQiu in https://github.com/apache/hudi/pull/18947
• feat(flink): Backport Flink 2.1 Dremel nested Parquet reader to hudi-... by @cshuo in https://github.com/apache/hudi/pull/18953
• fix(clustering): retain missing partitions in selected/regex incremental scheduling by @fhan688 in https://github.com/apache/hudi/pull/18945
• fix(flink): Close write client properly in DefaultCleanHandler by @cshuo in https://github.com/apache/hudi/pull/18940
• feat(flink): Backport Flink 2.1 Dremel nested Parquet reader to hudi-... by @cshuo in https://github.com/apache/hudi/pull/18960
• feat(flink): Backport Flink 2.1 Dremel nested Parquet reader to hudi-... by @cshuo in https://github.com/apache/hudi/pull/18959
• docs: RFC-106 - Record Level and Secondary Index Support for Flink Writers by @danny0405 in https://github.com/apache/hudi/pull/17610
• perf(clean): Avoid extra getPathInfo RPC per file during clean execution by @voonhous in https://github.com/apache/hudi/pull/18963
• fix(spark): catch HoodieSchemaNotFoundException in 3-arg DefaultSource.createRelation by @lokeshj1703 in https://github.com/apache/hudi/pull/18977
• refactor: Add Lombok annotations to hudi-common module (part 7) by @voonhous in https://github.com/apache/hudi/pull/18944
• fix(flink): relocate org.apache.flink.dropwizard by @ericyuan915 in https://github.com/apache/hudi/pull/18982
• feat: add more metrics for delta streamer by @prashantwason in https://github.com/apache/hudi/pull/18085
• feat: Add HUDI version and engine properties to commit metadata by @prashantwason in https://github.com/apache/hudi/pull/18183
• [MINOR] Fix NPE in getInputFileSlices when RO path filter returns empty partition by @prashantwason in https://github.com/apache/hudi/pull/18639
• fix: Introduce ExecutorServiceBasedEngineContext to fix ClassNotFoundException in pre-commit validators on Java 11+ by @ashokkumar-allu in https://github.com/apache/hudi/pull/18585
• perf(spark): Resolve drop-partition-columns projection once per write... by @voonhous in https://github.com/apache/hudi/pull/18972
• fix(spark): strip hoodie* meta columns from CDC before/after images by @ad1happy2go in https://github.com/apache/hudi/pull/18948
• perf(spark): Parse bucket index hash-field config once instead of per... by @voonhous in https://github.com/apache/hudi/pull/18979
• perf(metadata): Parse RLI instant time once per batch instead of per ... by @voonhous in https://github.com/apache/hudi/pull/18965
• test(trino): de-flake TestHudi*FileOperations by disabling async table statistics by @wombatu-kun in https://github.com/apache/hudi/pull/18995
• perf(metadata): Avoid per-record enum-array clone and string parse when materializing MDT records by @voonhous in https://github.com/apache/hudi/pull/18997
• [MINOR] Cap UT_FT_10 Azure install to -T 2 to avoid flaky compiler heap OOM by @wombatu-kun in https://github.com/apache/hudi/pull/19008
• refactor(metadata): Replace misused stream reduce with a plain for-loop by @voonhous in https://github.com/apache/hudi/pull/18532
• perf(io): Derive log file size from AppendResult on append-handle close by @voonhous in https://github.com/apache/hudi/pull/19002
• refactor: Add Lombok annotations to hudi-common module (part 8) by @voonhous in https://github.com/apache/hudi/pull/18957
• [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test by @wombatu-kun in https://github.com/apache/hudi/pull/19014
• fix(flink): fix the mor small file record size estimation by @danny0405 in https://github.com/apache/hudi/pull/18991
• perf(kafka-connect): reuse AvroConvertor across records in the connect writer by @wombatu-kun in https://github.com/apache/hudi/pull/19015
• fix(spark): Propagate merge configs to file group reader during clustering by @ad1happy2go in https://github.com/apache/hudi/pull/19007
• perf(kafka-connect): memoize file id per partition path in the connect writer by @wombatu-kun in https://github.com/apache/hudi/pull/19016
• perf(kafka-connect): use a pre-sized ArrayList when flushing buffered records by @wombatu-kun in https://github.com/apache/hudi/pull/19017
• chore(test): document macOS-specific cleaner consistency-guard slowness (#17714) by @voonhous in https://github.com/apache/hudi/pull/19003
• perf(metadata): Resolve column-stats field schemas once per collection instead of per record by @voonhous in https://github.com/apache/hudi/pull/19000
• test(hadoop-mr): enable rollback case in HoodieRealtimeRecordReader.testReader by @oglego in https://github.com/apache/hudi/pull/18693
• feat(flink): Support writing out-of-line BLOB columns by @kbuci in https://github.com/apache/hudi/pull/18958
• fix(flink): reuse the preceeding avg size if there is no eligible estimation by @danny0405 in https://github.com/apache/hudi/pull/19022
• test(flink): de-flake testStreamReadMorTableWithCompactionFromEarliest by @wombatu-kun in https://github.com/apache/hudi/pull/19019
• test(flink): retry short CollectSink reads to de-flake stream-read ITs by @wombatu-kun in https://github.com/apache/hudi/pull/19030
• test(trino): de-flake TestHudi*FileOperations by asserting only synchronous reads by @wombatu-kun in https://github.com/apache/hudi/pull/19004
• fix: prevent heartbeat timer from being permanently killed by slow or delayed heartbeats by @prashantwason in https://github.com/apache/hudi/pull/18904
• perf: Add dedicated batch size config for LSM timeline migration on u... by @cshuo in https://github.com/apache/hudi/pull/19052
• test(clean): cover executing pending clean before scheduling a new one by @suryaprasanna in https://github.com/apache/hudi/pull/19051
• test(spark): de-flake TestHoodieClientMultiWriter early-conflict detection by @voonhous in https://github.com/apache/hudi/pull/19069
• fix(test): close TestingServer + write clients on all exit paths in TestHoodieClientMultiWriter by @nsivabalan in https://github.com/apache/hudi/pull/19062
• [MINOR] Guard detailed metadata size metrics with a config by @nsivabalan in https://github.com/apache/hudi/pull/18803
• test(metadata): Add test coverage for deferred RLI init and bulk_insert by @nsivabalan in https://github.com/apache/hudi/pull/18865
• fix(spark): support consistent hashing clustering on non-partitioned tables by @ad1happy2go in https://github.com/apache/hudi/pull/18968
• chore(integ-test): bump trino-jdbc 390 to 481 by @voonhous in https://github.com/apache/hudi/pull/19073
• fix: Fix Typo in PartitionTTLStrategyType#getPartitionTTLStrategyClassName by @wangxianghu in https://github.com/apache/hudi/pull/19076
• fix: Skip missing properties files gracefully in DFSPropertiesConfiguration by @nsivabalan in https://github.com/apache/hudi/pull/18805
• [HUDI-18827] Fix per-task write token for MOR (table v6) rollback log files by @nsivabalan in https://github.com/apache/hudi/pull/18828
• fix(spark): reject INSERT_OVERWRITE when overlapping with pending clustering by @nsivabalan in https://github.com/apache/hudi/pull/18829
• fix(streamer): override all deserialize() overloads in KafkaAvroSchemaDeserializer by @nsivabalan in https://github.com/apache/hudi/pull/18892
• fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis by @nada-attia in https://github.com/apache/hudi/pull/18147
• [MINOR] Forward spark.hoodie.* SparkConf to write path (parity with read path) by @prashantwason in https://github.com/apache/hudi/pull/18650
• fix(partition-ttl): Fix the integer overflow issue when TTL exceeded 24 days. by @wangxianghu in https://github.com/apache/hudi/pull/19075
• perf(common): Avoid per-record HoodieSchema rebuilds on Avro read/merge paths by @voonhous in https://github.com/apache/hudi/pull/18967
• test(flink): de-flake testLookupJoin lookup-join IT by @wombatu-kun in https://github.com/apache/hudi/pull/19093
• fix(partition-ttl): Fix IllegalArgumentException in KeepByTimeStrategy when no candidate partitions exist by @wangxianghu in https://github.com/apache/hudi/pull/19092
• feat(spark): add repair_orphan_files stored procedure by @mahsoodebrahim in https://github.com/apache/hudi/pull/19121
• fix: remove the dependency to flink-table-planner by @danny0405 in https://github.com/apache/hudi/pull/19131
• chore(glue-sync): Ignore EntityNotFoundException when dropping Glue partitions by @wangxianghu in https://github.com/apache/hudi/pull/19142
• fix(storage-format): return all records when scanning multi-block native HFiles by @yihua in https://github.com/apache/hudi/pull/19146
• refactor(spark): consolidate the vendored AvroUtils fork into hudi-spark-common by @yihua in https://github.com/apache/hudi/pull/19147
• refactor(spark): share Spark 4.x partition-values and mapping classes in hudi-spark4-common by @yihua in https://github.com/apache/hudi/pull/19148
• refactor(spark): dedup catalyst utils and analysis rules across Spark version modules by @yihua in https://github.com/apache/hudi/pull/19149
• refactor(spark): consolidate duplicated small utils across Spark version modules by @yihua in https://github.com/apache/hudi/pull/19150
• refactor: Retire leftover Avro Schema usages in AvroSchemaUtils and LSMTimelineWriter by @voonhous in https://github.com/apache/hudi/pull/19153
• refactor: Migrate reconcileSchema/reconcileSchemaRequirements to HoodieSchema by @voonhous in https://github.com/apache/hudi/pull/19154
• perf(common): replace BitSet with a fixed word array in the ported bloom filter by @yihua in https://github.com/apache/hudi/pull/19140
• test: handle expected OCC conflict in concurrent Java writer test by @yihua in https://github.com/apache/hudi/pull/19124
• refactor(flink): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19156
• refactor(hadoop-mr): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19157
• refactor(hadoop-common): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19160
• refactor(common): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19158
• fix(spark): correct self-recursive equals in ProcedureParameterImpl by @yihua in https://github.com/apache/hudi/pull/19167
• refactor(cli): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19159
• test(spark): add streaming source and writer support coverage by @yihua in https://github.com/apache/hudi/pull/19166
• refactor(spark): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19184
• refactor(aws): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19186
• refactor(utilities): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19185
• refactor(examples): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19187
• fix(spark): make export_instants descending order work by @yihua in https://github.com/apache/hudi/pull/19172
• refactor(sync): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19189
• refactor(client): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19155
• refactor(io): use SLF4J parameterized logging instead of string concatenation by @voonhous in https://github.com/apache/hudi/pull/19188
• test(trino): de-flake testRecordLevelFileSkipping by setting the record-index wait timeout by @wombatu-kun in https://github.com/apache/hudi/pull/19213
• fix(flink): avoid ArrayIndexOutOfBoundsException in NestedColumnReader across the read batch boundary by @ericyuan915 in https://github.com/apache/hudi/pull/19210
• test(integ-test): add Testcontainers E2E for VECTOR/BLOB/VARIANT Hive... by @voonhous in https://github.com/apache/hudi/pull/19203
• fix(spark): demote spurious per-write INFO logs to debug by @yihua in https://github.com/apache/hudi/pull/19170
• test(client): cover low-coverage small classes across hudi-client by @yihua in https://github.com/apache/hudi/pull/19224
• test(integ-test): add Spark 4.1.1 stack to hive-sync E2E matrix by @voonhous in https://github.com/apache/hudi/pull/19216
• fix(spark): shade parquet-variant into the spark bundle for Spark 4.1+ by @voonhous in https://github.com/apache/hudi/pull/19235
• test(spark): add extended SQL parser coverage for index DDL and Hudi column types by @yihua in https://github.com/apache/hudi/pull/19218
• fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort by @vinothchandar in https://github.com/apache/hudi/pull/19181
• fix(common): name the offending expression in BindVisitor's unsupported-predicate error by @wombatu-kun in https://github.com/apache/hudi/pull/19241
• refactor(spark): consolidate the vendored 3.x Avro serde forks into hudi-spark3-common by @yihua in https://github.com/apache/hudi/pull/19168
• fix(flink): prevent data loss on global failover for streaming writes by @Joy-2000 in https://github.com/apache/hudi/pull/19237
• fix(spark): read INLINE blobs as CONTENT on internal write-side Lance... by @voonhous in https://github.com/apache/hudi/pull/19236
• fix: Remove spaces around partition columns by @linliu-code in https://github.com/apache/hudi/pull/18423
• fix: align log4j2 and slf4j versions to resolve IntelliJ test failures by @suryaprasanna in https://github.com/apache/hudi/pull/18177
• fix(spark): match the staged table, not LogicalWriteInfo, in BasicStagedTable.newWriteBuilder by @wombatu-kun in https://github.com/apache/hudi/pull/19251
• test(spark): cover the legacy parquet read path with file-group reader disabled by @yihua in https://github.com/apache/hudi/pull/19133
• fix: reuse Inflater/Deflater in BitCaskDiskMap to avoid JDK8 finalizer contention by @vamsikarnika in https://github.com/apache/hudi/pull/18818
• fix: Improve error message for conflict resolution by @vinishjail97 in https://github.com/apache/hudi/pull/18119
• fix(flink): remove the Source V2 read teardown race via materialized bounded minibatches by @wombatu-kun in https://github.com/apache/hudi/pull/19202
• docs: claim RFC-108 Multi-dataset incremental reads in Hudi Streamer by @ashokkumar-allu in https://github.com/apache/hudi/pull/19308
• test: fix flaky TestJavaHoodieBackedMetadata.testReattemptOfFailedClusteringCommit by @lokeshj1703 in https://github.com/apache/hudi/pull/19120
• test(common): add log-format reader and scanner coverage by @yihua in https://github.com/apache/hudi/pull/19223
• fix(metadata-table): add config to skip zero-size data files on MDT initialization by @nada-attia in https://github.com/apache/hudi/pull/18611
• fix(hive-sync): sync column and partition column comments to HMS by @rangareddy in https://github.com/apache/hudi/pull/19289
• test(spark): add unit coverage for Spark row, schema and sort utilities by @yihua in https://github.com/apache/hudi/pull/19219
• test(common): add unit coverage for config and utility classes by @yihua in https://github.com/apache/hudi/pull/19220
• test(client): add unit coverage for client utilities and services by @yihua in https://github.com/apache/hudi/pull/19222
• fix(hive-sync): close proxied IMetaStoreClient in HoodieHiveSyncClient.close() to prevent HMS connection leak by @skywalker0618 in https://github.com/apache/hudi/pull/19331
• fix(flink): avoid reusing split reader functions across fetchers by @cshuo in https://github.com/apache/hudi/pull/19315
• test(common): add unit coverage for metrics reporters and schema utilities by @yihua in https://github.com/apache/hudi/pull/19221
• fix(hive-sync): fix partition-value parsing on '=' and pushdown comparator overflow by @wombatu-kun in https://github.com/apache/hudi/pull/19336
• fix: Schema evolution: relax existing column to nullable in reconcileSchema by @vamsikarnika in https://github.com/apache/hudi/pull/19337
• fix(hive-sync): set HMS table createTime in seconds instead of milliseconds by @wombatu-kun in https://github.com/apache/hudi/pull/19335
• chore: upload Flink integration-test coverage by @danny0405 in https://github.com/apache/hudi/pull/19343
• fix: do not fall back to timeline server markers on transient HDFS failures by @vamshipasunuru1 in https://github.com/apache/hudi/pull/18887
• refactor(spark): share the Spark 3.x legacy parquet file format via a common base by @yihua in https://github.com/apache/hudi/pull/19341
• fix(spark): improve error message when precombine field value is null by @prashantwason in https://github.com/apache/hudi/pull/18061
• fix(hive-sync): keep HMS lock heartbeat alive and release its thread pool on close by @wombatu-kun in https://github.com/apache/hudi/pull/19334
• fix(utilities): guard Source.releaseResources() against transient RDD unpersist failures by @niks002 in https://github.com/apache/hudi/pull/19328
• chore(utilities): add SQS backlog / in-flight visibility to S3 events source by @Davis-Zhang-Onehouse in https://github.com/apache/hudi/pull/19333
• fix(reader): derive pre-v9 CDC delete markers from the effective payload class by @lokeshj1703 in https://github.com/apache/hudi/pull/19348
• test(common): add LSM file group read path coverage by @danny0405 in https://github.com/apache/hudi/pull/19347
• fix(flink): make hudi-flink-bundle built with flink-bundle-shade-hive usable for Hive sync by @skywalker0618 in https://github.com/apache/hudi/pull/19330
• fix: Handle map/array-nested leaf columns in column stats collection during MOR log-append by @vinishjail97 in https://github.com/apache/hudi/pull/19126
• test(spark): Fix flaky TestSparkFilterHelper by giving it its own SparkSession by @wombatu-kun in https://github.com/apache/hudi/pull/19356
• docs(docker): document build_docker_images.sh flags by @201573 in https://github.com/apache/hudi/pull/18687
• fix(metadata-table): follow-ups for the zero-size file skip (#18611) by @voonhous in https://github.com/apache/hudi/pull/19355
• fix(flink): normalize row logical conversions and improve coverage by @danny0405 in https://github.com/apache/hudi/pull/19351
• fix(flink): deduplicate delete keys and cover write paths by @danny0405 in https://github.com/apache/hudi/pull/19354
• fix(timeline-service): fail marker creation requests when marker flush fails by @yihua in https://github.com/apache/hudi/pull/19368
• test(hudi-client): improve metadata table writer coverage by @cshuo in https://github.com/apache/hudi/pull/19363
• chore(deps): bump io.airlift:aircompressor from 0.27 to 2.0.3 by @dependabot[bot] in https://github.com/apache/hudi/pull/18247
• chore(deps): bump org.apache.thrift:libthrift from 0.14.0 to 0.23.0 in /packaging/hudi-integ-test-bundle by @dependabot[bot] in https://github.com/apache/hudi/pull/18910
• chore(deps): bump org.apache.commons:commons-configuration2 from 2.11.0 to 2.15.0 in /packaging/hudi-cli-bundle by @dependabot[bot] in https://github.com/apache/hudi/pull/18801
• fix(hive-sync): stop the HMS lock heartbeat once the metastore drops the lock by @wombatu-kun in https://github.com/apache/hudi/pull/19370
• fix(record-index-bootstrap): sort record index keys by UTF-8 bytes to match HFile sorting by @nada-attia in https://github.com/apache/hudi/pull/18941
• perf(flink): Parse bucket index hash-field config once instead of per ... by @voonhous in https://github.com/apache/hudi/pull/18993
• fix: Fix lock expiration metric by @linliu-code in https://github.com/apache/hudi/pull/18492
• feat(trino): Migrate the Trino-Hudi connector into the Hudi repo (RFC-105) by @voonhous in https://github.com/apache/hudi/pull/18837
• feat(hive-sync): batch and parallelize HiveQL partition operations by @nsivabalan in https://github.com/apache/hudi/pull/18984
• fix(hive-sync): drop the unreachable HMS lock timeout-recovery path by @wombatu-kun in https://github.com/apache/hudi/pull/19371
• fix(metadata): correct index definition lookup and improve mdt read coverage by @cshuo in https://github.com/apache/hudi/pull/19359
• fix(common): load single archived instant details by @danny0405 in https://github.com/apache/hudi/pull/19385
• fix(schema): gate timestamp-precision change behind a per-field verdict by @yihua in https://github.com/apache/hudi/pull/19029
• fix(reader): give the file group reader schema handler the merged record-merge properties by @wombatu-kun in https://github.com/apache/hudi/pull/19389
• fix(hive-sync): stop HiveDriverPool from swallowing a failed SQL batch by @wombatu-kun in https://github.com/apache/hudi/pull/19391
• feat(trino): resolve merge-required columns from the table schema by @voonhous in https://github.com/apache/hudi/pull/19288
• fix(client): fix NPE in schema conflict resolution on commits with null writer schema by @yihua in https://github.com/apache/hudi/pull/19388
• test(flink): improve streamer config and schema provider coverage by @danny0405 in https://github.com/apache/hudi/pull/19392
• feat(release): add a JDK 25 staging path for org.apache.hudi:hudi-trino by @voonhous in https://github.com/apache/hudi/pull/19410
• test(trino): add MoR read tests for delete markers, custom payloads and commit-time ordering by @voonhous in https://github.com/apache/hudi/pull/19295
• feat(client): enrich write commit callback message and fire it for table-service commits by @codope in https://github.com/apache/hudi/pull/18988
• perf(common): avoid UTF-8 allocations in string comparator by @cshuo in https://github.com/apache/hudi/pull/19414
• fix(hive-sync): make skip_ro_suffix take precedence over sync_snapshot_with_table_name by @yihua in https://github.com/apache/hudi/pull/19427
• fix(utilities): include metadata table index-init instants in the record index validation snapshot by @Davis-Zhang-Onehouse in https://github.com/apache/hudi/pull/19395
• fix(trino): read uncompacted MDT HFILE log deltas and guard index pruning by @voonhous in https://github.com/apache/hudi/pull/19298
• fix(flink): rethrow StreamWriteOperatorCoordinator start() failures by @Joy-2000 in https://github.com/apache/hudi/pull/19432
• fix(trino): remap pushed-down predicate columns to physical file ordinals by @wombatu-kun in https://github.com/apache/hudi/pull/19456
• test(java-client): cover both marker types with the embedded timeline server by @rangareddy in https://github.com/apache/hudi/pull/19444
• fix(metrics): explain how to enable the CloudWatch reporter when hudi-aws is absent by @rangareddy in https://github.com/apache/hudi/pull/19418
• fix(timeline): do not NPE on archived instants without a completion time by @rangareddy in https://github.com/apache/hudi/pull/19452
• perf(trino): cache decimal Avro schema in HudiAvroSerializer instead ... by @voonhous in https://github.com/apache/hudi/pull/19483
• fix(meta-sync): advance last commit time synced when it trails the active timeline midpoint by @yihua in https://github.com/apache/hudi/pull/19239
• fix(fs): stop depending on the optional FileSystem#getScheme() by @rangareddy in https://github.com/apache/hudi/pull/19470
• fix(schema): require a per-field override to promote a bare long to a timestamp logical type by @yihua in https://github.com/apache/hudi/pull/19384
• fix(trino): report real block size and slice splits solely by target_split_size by @voonhous in https://github.com/apache/hudi/pull/19478
• perf(trino): drop the decimal schema cache and memoize prefilled values by @voonhous in https://github.com/apache/hudi/pull/19495
• fix(flink): close CDC image spillable maps on failures by @danny0405 in https://github.com/apache/hudi/pull/19482
• fix(metrics): do not drop the whole CloudWatch batch on one unmappable metric name by @rangareddy in https://github.com/apache/hudi/pull/19476
• test(trino): add a Trino E2E testcontainers pipeline for the RFC-105 connector by @voonhous in https://github.com/apache/hudi/pull/19217
• fix(flink): close lookup reader after cache reload attempts by @danny0405 in https://github.com/apache/hudi/pull/19503
• fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support by @yihua in https://github.com/apache/hudi/pull/19512
• fix(metrics): route the reflection failures a CloudWatch skew actually produces by @rangareddy in https://github.com/apache/hudi/pull/19477
• fix(client): report completed timeline action in clustering callback by @codope in https://github.com/apache/hudi/pull/19464
• perf(flink): use a shared work-stealing split pool for Source V2 bounded reads by @ericyuan915 in https://github.com/apache/hudi/pull/19520
• fix(flink): preserve Avro fixed decimal widths in Parquet writes by @cshuo in https://github.com/apache/hudi/pull/19522
• fix: harden async indexing and improve coverage by @danny0405 in https://github.com/apache/hudi/pull/19537
• fix(storage-format): emit HBase-readable block-index keys in the native HFile writer by @yihua in https://github.com/apache/hudi/pull/19071
• fix(storage-format): fix multi-block previous-block offset and add byte-level HFile writer tests by @yihua in https://github.com/apache/hudi/pull/19083
• feat(hive-sync): parallelize DROP partitions in HiveQL sync mode by @nsivabalan in https://github.com/apache/hudi/pull/19033
• docs: claim RFC-110 native full-text search by @danny0405 in https://github.com/apache/hudi/pull/19614
• fix(avro): fetch correct avro version from pom.properties for shade / fat jars by @Rajeev-01 in https://github.com/apache/hudi/pull/19596
• chore(deps): bump org.apache.logging.log4j:log4j-api from 2.25.4 to 2.25.5 by @dependabot[bot] in https://github.com/apache/hudi/pull/19628
• docs: rfc-107 Support data partition aware RocksDB RecordIndexBackend by @HuangZhenQiu in https://github.com/apache/hudi/pull/19046
• fix(test): cover SparkContext shutdown path in HoodieMetadataTableValidator by @lokeshj1703 in https://github.com/apache/hudi/pull/19349
• fix(core): decode strings read from storage as UTF-8 by @linliu-code in https://github.com/apache/hudi/pull/19454
• feat: Add source path field in table created from S3/GCS source by @kroushan-nit in https://github.com/apache/hudi/pull/14257
• fix: Parallelize cloud object existence checks in S3EventsHoodieIncrSource by @vinishjail97 in https://github.com/apache/hudi/pull/18252
• feat(core): Pass the number of updates from workload profiling through to merge handle by @yihua in https://github.com/apache/hudi/pull/18396
• feat(trino): track trinodb/trino master via a pinned SHA by @voonhous in https://github.com/apache/hudi/pull/19642
• fix(common): normalize hudi table base path in implicit-key lock providers by @Davis-Zhang-Onehouse in https://github.com/apache/hudi/pull/18814
• fix(spark): return partition values in Catalyst's internal representation by @SEPURI-SAI-KRISHNA in https://github.com/apache/hudi/pull/19652
• feat(kinesis): Cross-account Kinesis source support via STS assume-role in JsonKinesisSource by @suryadanny in https://github.com/apache/hudi/pull/19383
• fix(flink): sort ClientIds heartbeat files numerically instead of lexicographically by @adityaanikam in https://github.com/apache/hudi/pull/19653
• fix(storage-format): ignore temporary LSM timeline manifests by @danny0405 in https://github.com/apache/hudi/pull/19659
• fix(spark): don't prune files with unknown null-count on IS NULL by @SEPURI-SAI-KRISHNA in https://github.com/apache/hudi/pull/19646
• fix(spark): make new schema evolution fields nullable by @danny0405 in https://github.com/apache/hudi/pull/19665
• fix(spark): align libthrift with the Hive 2.3.10 client jars by @voonhous in https://github.com/apache/hudi/pull/19682
• fix(lock): stop guarding the filesystem lock with an interned string literal and make the provider serializable by @rangareddy in https://github.com/apache/hudi/pull/19486
• fix: support reading tables partitioned on a nested column by @vinishjail97 in https://github.com/apache/hudi/pull/19123
• [MINOR] Reword the hive.libthrift.version comment in the root pom by @voonhous in https://github.com/apache/hudi/pull/19704
• fix(kinesis): replace ASL-licensed KPL deaggregation with a native decoder by @voonhous in https://github.com/apache/hudi/pull/19707
• fix(flink): keep the declared joda-time in hudi-flink-bundle with flink-bundle-shade-hive2 by @skywalker0618 in https://github.com/apache/hudi/pull/19711
• fix(spark): make slash-separated date partitioning work on the row writer path by @SEPURI-SAI-KRISHNA in https://github.com/apache/hudi/pull/19648
• fix(flink): prevent reusing diverged buffers on memory exhaustion by @fhan688 in https://github.com/apache/hudi/pull/19692
• fix(hive-sync): call Driver.destroy() so HiveQL sync stops leaking Drivers into ShutdownHookManager by @skywalker0618 in https://github.com/apache/hudi/pull/19718
• fix(spark-sql): resolve a partition path without validating the record key by @linliu-code in https://github.com/apache/hudi/pull/19709
• fix(hive-sync): pass the default partition through the slash-encoded value extractors by @SEPURI-SAI-KRISHNA in https://github.com/apache/hudi/pull/19710
• docs(hudi-io): fill in the HFile format details the doc was missing by @rangareddy in https://github.com/apache/hudi/pull/19721
• fix(utilities): use endOffsets when no offset is greater than the checkpoint timestamp by @wangxianghu in https://github.com/apache/hudi/pull/19658
• fix(core): resolve rollback storage from the partition path, not the default URI by @vinishjail97 in https://github.com/apache/hudi/pull/19735
• fix(spark): make partition DDL commands honor slash separated date partitioning by @SEPURI-SAI-KRISHNA in https://github.com/apache/hudi/pull/19703
• fix: ensure CuratorFramework is always closed in BaseZookeeperBasedLo... by @mailtoboggavarapu-coder in https://github.com/apache/hudi/pull/19741
• fix: ensure AmazonDynamoDBLockClient is always closed in DynamoDBBasedLockProviderBase.close() by @mailtoboggavarapu-coder in https://github.com/apache/hudi/pull/19742
• fix(trino): skip predicate pushdown on type-evolved parquet columns by @wombatu-kun in https://github.com/apache/hudi/pull/19467
• test(io): drive the HFile writer tests through byte arrays instead of a file by @rangareddy in https://github.com/apache/hudi/pull/19618
• chore(docker): address PR [#18520] review comments for Spark 4.0.1 stack by @voonhous in https://github.com/apache/hudi/pull/18524
• fix(storage): resolve HoodieStorage from a path instead of a default local URI by @yihua in https://github.com/apache/hudi/pull/19737
• fix(streamer): route configured write table version into sample-writes flow by @lokeshj1703 in https://github.com/apache/hudi/pull/19746
• fix(common): escape dots in single-level hive-style partition column names by @lokeshj1703 in https://github.com/apache/hudi/pull/19751
• fix(client): create a new log file version instead of reopening the latest on pre-v8 writes by @lokeshj1703 in https://github.com/apache/hudi/pull/19755
• fix(timeline-service): support response charset for Timeline Server interaction by @lokeshj1703 in https://github.com/apache/hudi/pull/19753
• fix(debezium): backfill Postgres TOAST columns on a copy so the merge result survives by @lokeshj1703 in https://github.com/apache/hudi/pull/19749
• fix(trino): read LSM archived-timeline parquet files through a Trino-... by @voonhous in https://github.com/apache/hudi/pull/19732
• fix(hive): read the skeleton file when a bootstrap query projects no columns by @rangareddy in https://github.com/apache/hudi/pull/19510
• fix(streamer): Include start commit in S3/GCS IncrSource incremental query by @yihua in https://github.com/apache/hudi/pull/18949
• fix(sync): keep each partition's recorded location when cascading Glue schema changes by @niranjan-1408 in https://github.com/apache/hudi/pull/19761
• fix(sync): drop the duplicate UpdateTable in updateTableSchema by @niranjan-1408 in https://github.com/apache/hudi/pull/19762
• fix(utilities): close JDBC connection in UtilHelpers.getJDBCSchema to prevent connection leak by @mailtoboggavarapu-coder in https://github.com/apache/hudi/pull/19772
• fix(lock): name the cause on FAILED_TO_RELEASE in StorageBasedLockProvider by @pkgajulapalli in https://github.com/apache/hudi/pull/19574
• docs(hudi-notebooks): note S3A works with any S3-compatible object store by @goanpeca in https://github.com/apache/hudi/pull/19423
• fix(flink): run mapGroupsByKey in dedicated fork join pool by @cshuo in https://github.com/apache/hudi/pull/19819
• perf(flink): preempt inactive write buckets on memory exhaustion by @fhan688 in https://github.com/apache/hudi/pull/19728
• feat: Allow disabling the partition stats index independently of column stats by @vinishjail97 in https://github.com/apache/hudi/pull/19111
• test(spark): add catalog and V2 table DDL coverage by @yihua in https://github.com/apache/hudi/pull/19162
• fix(common): prevent multi-table commits failing with metrics enabled by @1fanwang in https://github.com/apache/hudi/pull/19787
• fix(core): record the ordering field when upgrading a table from version 1 by @yihua in https://github.com/apache/hudi/pull/19803
• fix(spark-sql): resolve MERGE INTO partition columns so records are not mis-partitioned by @linliu-code in https://github.com/apache/hudi/pull/19610
• fix(build): exclude bundled Avro from the DataHub sync bundle by @yihua in https://github.com/apache/hudi/pull/19814
• fix(tests): reach TFramedTransport at the package libthrift 0.14 moved it to by @yihua in https://github.com/apache/hudi/pull/19830
• refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 by @rangareddy in https://github.com/apache/hudi/pull/19815
• fix(hive-sync): close the SessionState HiveQL sync starts by @skywalker0618 in https://github.com/apache/hudi/pull/19812
• fix(trino): keep none() when splitting predicates by @voonhous in https://github.com/apache/hudi/pull/19863
• fix: close InflaterInputStream in HoodieAvroDataBlock.decompress() to prevent native memory leak by @mailtoboggavarapu-coder in https://github.com/apache/hudi/pull/19861
• fix(common): keep committed logs visible under an uncommitted base instant by @zhaoyudi-creator in https://github.com/apache/hudi/pull/19785
• test(spark): reference disjoint blob ranges in Lance out-of-line test by @voonhous in https://github.com/apache/hudi/pull/19916
• fix(spark): merge identical blob descriptors in the batched read by @voonhous in https://github.com/apache/hudi/pull/19913
• fix: close the outer file stream of inline reads by @voonhous in https://github.com/apache/hudi/pull/19936
• fix: close the filesystem's stream on rewrap by @voonhous in https://github.com/apache/hudi/pull/19935
• fix(spark): keep the embedded Derby metastore out of the RAT check by @voonhous in https://github.com/apache/hudi/pull/19792
• feat(metadata): support RLI and SI for parquet files written outside Hudi by @vinishjail97 in https://github.com/apache/hudi/pull/19869
• fix(core): keep plan generator extra metadata when scheduling compaction by @yihua in https://github.com/apache/hudi/pull/19941
• fix(docker): pull MinIO images from quay.io by @voonhous in https://github.com/apache/hudi/pull/19955
• fix(trino): satisfy the HOODIE_STORAGE_CLASS constructor contract by @OjashKush in https://github.com/apache/hudi/pull/19882
• chore(trino): report dependency drift against the plugin by @voonhous in https://github.com/apache/hudi/pull/19959
• test(trino): build the E2E server image from the pin by @voonhous in https://github.com/apache/hudi/pull/19953
• fix(build): exclude nested .mvn directories from the RAT check by @wombatu-kun in https://github.com/apache/hudi/pull/19462
• fix(common): resolve nested Avro records by class built by Hudi by @voonhous in https://github.com/apache/hudi/pull/19956
• perf(common): cache nested Avro record class lookup by @voonhous in https://github.com/apache/hudi/pull/19972
• fix(trino): unbind HudiCacheKeyProvider on 1.2.1 by @voonhous in https://github.com/apache/hudi/pull/19985

New Contributors

• @kartikeyaagrawal made their first contribution in https://github.com/apache/hudi/pull/18542
• @ashokkumar-allu made their first contribution in https://github.com/apache/hudi/pull/18584
• @mahsoodebrahim made their first contribution in https://github.com/apache/hudi/pull/18696
• @ericyuan915 made their first contribution in https://github.com/apache/hudi/pull/18982
• @oglego made their first contribution in https://github.com/apache/hudi/pull/18693
• @vamshipasunuru1 made their first contribution in https://github.com/apache/hudi/pull/18887
• @niks002 made their first contribution in https://github.com/apache/hudi/pull/19328
• @201573 made their first contribution in https://github.com/apache/hudi/pull/18687
• @suryadanny made their first contribution in https://github.com/apache/hudi/pull/19383
• @adityaanikam made their first contribution in https://github.com/apache/hudi/pull/19653
• @SEPURI-SAI-KRISHNA made their first contribution in https://github.com/apache/hudi/pull/19646
• @niranjan-1408 made their first contribution in https://github.com/apache/hudi/pull/19761
• @goanpeca made their first contribution in https://github.com/apache/hudi/pull/19423
• @1fanwang made their first contribution in https://github.com/apache/hudi/pull/19787
• @zhaoyudi-creator made their first contribution in https://github.com/apache/hudi/pull/19785
• @OjashKush made their first contribution in https://github.com/apache/hudi/pull/19882

Full Changelog: https://github.com/apache/hudi/compare/release-1.2.0...release-1.2.1