Lance Files

What's Changed

Breaking Changes 🛠

• feat!: use FilteredReadExec in the planner by @westonpace in https://github.com/lancedb/lance/pull/3813
• feat!: consolidated index cache by @wjones127 in https://github.com/lancedb/lance/pull/4047

New Features 🎉

• feat: allow users to control the full zip rep index cache by @Xuanwo in https://github.com/lancedb/lance/pull/4157
• feat: add rle encoding support by @Xuanwo in https://github.com/lancedb/lance/pull/4177
• feat: support inferring system index type by @jackye1995 in https://github.com/lancedb/lance/pull/4210
• feat(rust): support trace dataset events by @yanghua in https://github.com/lancedb/lance/pull/4145
• feat: support prewarm for btree by @BubbleCal in https://github.com/lancedb/lance/pull/4235
• feat(java): support get lance schema with field ids by @majin1102 in https://github.com/lancedb/lance/pull/4207
• feat: support sql api for dataset by @yanghua in https://github.com/lancedb/lance/pull/4086
• feat: add native oss support for lance by @Xuanwo in https://github.com/lancedb/lance/pull/4247
• feat: enable oss support in python and java by @Xuanwo in https://github.com/lancedb/lance/pull/4254
• refactor: rewrite delete implementation to use dataset-level scanner (#4158) by @wjones127 in https://github.com/lancedb/lance/pull/4165
• feat: add retry-based conflict resolution for update operations by @wjones127 in https://github.com/lancedb/lance/pull/4167
• feat: add 64-bit offset support to FSST compression for large binary/string data in V2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/4092
• feat: extend merge_insert fast path for UpdateIf and update-only operations by @wjones127 in https://github.com/lancedb/lance/pull/4253
• feat: expose sql api for python dataset by @Jay-ju in https://github.com/lancedb/lance/pull/4267
• feat(java): support replace schema and field metadata by @majin1102 in https://github.com/lancedb/lance/pull/4119
• feat: add claude md for future agentic work by @Xuanwo in https://github.com/lancedb/lance/pull/4245
• feat: implement byte stream split for doubles by @Xuanwo in https://github.com/lancedb/lance/pull/4236
• feat: support ordering for pylance sdk by @SaintBacchus in https://github.com/lancedb/lance/pull/3556
• feat: support skipping auto cleanup during write by @jackye1995 in https://github.com/lancedb/lance/pull/4293
• feat: expose IVF centroids to the python API by @Sbargaoui in https://github.com/lancedb/lance/pull/4288

Bug Fixes 🐛

• fix: forward compatibility of pylance load_indices by @jackye1995 in https://github.com/lancedb/lance/pull/4208
• fix: contributing link fixed on README.md by @lalitx17 in https://github.com/lancedb/lance/pull/4211
• fix: move BYTES_READ_COUNTER update after await to ensure accuracy by @allenanswerzq in https://github.com/lancedb/lance/pull/4203
• fix: broken image link in README by @LeoReeYang in https://github.com/lancedb/lance/pull/4228
• fix: normalize_fsl should handle nulls by @Xuanwo in https://github.com/lancedb/lance/pull/4226
• fix: phrase query may miss some results by @BubbleCal in https://github.com/lancedb/lance/pull/4223
• fix: handle index statistics for system indexes by @jackye1995 in https://github.com/lancedb/lance/pull/4229
• fix: btree may load the same page multiple times by @BubbleCal in https://github.com/lancedb/lance/pull/4246
• fix: allow rowid in columns for scanner by @Jay-ju in https://github.com/lancedb/lance/pull/4116
• fix: fix errors caused by simultaneous merge by @westonpace in https://github.com/lancedb/lance/pull/4279
• fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4284
• fix: proper case-insensitive comparison for filter by @wjones127 in https://github.com/lancedb/lance/pull/4278
• fix: ensure _rowid comes after _distance and _score by @westonpace in https://github.com/lancedb/lance/pull/4294

Documentation 📚

• docs: optional values are "present" or "absent" by @jbapple in https://github.com/lancedb/lance/pull/4184
• docs: separate Quickstart into three tutorials by @davidmyriel in https://github.com/lancedb/lance/pull/4200
• docs: add complete blob writing example and fix take_blobs usage by @TaoKevinKK in https://github.com/lancedb/lance/pull/4214
• docs: add new homepage and sidebar by @davidmyriel in https://github.com/lancedb/lance/pull/4231
• docs: separate table and file format spec by @jackye1995 in https://github.com/lancedb/lance/pull/4264

Performance Improvements 🚀

• perf: improve btree performance for range query by @BubbleCal in https://github.com/lancedb/lance/pull/4248
• perf: wrap Dataset with Arc in SqlQueryBuilder by @wojiaodoubao in https://github.com/lancedb/lance/pull/4275

Other Changes

• refactor: unify two branches of add_offsets to make it more clear by @allenanswerzq in https://github.com/lancedb/lance/pull/4225
• refactor: avoid fri abbrevation by @jackye1995 in https://github.com/lancedb/lance/pull/4285

New Contributors

• @lalitx17 made their first contribution in https://github.com/lancedb/lance/pull/4211
• @davidmyriel made their first contribution in https://github.com/lancedb/lance/pull/4200
• @TaoKevinKK made their first contribution in https://github.com/lancedb/lance/pull/4214
• @Sbargaoui made their first contribution in https://github.com/lancedb/lance/pull/4288

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.1...v0.32.0