Lance Files

What's Changed

Breaking Changes 🛠

• perf!: improve conflict resolution performance by @wjones127 in https://github.com/lancedb/lance/pull/3882
• feat!: introduce fragment reuse index to defer compaction index remap by @jackye1995 in https://github.com/lancedb/lance/pull/3847
• feat!: add minimum probes and maximum probes to IVF search by @westonpace in https://github.com/lancedb/lance/pull/3903

New Features 🎉

• feat: support slop param for phrase query by @BubbleCal in https://github.com/lancedb/lance/pull/3859
• feat: enhance NGram index support for large strings by @BubbleCal in https://github.com/lancedb/lance/pull/3893
• feat: support boolean query by @BubbleCal in https://github.com/lancedb/lance/pull/3879
• feat: enhancing java LanceFileReader by adding projection push down and ranges lookup functionality by @leaves12138 in https://github.com/lancedb/lance/pull/3888
• feat: support lance rust log in jni by @frankliee in https://github.com/lancedb/lance/pull/3901
• feat: add storage options for LanceFileWriter by @leaves12138 in https://github.com/lancedb/lance/pull/3900
• feat: expose auto cleanup config option for pylance by @yanghua in https://github.com/lancedb/lance/pull/3857
• feat: update to the latest versions of arrow and datafusion, remove pin on chrono by @westonpace in https://github.com/lancedb/lance/pull/3890
• feat: remap an index with fragment reuse index by @jackye1995 in https://github.com/lancedb/lance/pull/3906
• feat: ignore indices created in newer format by @BubbleCal in https://github.com/lancedb/lance/pull/3867
• feat: trim fragment reuse index after remapping by @jackye1995 in https://github.com/lancedb/lance/pull/3911
• feat: add load_index_by_name to reteive a unique index by name by @jackye1995 in https://github.com/lancedb/lance/pull/3931

Bug Fixes 🐛

• fix: remove modified fragments indices during horizontal merge_insert by @westonpace in https://github.com/lancedb/lance/pull/3863
• fix: last item could be dropped from directory listing by @jackye1995 in https://github.com/lancedb/lance/pull/3892
• fix: handle caching with manifest naming scheme migration by @wjones127 in https://github.com/lancedb/lance/pull/3896
• fix: deserialize PhraseQuery without slop field by @wjones127 in https://github.com/lancedb/lance/pull/3913
• fix: PrimitiveStructualEncoder extract_validity buffer should follow flag keep_original_array by @leaves12138 in https://github.com/lancedb/lance/pull/3899
• fix: unexpected posting block size when remap posting list by @BubbleCal in https://github.com/lancedb/lance/pull/3921
• fix: some compile error for index by @Jay-ju in https://github.com/lancedb/lance/pull/3920
• fix: skip non-leaf fields when projecting by column names in 2.1 by @westonpace in https://github.com/lancedb/lance/pull/3918
• fix: prevent and handle empty fragments by @wjones127 in https://github.com/lancedb/lance/pull/3934

Documentation 📚

• docs: fix some grammatical errors for operations comments by @majin1102 in https://github.com/lancedb/lance/pull/3930
• docs: add field encoding spec by @Jay-ju in https://github.com/lancedb/lance/pull/3820
• docs: add ray distributed merge columns by @Jay-ju in https://github.com/lancedb/lance/pull/3919

Performance Improvements 🚀

• perf: remove the unnecessary copying while indexing PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3894

New Contributors

• @leaves12138 made their first contribution in https://github.com/lancedb/lance/pull/3888

Full Changelog: https://github.com/lancedb/lance/compare/v0.28.0...v0.29.0