Lance Files

What's Changed

Breaking Changes 🛠

• feat!: auto-remap indexes before scan by @jackye1995 in https://github.com/lancedb/lance/pull/3971
• feat!: move file metadata cache to bytes capacity by @wjones127 in https://github.com/lancedb/lance/pull/3949

New Features 🎉

• feat(java): support checkout version by @yanghua in https://github.com/lancedb/lance/pull/3907
• feat: wrap FilterExec in LanceFilterExec by @wkalt in https://github.com/lancedb/lance/pull/3939
• feat: add storage options for LanceFileReader by @leaves12138 in https://github.com/lancedb/lance/pull/3941
• feat: prefix matching for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3935
• feat(java): support list versions and checkout version in Dataset by @majin1102 in https://github.com/lancedb/lance/pull/3945
• feat: add a few lance examples in Rust by @HaochengLIU in https://github.com/lancedb/lance/pull/3817
• feat: rebase conflict between rewrite and frag reuse index cleanup by @jackye1995 in https://github.com/lancedb/lance/pull/3952
• feat(java): support tag operations on Java dataset by @majin1102 in https://github.com/lancedb/lance/pull/3958
• feat(java): support restoring to the checked out version by @majin1102 in https://github.com/lancedb/lance/pull/3969
• feat: expose update_config api for python by @yanghua in https://github.com/lancedb/lance/pull/3963
• feat: support returning tags in order of version with descending order as default by @Jay-ju in https://github.com/lancedb/lance/pull/4001
• feat: add support for must_not clause in Boolean queries by @BubbleCal in https://github.com/lancedb/lance/pull/4012
• feat: support unenforced primary key concept in schema by @jackye1995 in https://github.com/lancedb/lance/pull/4002
• feat: redirect Spark documentation to new lance-spark website by @jackye1995 in https://github.com/lancedb/lance/pull/4032
• feat: support enable/disable auto-cleanup by @yanghua in https://github.com/lancedb/lance/pull/4037
• feat: add cache stats for new cache by @wjones127 in https://github.com/lancedb/lance/pull/4024

Bug Fixes 🐛

• fix: correctly specify schema in encoded substrait for expr by @westonpace in https://github.com/lancedb/lance/pull/3937
• fix: newer indices may be read from cache by @BubbleCal in https://github.com/lancedb/lance/pull/3936
• fix: null value rows missing after bitmap remap by @jackye1995 in https://github.com/lancedb/lance/pull/3956
• fix: rebase transactions from oldest to latest by @jackye1995 in https://github.com/lancedb/lance/pull/3955
• fix: typos in documentation comments by @leopardracer in https://github.com/lancedb/lance/pull/3970
• fix: call remove_outer_validity() before dictionary_encode by @luohao in https://github.com/lancedb/lance/pull/3974
• fix: don't crash when reading index trained on old version by @westonpace in https://github.com/lancedb/lance/pull/3978
• fix: handle case in 2.1 writer where lists are masked by null struct by @westonpace in https://github.com/lancedb/lance/pull/3946
• fix: don't panic on DF table provider if test has filter on indexed column by @westonpace in https://github.com/lancedb/lance/pull/3954
• fix: respect precomputed partitions and shuffle buffers for v3 index by @BubbleCal in https://github.com/lancedb/lance/pull/3910
• fix: panic caused by reading empty posting list by @BubbleCal in https://github.com/lancedb/lance/pull/4019
• fix: optimize index with no new data by @wjones127 in https://github.com/lancedb/lance/pull/4017
• fix: filter out null row ids by @BubbleCal in https://github.com/lancedb/lance/pull/4020
• fix: separate loading configuration handling for jieba and lindera by @KazuhitoT in https://github.com/lancedb/lance/pull/3932
• fix: reading large fragment reuse index details fails with file not found by @jackye1995 in https://github.com/lancedb/lance/pull/4035
• fix: use i32 for dictionary encoding indices by @luohao in https://github.com/lancedb/lance/pull/3898
• fix: optimizing FTS causes error if it's without positions by @BubbleCal in https://github.com/lancedb/lance/pull/4044

Other Changes

• refactor: change the default process-wide IOPS limit to be 128 by @westonpace in https://github.com/lancedb/lance/pull/3972

New Contributors

• @Xuanwo made their first contribution in https://github.com/lancedb/lance/pull/3960
• @leopardracer made their first contribution in https://github.com/lancedb/lance/pull/3970

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.0...v0.30.0