Lance Files

What's Changed

Breaking Changes 🛠

• feat!: refactor Dataset#config api and expose it via pylance by @yanghua in https://github.com/lancedb/lance/pull/4041

New Features 🎉

• feat: add column projection to LanceFileReader by @wojiaodoubao in https://github.com/lancedb/lance/pull/4031
• feat: add consecutive batch shard sampler for pytorch by @Jay-ju in https://github.com/lancedb/lance/pull/3886
• feat: add Trino and PostgreSQL integrations by @jackye1995 in https://github.com/lancedb/lance/pull/4054
• feat: expose num_rows api for python LanceFileReader by @yanghua in https://github.com/lancedb/lance/pull/4053
• feat: add IVF_HNSW_FLAT index by @BubbleCal in https://github.com/lancedb/lance/pull/3671
• feat: introduce IndexFileVersion for Python API by @BubbleCal in https://github.com/lancedb/lance/pull/4040
• feat: add created_at for indices by @wjones127 in https://github.com/lancedb/lance/pull/4051
• feat: support large_string/large_binary in lance format v2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/3967
• feat: upgrade arrow object_store to 0.12.2 by @jackye1995 in https://github.com/lancedb/lance/pull/4080
• feat: add random take bench based on file reader api and fragment api by @wojiaodoubao in https://github.com/lancedb/lance/pull/4076
• feat: support IVF_SQ index by @BubbleCal in https://github.com/lancedb/lance/pull/4096
• feat: add ignore_missing flag for safe dataset deletion by @Jay-ju in https://github.com/lancedb/lance/pull/3667
• feat: add a toggle for automatic rank/world_size discovery by @Jay-ju in https://github.com/lancedb/lance/pull/3633
• feat: add N-Gram tokenizer for FTS index by @BubbleCal in https://github.com/lancedb/lance/pull/4102
• feat(java): support config operation for Dataset by @majin1102 in https://github.com/lancedb/lance/pull/4039
• feat(datagen): add random words generator by @Xuanwo in https://github.com/lancedb/lance/pull/4099
• feat: allow _rowid and _rowaddr in read_columns for dataset.add_columns / fragment.merge_columns by @westonpace in https://github.com/lancedb/lance/pull/4120
• feat(encoding): cache repetition index for FullZip encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4104

Bug Fixes 🐛

• fix: can't search on a *_SQ index with f64 vectors by @BubbleCal in https://github.com/lancedb/lance/pull/3672
• fix: torchdataset for lance does not support S3 by @Jay-ju in https://github.com/lancedb/lance/pull/4045
• fix: incorrect parameter order when initializing ShardedFixedBatchSam… by @ddupg in https://github.com/lancedb/lance/pull/4064
• fix: take benchmark can not read dataset by @yanghua in https://github.com/lancedb/lance/pull/4065
• fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4068
• fix: better errors on bad filters by @westonpace in https://github.com/lancedb/lance/pull/4073
• fix(doc): fix doc typo in Ray integration documentation for Lance Data Sink by @kilavvy in https://github.com/lancedb/lance/pull/3984
• fix: don't check vector indices when loading scalar index by @BubbleCal in https://github.com/lancedb/lance/pull/4077
• fix: handle the case of all partitions are empty by @BubbleCal in https://github.com/lancedb/lance/pull/4085
• fix: use std::fs::copy for local filesystem cross-filesystem copies by @wjones127 in https://github.com/lancedb/lance/pull/4088
• fix: typo in feature_flags.rs by @b4l in https://github.com/lancedb/lance/pull/4093
• fix: wrong cp command for java native lib name by @yanghua in https://github.com/lancedb/lance/pull/4105
• fix: make max_fragment_id optional to prevent fragment ID reuse by @wjones127 in https://github.com/lancedb/lance/pull/4083
• fix: panic when optimize empty PQ partition by @BubbleCal in https://github.com/lancedb/lance/pull/4108
• fix: panic when commit fragment with format v2.1 by @ddupg in https://github.com/lancedb/lance/pull/4122

Documentation 📚

• docs: fix write_dataset append mode obsolete description by @wojiaodoubao in https://github.com/lancedb/lance/pull/3866

Performance Improvements 🚀

• perf: fast upsert with no indices by @wjones127 in https://github.com/lancedb/lance/pull/3858
• perf: faster kmeans by @BubbleCal in https://github.com/lancedb/lance/pull/4070

Other Changes

• refactor: don't use schema dictionary unless using legacy storage by @westonpace in https://github.com/lancedb/lance/pull/4023
• refactor: move 2.0 code into its own module by @westonpace in https://github.com/lancedb/lance/pull/4046
• refactor: move delete implementation into it's own file by @wjones127 in https://github.com/lancedb/lance/pull/4074
• Revert "chore: update tantivy version to 0.24.1 (#4098)" by @Xuanwo in https://github.com/lancedb/lance/pull/4101
• refactor(datagen): add RandomWord and RandomSentence to replace RandomWords by @Xuanwo in https://github.com/lancedb/lance/pull/4130

New Contributors

• @wojiaodoubao made their first contribution in https://github.com/lancedb/lance/pull/4031
• @ddupg made their first contribution in https://github.com/lancedb/lance/pull/4064
• @bjurkovski made their first contribution in https://github.com/lancedb/lance/pull/4068
• @kilavvy made their first contribution in https://github.com/lancedb/lance/pull/3984
• @b4l made their first contribution in https://github.com/lancedb/lance/pull/4093
• @Dig-Doug made their first contribution in https://github.com/lancedb/lance/pull/4098
• @xhwhis made their first contribution in https://github.com/lancedb/lance/pull/4106

Full Changelog: https://github.com/lancedb/lance/compare/v0.30.0...v0.31.0