Lance Files

What's Changed

New Features 🎉

• feat: use mkdocs for documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4132
• docs: add catalog integration doc section pointing to Lance Namespace by @jackye1995 in https://github.com/lancedb/lance/pull/4148
• docs: add DataFusion and DuckDB documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4149
• feat: expose max_page_bytes arg for python LanceFileWriter by @wojiaodoubao in https://github.com/lancedb/lance/pull/4131
• docs: add lancelot page by @jackye1995 in https://github.com/lancedb/lance/pull/4147
• feat: add utility to capture trace events in python by @westonpace in https://github.com/lancedb/lance/pull/4123
• feat: introduce MemWAL index and system index by @jackye1995 in https://github.com/lancedb/lance/pull/4069

Bug Fixes 🐛

• fix: failed to write FTS index if the entire posting list is deleted by @BubbleCal in https://github.com/lancedb/lance/pull/4156
• fix: enter span in traced object store only when polled by @wkalt in https://github.com/lancedb/lance/pull/4154
• fix: shuffler fails to infer schema if the first batch is empty by @BubbleCal in https://github.com/lancedb/lance/pull/4173
• fix: remapping FTS missed to remap token set by @BubbleCal in https://github.com/lancedb/lance/pull/4180
• fix: update lindera and tantivy and replace lindera model by @KazuhitoT in https://github.com/lancedb/lance/pull/4144
• fix: don't panic in byte_width_opt if given a fixed-size-list of a variable-width type by @westonpace in https://github.com/lancedb/lance/pull/4186

Documentation 📚

• docs: ulimit to increase number of open files on OS X by @jbapple in https://github.com/lancedb/lance/pull/4187

Performance Improvements 🚀

• perf: don't persist intermediate results while indexing vectors by @BubbleCal in https://github.com/lancedb/lance/pull/4004
• perf: use HNSW to speed up compute_partitions by @BubbleCal in https://github.com/lancedb/lance/pull/4089
• perf: use size prefixing for zstd buffer compressor for better decompressing performance by @niyue in https://github.com/lancedb/lance/pull/4029

Other Changes

• refactor: refactor full zip schedule_ranges_rep by @Xuanwo in https://github.com/lancedb/lance/pull/4142
• refactor: rename UploadState state change function by @allenanswerzq in https://github.com/lancedb/lance/pull/4170
• refactor: implement type-safe cache interface with CacheKey trait by @wjones127 in https://github.com/lancedb/lance/pull/4075

New Contributors

• @allenanswerzq made their first contribution in https://github.com/lancedb/lance/pull/4170
• @jbapple made their first contribution in https://github.com/lancedb/lance/pull/4187

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.0...v0.31.1