Lance Files

What's Changed

Critical Fixes ‼️

• fix(encoding): bounds-check the length prefix in VariableFullZipDecoder::unzip by @professor-moody in https://github.com/lance-format/lance/pull/8138
• fix(encoding): reject out-of-bounds variable-width offsets on decode by @Xuanwo in https://github.com/lance-format/lance/pull/8144

Both fixes were backported to release/v7.1 in #8156.

New Features 🎉

• feat: clean up data files on failed writes by @wjones127 in https://github.com/lance-format/lance/pull/6320
• feat: query with scalar index support fast search ability by @zhangyue19921010 in https://github.com/lance-format/lance/pull/6784
• feat: support batch vector queries by @LeoReeYang in https://github.com/lance-format/lance/pull/6828
• feat: expose index segments in Python scanner by @hfutatzhanghb in https://github.com/lance-format/lance/pull/6872
• feat(namespace-dir): implement table tag operations by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/6886
• fix(mem_wal): suppress stale LSM vector-search reads via block-list post-filter by @hamersaw in https://github.com/lance-format/lance/pull/6899
• feat(rust): support datafusion expressions for merge insert predicates by @Ar-maan05 in https://github.com/lance-format/lance/pull/6934
• feat(mem_wal): local-scoring FTS search for LSM scanner by @touch-of-grey in https://github.com/lance-format/lance/pull/6951
• feat: add ICU FTS tokenizer by @Xuanwo in https://github.com/lance-format/lance/pull/6956
• feat: support update by _row_id for python and java by @yanghua in https://github.com/lance-format/lance/pull/6976

Bug Fixes 🐛

• fix(docs): update java doc url from com.lancedb to org.lance by @HuaHuaY in https://github.com/lance-format/lance/pull/6467
• fix: use relaxed recall assertion for HNSW-based indexes by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/6620
• fix(mem_wal): build secondary indexes when flushing active memtable by @hamersaw in https://github.com/lance-format/lance/pull/6901
• fix(index): skip scalar index optimize when no unindexed fragments by @zhangyue19921010 in https://github.com/lance-format/lance/pull/6905
• fix(mem_wal): eliminate fresh-tier stale reads (dedup-on-scan + deletion-vector-on-flush) by @hamersaw in https://github.com/lance-format/lance/pull/6929
• fix: coerce filter expressions before simplifying by @ddupg in https://github.com/lance-format/lance/pull/6935
• fix: apply filtered read limit after refine filter by @Xuanwo in https://github.com/lance-format/lance/pull/6937
• fix: increase nprobes in test_query_delta_indices for deterministic recall by @Jay-ju in https://github.com/lance-format/lance/pull/6953
• fix(index): clean up IVF shuffler temp dir after index build by @geserdugarov in https://github.com/lance-format/lance/pull/6957
• fix(java): rename DatasetDelta JNI bindings to match Java declarations by @LuciferYang in https://github.com/lance-format/lance/pull/6963
• fix: apply fragment-bitmap allow-list to stable-row-id deletion mask by @ragnorc in https://github.com/lance-format/lance/pull/6965
• fix(encoding): handle u16 num_levels overflow in miniblock List encoding by @touch-of-grey in https://github.com/lance-format/lance/pull/6989

Documentation 📚

• docs: add new index page for integrations and fix broken build by @prrao87 in https://github.com/lance-format/lance/pull/6915
• docs: fix preview pylance install instructions by @prrao87 in https://github.com/lance-format/lance/pull/6931

Performance Improvements 🚀

• perf: use roaring's range iter to speedup mask_to_offset_ranges by @westonpace in https://github.com/lance-format/lance/pull/6871
• perf(fts): lazy-load PostingListReader metadata by @wkalt in https://github.com/lance-format/lance/pull/6873
• perf: fix very long row count times in ivf sampling by @cmccabe in https://github.com/lance-format/lance/pull/6930
• perf: avoid extra stat for ranged OpenDAL reads by @Xuanwo in https://github.com/lance-format/lance/pull/6942

Other Changes

• refactor(index): use arrow stats for scalar index stats by @HaochengLIU in https://github.com/lance-format/lance/pull/6808
• refactor: move mask code into new crate by @westonpace in https://github.com/lance-format/lance/pull/6879
• refactor(bench): replace mem_wal_read/vector with focused LSM benchmarks by @touch-of-grey in https://github.com/lance-format/lance/pull/6882
• chore: improve lance maintenance path by @vip892766gma in https://github.com/lance-format/lance/pull/6943

Full Changelog: https://github.com/lance-format/lance/compare/release-root/7.1.0-beta.N...v7.1.0