Lance Files

What's Changed

Breaking Changes 🛠

• refactor!: vendor the tokenizer stack into lance by @Xuanwo in https://github.com/lance-format/lance/pull/6512
• perf!: run scheduler initialize eagerly in async read_tasks by @westonpace in https://github.com/lance-format/lance/pull/6710

New Features 🎉

• feat: support segmented inverted index build and search by @Xuanwo in https://github.com/lance-format/lance/pull/6305
• feat(dictionary-namespace): support table related operation by @zhangyue19921010 in https://github.com/lance-format/lance/pull/6308
• feat: clean up transaction files on failed commits by @wjones127 in https://github.com/lance-format/lance/pull/6319
• feat: add planned blob reads with source-level coalescing by @Xuanwo in https://github.com/lance-format/lance/pull/6352
• refactor: use exact base-scoped store bindings by @Xuanwo in https://github.com/lance-format/lance/pull/6422
• feat: wire batch_size_bytes to Python and public Rust API by @westonpace in https://github.com/lance-format/lance/pull/6428
• feat(vector): add partition search parallelism by @BubbleCal in https://github.com/lance-format/lance/pull/6475
• feat(index): support float16 and float64 in IVF_FLAT by @BubbleCal in https://github.com/lance-format/lance/pull/6476
• feat: batch chopping fallback for filtered read by @westonpace in https://github.com/lance-format/lance/pull/6482
• feat(java): add Dataset.sample() API by @beinan in https://github.com/lance-format/lance/pull/6500
• feat: add ANN proto codecs and extract table_identifier module by @LuQQiu in https://github.com/lance-format/lance/pull/6503
• feat: add configurable blob v2 pack file size by @hamersaw in https://github.com/lance-format/lance/pull/6508
• feat: expose has_stable_row_ids property on LanceDataset by @pengw0048 in https://github.com/lance-format/lance/pull/6531
• feat: expose base scoped store bindings to python by @zhangyue19921010 in https://github.com/lance-format/lance/pull/6547
• feat: support zonemap index segments by @beinan in https://github.com/lance-format/lance/pull/6593
• feat: update lance-namespace to 0.7.2 and align namespace declared table lifecycle by @jackye1995 in https://github.com/lance-format/lance/pull/6608
• feat: generalize dynamic object store credentials by @jackye1995 in https://github.com/lance-format/lance/pull/6609
• feat: add ANNIvfPartitionExecProto by @LuQQiu in https://github.com/lance-format/lance/pull/6612
• feat: add prefilter_type to ANNIvfSubIndexExecProto by @LuQQiu in https://github.com/lance-format/lance/pull/6613
• feat: replace Azure SDK and google-cloud-auth with direct reqwest for credential vending by @jackye1995 in https://github.com/lance-format/lance/pull/6617
• feat(io): bypass backpressure for io_buffer_size=0 and 2.0 indirect I/O by @westonpace in https://github.com/lance-format/lance/pull/6627

Bug Fixes 🐛

• fix: warn and clamp LANCE_INITIAL_UPLOAD_SIZE instead of panicking by @LuciferYang in https://github.com/lance-format/lance/pull/6389
• fix: keep delete-by-source fast path with scalar indexes by @Xuanwo in https://github.com/lance-format/lance/pull/6435
• fix: include column_metadatas and column_infos in CachedFileMetadata::DeepSizeOf by @jiaoew1991 in https://github.com/lance-format/lance/pull/6480
• fix(index): preserve fts prewarm position codec by @BubbleCal in https://github.com/lance-format/lance/pull/6485
• fix: handle FlatBin quantization in optimize_vector_indices_v2 by @jackye1995 in https://github.com/lance-format/lance/pull/6488
• fix: use logical OR instead of bitwise OR in conflict resolver by @dentiny in https://github.com/lance-format/lance/pull/6492
• fix: add dir_listing_to_manifest_migration_enabled flag to avoid extra object store calls by @jackye1995 in https://github.com/lance-format/lance/pull/6507
• fix: prevent arithmetic overflow in U64Segment encoding selection for sparse/extreme row id ranges by @ivscheianu in https://github.com/lance-format/lance/pull/6516
• fix: bump jieba-rs to 0.9.0 to fix build-no-lock CI by @westonpace in https://github.com/lance-format/lance/pull/6518
• fix: blob projection schema compatibility by @Xuanwo in https://github.com/lance-format/lance/pull/6521
• fix(namespace): serialize manifest mutations by @Xuanwo in https://github.com/lance-format/lance/pull/6525
• fix: missing bumpversion entry for lance-tokenizer by @Xuanwo in https://github.com/lance-format/lance/pull/6526
• fix: scale default memory pool size by partition count by @westonpace in https://github.com/lance-format/lance/pull/6562
• fix: apply fragment bitmap allow-list to index search results by @westonpace in https://github.com/lance-format/lance/pull/6563
• fix: hard cap batch size in merge_insert to prevent sort failures by @westonpace in https://github.com/lance-format/lance/pull/6564
• fix: index type try_from miss RTree and BLOOMFILTER by @wojiaodoubao in https://github.com/lance-format/lance/pull/6568
• fix(namespace): align error handling with namespace spec by @jackye1995 in https://github.com/lance-format/lance/pull/6575
• fix: reject Rewrite vs CreateIndex when FRI groups straddle bitmap by @wjones127 in https://github.com/lance-format/lance/pull/6610
• fix(json): detect float64-stored numbers in json type extraction by @dentiny in https://github.com/lance-format/lance/pull/6622
• fix: respect LANCE_DEFAULT_IO_BUFFER_SIZE if it has been set by @westonpace in https://github.com/lance-format/lance/pull/6636
• fix: make vector distance schema nullable by @BubbleCal in https://github.com/lance-format/lance/pull/6649

Documentation 📚

• docs: tighten python environment workflow guidance by @Xuanwo in https://github.com/lance-format/lance/pull/6520
• docs: fix broken intra-doc link in DatasetPreFilter by @LuciferYang in https://github.com/lance-format/lance/pull/6579
• docs: correct repetition level example in encoding docs by @BubbleCal in https://github.com/lance-format/lance/pull/6585

Performance Improvements 🚀

• perf: intern DataFile fields/column_indices to reduce manifest memory by @beinan in https://github.com/lance-format/lance/pull/6477
• perf: intern RowDatasetVersionMeta inline bytes to reduce manifest memory by @beinan in https://github.com/lance-format/lance/pull/6499
• perf: add SIMD-accelerated u8 dot product for SQ distance by @justinrmiller in https://github.com/lance-format/lance/pull/6506
• perf: add SIMD kernels for bf16 distance functions by @justinrmiller in https://github.com/lance-format/lance/pull/6510
• perf: submit I/O requests eagerly in FullZipScheduler by @hushengquan in https://github.com/lance-format/lance/pull/6513
• perf: add SIMD-accelerated u8 L2 and cosine distance kernels by @justinrmiller in https://github.com/lance-format/lance/pull/6517
• perf: speed up RaBitQ 4-bit LUT distance on ARM by 16x by @justinrmiller in https://github.com/lance-format/lance/pull/6537
• perf: add explicit SIMD types and distance kernels for f64 by @justinrmiller in https://github.com/lance-format/lance/pull/6540
• perf: don't spawn the scheduling on a separate thread for small reads by @westonpace in https://github.com/lance-format/lance/pull/6637
• perf: avoid materializing RoaringBitmap::full() in fragment allow-list by @wkalt in https://github.com/lance-format/lance/pull/6664
• perf: revert inline scheduling by @westonpace in https://github.com/lance-format/lance/pull/6709

Full Changelog: https://github.com/lance-format/lance/compare/release-root/6.0.0-beta.N...v6.0.0