| Name | Modified | Size | Downloads / Week |
|---|---|---|---|
| Parent folder | |||
| README.md | 2026-03-13 | 20.5 kB | |
| v3.0.0 source code.tar.gz | 2026-03-13 | 15.0 MB | |
| v3.0.0 source code.zip | 2026-03-13 | 15.9 MB | |
| Totals: 3 Items | 30.9 MB | 0 | |
What's Changed
Breaking Changes π
- feat!: support index progress reporting via callbacks by @wkalt in https://github.com/lance-format/lance/pull/5910
- perf!: remove shuffle buffer by @wkalt in https://github.com/lance-format/lance/pull/5912
- feat!: upgrade DataFusion dependency to 52.1.0 by @wjones127 in https://github.com/lance-format/lance/pull/6015
- refactor!: refactor java access to file format version by @jackye1995 in https://github.com/lance-format/lance/pull/6053
- refactor!: remove create_empty_table usage by @jackye1995 in https://github.com/lance-format/lance/pull/6087
- fix!: bump IVF_RQ version for compatibility check by @BubbleCal in https://github.com/lance-format/lance/pull/6097
Critical Fixes βΌοΈ
- fix: deduplicate row addresses in take to prevent panic by @wjones127 in https://github.com/lance-format/lance/pull/5881
- fix: fts flat search drops rows when avg_doc_length < 1.0 by @wjones127 in https://github.com/lance-format/lance/pull/5897
- fix: invalidate index fragment bitmaps after data replacement and stale merge by @wjones127 in https://github.com/lance-format/lance/pull/5929
New Features π
- feat: add RLE support for block by @yingjianwu98 in https://github.com/lance-format/lance/pull/4937
- feat: compress complex all null by @yingjianwu98 in https://github.com/lance-format/lance/pull/4990
- feat: support cleanup across branches by @majin1102 in https://github.com/lance-format/lance/pull/5009
- feat: dictionary index always32 bits by @yingjianwu98 in https://github.com/lance-format/lance/pull/5011
- feat: abort dictionary encode if not useful by @yingjianwu98 in https://github.com/lance-format/lance/pull/5055
- feat(cdf): cdf support upsert for views by @zhangyue19921010 in https://github.com/lance-format/lance/pull/5369
- feat(compaction): binary copy capability for compaction by @zhangyue19921010 in https://github.com/lance-format/lance/pull/5434
- feat: expose
use_scalar_indexparam in Java scanner by @xloya in https://github.com/lance-format/lance/pull/5487 - feat(python): expose search_filter in scanner by @wojiaodoubao in https://github.com/lance-format/lance/pull/5506
- feat: add alter column nullable to non-nullable support by @Xuanwo in https://github.com/lance-format/lance/pull/5589
- feat: evolute all_null_layout to constant layout by @Xuanwo in https://github.com/lance-format/lance/pull/5641
- feat(java): support creating IVF_RQ index by @majin1102 in https://github.com/lance-format/lance/pull/5648
- feat(java): support building vector index distributively by @majin1102 in https://github.com/lance-format/lance/pull/5664
- feat(rust): add datafusion catalog_provider through namespace by @majin1102 in https://github.com/lance-format/lance/pull/5686
- feat: support List and Struct type for KeyValue in inserted_rows.rs by @wojiaodoubao in https://github.com/lance-format/lance/pull/5713
- feat: support tencent cos by @ztorchan in https://github.com/lance-format/lance/pull/5740
- feat: add Lance-HF docs to lance.org/integrations/huggingface/ by @prrao87 in https://github.com/lance-format/lance/pull/5748
- feat(python): support namespace for tensorflow by @yuqi1129 in https://github.com/lance-format/lance/pull/5750
- feat: add range to External blob by @wojiaodoubao in https://github.com/lance-format/lance/pull/5765
- feat(java): support json extraction by scanning by @majin1102 in https://github.com/lance-format/lance/pull/5770
- feat: introduce RowIdSet and RowIdMask by @yanghua in https://github.com/lance-format/lance/pull/5771
- feat: expose blob handling APIs to python by @Xuanwo in https://github.com/lance-format/lance/pull/5790
- feat: add blob handling support for fragment by @Xuanwo in https://github.com/lance-format/lance/pull/5801
- feat: add plan/execute separation to FilteredReadExec by @LuQQiu in https://github.com/lance-format/lance/pull/5843
- feat: add LSM scanner with point lookup and vector search support by @touch-of-grey in https://github.com/lance-format/lance/pull/5850
- feat: add rename table implementations to REST namespaces by @bryanck in https://github.com/lance-format/lance/pull/5874
- feat(python): expose enable_stable_row_ids in commit() by @fecet in https://github.com/lance-format/lance/pull/5908
- feat: support aggregate in scanner by @jackye1995 in https://github.com/lance-format/lance/pull/5911
- feat: spill page metadata to disk during IVF shuffle by @wkalt in https://github.com/lance-format/lance/pull/5921
- feat: add third party licenses lists by @jackye1995 in https://github.com/lance-format/lance/pull/5922
- feat(java): support session by @jackye1995 in https://github.com/lance-format/lance/pull/5931
- feat: make geodatafusion/geoarrow optional via
geofeature flag by @apoc in https://github.com/lance-format/lance/pull/5934 - perf: create local writer for efficient local writes by @wkalt in https://github.com/lance-format/lance/pull/5939
- feat: add python and java binding for aggregate by @jackye1995 in https://github.com/lance-format/lance/pull/5951
- feat: add proto serialization for FilteredReadExec by @LuQQiu in https://github.com/lance-format/lance/pull/5954
- feat: create an arrow-scalar crate utilizing arrow-row and arrow-data by @westonpace in https://github.com/lance-format/lance/pull/5955
- feat: add progress monitoring via callbacks for inverted indexes by @vivek-bharathan in https://github.com/lance-format/lance/pull/5958
- feat: add size to object store tracing by @wjones127 in https://github.com/lance-format/lance/pull/5962
- feat: update minimum supported rust version from 1.88 to 1.91 by @westonpace in https://github.com/lance-format/lance/pull/5964
- feat: add Dataset::with_object_store for request-scoped store overrides by @wkalt in https://github.com/lance-format/lance/pull/5966
- feat: support namespace as external manifest store by @jackye1995 in https://github.com/lance-format/lance/pull/5968
- feat: serialize storage options in table identifier proto by @LuQQiu in https://github.com/lance-format/lance/pull/5973
- feat(core): add Levenshtein-based suggestions to not-found errors in schema by @HemantSudarshan in https://github.com/lance-format/lance/pull/5976
- feat: add URI-based commit support to Java SDK by @hamersaw in https://github.com/lance-format/lance/pull/5978
- fix: concurrent read and write to directory namespace by @jackye1995 in https://github.com/lance-format/lance/pull/5983
- feat: add ability to pass custom headers to objectstore requests by @hamersaw in https://github.com/lance-format/lance/pull/5989
- feat: add DeleteResult with num_deleted_rows by @wkalt in https://github.com/lance-format/lance/pull/6001
- feat: introduce IncompatibleTransaction error by @wjones127 in https://github.com/lance-format/lance/pull/6003
- feat: surface ambiguous merge insert error as
InvalidInputby @wjones127 in https://github.com/lance-format/lance/pull/6048 - feat(blob): distribute blob sidecar keys with reversed binary ids by @Xuanwo in https://github.com/lance-format/lance/pull/6060
- feat: handle JSONB literals in Lance SQL planner by @wkalt in https://github.com/lance-format/lance/pull/6061
- feat(java): expose Dataset.dropIndex method to drop specific index by @fangbo in https://github.com/lance-format/lance/pull/6065
- feat(blob): map external blob URIs to multi-base base ids by @Xuanwo in https://github.com/lance-format/lance/pull/6066
- feat: add env toggle for repetition index cache on read by @Xuanwo in https://github.com/lance-format/lance/pull/6069
- feat(compaction): single reserve_fragment_ids after rewriting files by @hamersaw in https://github.com/lance-format/lance/pull/6072
- feat: expose compaction binary copy configuration through python and java SDKs by @hamersaw in https://github.com/lance-format/lance/pull/6074
- feat: mark 2.2 as stable and add 2.3 as the next file format version by @Xuanwo in https://github.com/lance-format/lance/pull/6088
- feat: support prewarm for IVF-based ANN indices by @wjones127 in https://github.com/lance-format/lance/pull/6090
Bug Fixes π
- fix: ensure blob encoding work when using file reader directly by @rahil-c in https://github.com/lance-format/lance/pull/5193
- fix: support system columns in dataset.take* operations by @hamersaw in https://github.com/lance-format/lance/pull/5722
- fix: skip missing indices in compaction rewrite by @AndreaBozzo in https://github.com/lance-format/lance/pull/5739
- fix(lance-linalg): check fp16kernels feature before arch-specific code by @durch in https://github.com/lance-format/lance/pull/5747
- refactor: align blob behavior that write via file format version, read via layout by @Xuanwo in https://github.com/lance-format/lance/pull/5752
- fix: fix deletion when using file-object-store:// by @cmccabe in https://github.com/lance-format/lance/pull/5760
- fix: remove unreasonable nullable check for data types in hash_joiner during merge operation by @zhangyue19921010 in https://github.com/lance-format/lance/pull/5784
- fix: allow unused_unsafe for __cpuid to support both stable and nightly by @jackye1995 in https://github.com/lance-format/lance/pull/5793
- fix: set JUnit dependency as test scope by @bryanck in https://github.com/lance-format/lance/pull/5815
- fix(java): transaction fatal bug in java transaction api by @wojiaodoubao in https://github.com/lance-format/lance/pull/5824
- fix: fix remap so that it handles deletions correctly by @westonpace in https://github.com/lance-format/lance/pull/5828
- fix: inconsistent transposed pq code and metadata when build ivf_pq index distributedly by @yanghua in https://github.com/lance-format/lance/pull/5834
- fix: improve error messages in FixedSizeListArrayExt::convert_to_floating_point by @LuciferYang in https://github.com/lance-format/lance/pull/5836
- fix(java): panic when reading CreateIndex transaction by @majin1102 in https://github.com/lance-format/lance/pull/5853
- fix: fix mini-block dictionary bitpacking panic by @Xuanwo in https://github.com/lance-format/lance/pull/5860
- fix: fix boolean inline constant decoding by @Xuanwo in https://github.com/lance-format/lance/pull/5862
- fix: handle NULL elements in LABEL_LIST index results and explain_plan by @fenfeng9 in https://github.com/lance-format/lance/pull/5867
- fix: open additional storage options provider related apis in lance dataset by @jackye1995 in https://github.com/lance-format/lance/pull/5869
- fix: flaky test test_ann_prefilter for HNSW by @BubbleCal in https://github.com/lance-format/lance/pull/5870
- fix(java): init allocator for new dataset when checkout branch/tag by @fangbo in https://github.com/lance-format/lance/pull/5876
- fix: avoid panic when repdef serializes empty offsets by @fenfeng9 in https://github.com/lance-format/lance/pull/5890
- fix(java): align version type from i32 to u64 by @majin1102 in https://github.com/lance-format/lance/pull/5892
- fix: avoid bitmap range panic on inverted bounds by @fenfeng9 in https://github.com/lance-format/lance/pull/5893
- fix: split index_statistics to reduce rustc query depth by @Xuanwo in https://github.com/lance-format/lance/pull/5894
- fix: avoid panic on empty list LABEL_LIST filters by @fenfeng9 in https://github.com/lance-format/lance/pull/5914
- fix: correct OR null semantics for nullable masks by @fenfeng9 in https://github.com/lance-format/lance/pull/5919
- fix: respect fragment restrictions in vector and FTS searches when requested fragments by @yingjianwu98 in https://github.com/lance-format/lance/pull/5924
- fix: don't drop field metadata on merge insert path by @westonpace in https://github.com/lance-format/lance/pull/5927
- fix: apply SchemaAdapter in Updater by @westonpace in https://github.com/lance-format/lance/pull/5928
- fix: correct OR null handling for BlockList|BlockList by @fenfeng9 in https://github.com/lance-format/lance/pull/5944
- fix: remove unnecessary column projection for count aggregate by @jackye1995 in https://github.com/lance-format/lance/pull/5950
- fix: respect requested indexed fragment in vector and FTS searches by @yingjianwu98 in https://github.com/lance-format/lance/pull/5953
- fix: spawn part load in fts training by @westonpace in https://github.com/lance-format/lance/pull/5977
- fix: rest namespace integration with table version apis by @jackye1995 in https://github.com/lance-format/lance/pull/5980
- fix(encoding): handle empty rows in variable packed struct decode by @Xuanwo in https://github.com/lance-format/lance/pull/5995
- fix: various bugs to namespace access by @jackye1995 in https://github.com/lance-format/lance/pull/5996
- fix: set namespace commit handler for LanceDataset.commit by @jackye1995 in https://github.com/lance-format/lance/pull/6002
- fix: fast_search limits full text search to indexed fragments by @BubbleCal in https://github.com/lance-format/lance/pull/6006
- fix: fast_search should ignore any unindexed data for vector search by @BubbleCal in https://github.com/lance-format/lance/pull/6007
- fix: correctly calculate max visible level when a list has no def by @westonpace in https://github.com/lance-format/lance/pull/6008
- perf: avoid oversized variable buffers in full-zip scan batches by @Xuanwo in https://github.com/lance-format/lance/pull/6013
- fix: make overwrites retryable instead of compatible by @jackye1995 in https://github.com/lance-format/lance/pull/6014
- fix(python): avoid interpreter shutdown panic in BackgroundExecutor by @Xuanwo in https://github.com/lance-format/lance/pull/6023
- fix(btree): include null pages in non-IsNull queries for correct thre⦠by @wkalt in https://github.com/lance-format/lance/pull/6043
- fix: allowing headers for static configuration to be consistent by @hamersaw in https://github.com/lance-format/lance/pull/6045
- fix(build): add Android aarch64 support to lance-linalg by @dardourimohamed in https://github.com/lance-format/lance/pull/6057
- fix: make blob v2 reads base-aware in multi-base datasets by @Xuanwo in https://github.com/lance-format/lance/pull/6064
- fix(lance-linalg): fix missing return value in u8x16::bit_and for non-x86_64/aarch64 targets by @cheungxi in https://github.com/lance-format/lance/pull/6068
- fix: resolve Python lint failure on main by @Xuanwo in https://github.com/lance-format/lance/pull/6073
- fix: restore main CI by formatting take_blob imports by @Xuanwo in https://github.com/lance-format/lance/pull/6082
- fix: incorrect deletion masking in
DatasetPreFilterby @cijiugechu in https://github.com/lance-format/lance/pull/6083 - fix: avoid thread pool contention between compression and write operations during FTS indexing by @BubbleCal in https://github.com/lance-format/lance/pull/6085
- fix: compile error for err_express by @zhangyue19921010 in https://github.com/lance-format/lance/pull/6094
Documentation π
- docs: add docs for branch by @majin1102 in https://github.com/lance-format/lance/pull/5104
- docs: fix issues in HF integration docs by @prrao87 in https://github.com/lance-format/lance/pull/5778
- docs: fix MkDocs protobuf reference for ConstantLayout by @Xuanwo in https://github.com/lance-format/lance/pull/5833
- docs(governance): introduce incubating subproject concept and update subproject list by @jackye1995 in https://github.com/lance-format/lance/pull/5847
- docs: add lance skills as user guide by @Xuanwo in https://github.com/lance-format/lance/pull/5877
- docs: add array type support by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/5884
- docs: add FTS docs by @prrao87 in https://github.com/lance-format/lance/pull/5888
- docs: add schema data types and field IDs documentation by @dik654 in https://github.com/lance-format/lance/pull/5925
- docs: add lance-trino doc by @jackye1995 in https://github.com/lance-format/lance/pull/5943
- docs: add new maintainers by @prrao87 in https://github.com/lance-format/lance/pull/5959
- docs: expand the FTS index doc explaining the training process and multiple partitions by @westonpace in https://github.com/lance-format/lance/pull/5988
- docs: clarify v2.2 nested drop rollback risk by @Xuanwo in https://github.com/lance-format/lance/pull/5999
- docs: require data_storage_version=2.2 in map type example by @Xuanwo in https://github.com/lance-format/lance/pull/6032
- docs: update file versioning matrix for 2.2 rollout by @Xuanwo in https://github.com/lance-format/lance/pull/6033
- docs: reorganize blob docs around blob v2 and clarify legacy compatibility by @Xuanwo in https://github.com/lance-format/lance/pull/6034
- docs: align 2.2 encoding docs and nested add-column notes by @Xuanwo in https://github.com/lance-format/lance/pull/6038
- docs: clarify how to generate TPCH benchmark dataset locally by @Xuanwo in https://github.com/lance-format/lance/pull/6063
Performance Improvements π
- perf: add vector throughput benchmark by @westonpace in https://github.com/lance-format/lance/pull/5644
- perf: add a full text search benchmark by @westonpace in https://github.com/lance-format/lance/pull/5665
- perf: don't concat the batches for writing posting lists by @BubbleCal in https://github.com/lance-format/lance/pull/5769
- perf: add a lightweight scheduler implementation by @westonpace in https://github.com/lance-format/lance/pull/5773
- perf: use cpu pool to process all posting lists by @BubbleCal in https://github.com/lance-format/lance/pull/5780
- perf: calculate cardinality lazily by @Xuanwo in https://github.com/lance-format/lance/pull/5783
- perf: replace flatmap in build_distance_table by @wkalt in https://github.com/lance-format/lance/pull/5898
- perf: change Dataset::sample to sort its random indices by @wkalt in https://github.com/lance-format/lance/pull/5915
- perf: build fp16kernels with NEON support on iOS by @chyyran in https://github.com/lance-format/lance/pull/5917
- perf: reduce peak memory in nullable training data sampling by @wkalt in https://github.com/lance-format/lance/pull/5935
- perf: upgrade roaring to 0.11 and improve bitmap/range conversions by @LuQQiu in https://github.com/lance-format/lance/pull/5961
- perf: speed up format v2.2 scans by adding shortcut for full page by @Xuanwo in https://github.com/lance-format/lance/pull/5981
- perf: speed up format 2.2 300% by spawning structural decode batch tasks by @Xuanwo in https://github.com/lance-format/lance/pull/5982
- perf: improve parallelism of data_stats by @wkalt in https://github.com/lance-format/lance/pull/5990
- perf: reduce peak memory during cosine IVF-PQ index training by @wkalt in https://github.com/lance-format/lance/pull/6016
- perf: fast rotation for RQ quantization by @BubbleCal in https://github.com/lance-format/lance/pull/6024
- perf: avoid re-open shard indices and small reads by @BubbleCal in https://github.com/lance-format/lance/pull/6026
- perf: disable auto FSST for binary fields by @Xuanwo in https://github.com/lance-format/lance/pull/6047
- perf: speedup flat fts by @westonpace in https://github.com/lance-format/lance/pull/6054
- perf: add dict-values compression controls with lz4 default by @Xuanwo in https://github.com/lance-format/lance/pull/6059
- perf: avoid frequent allocating when computing residual vectors by @BubbleCal in https://github.com/lance-format/lance/pull/6062
- perf: add take_blob benchmark with cache_repetition_index matrix by @Xuanwo in https://github.com/lance-format/lance/pull/6067
Other Changes
- refactor: change reader's get_range result to be a static future by @westonpace in https://github.com/lance-format/lance/pull/5755
- refactor(python): migrate torch.jit.script to torch.compile by @wjones127 in https://github.com/lance-format/lance/pull/5759
- test: fix tests broken by pandas 3 release by @westonpace in https://github.com/lance-format/lance/pull/5786
- refactor: use dict entries and encoded size instead of cardinality for dict decision by @Xuanwo in https://github.com/lance-format/lance/pull/5891
- refactor: correct panic message typos in OrderableScalarValue::cmp by @fenfeng9 in https://github.com/lance-format/lance/pull/5913
- refactor: deprecate list_indices and migrate tests to describe_indices by @fenfeng9 in https://github.com/lance-format/lance/pull/5945
- refactor: upgrade to SNAFU 0.9 by @shepmaster in https://github.com/lance-format/lance/pull/6071
Full Changelog: https://github.com/lance-format/lance/compare/release-root/3.0.0-beta.N...v3.0.0