Lance Files

What's Changed

Breaking Changes 🛠

• feat(encoding): compress v2.3 mini-block offsets by @Xuanwo in https://github.com/lance-format/lance/pull/8331
• feat(index): change btree scalar index to row-address domain by @westonpace in https://github.com/lance-format/lance/pull/9481
• refactor(index): pass index_version into ScalarIndexPlugin::load_index by @westonpace in https://github.com/lance-format/lance/pull/9518
• feat(mem_wal): evaluate fuzzy conjunctions on the active memtable by @hamersaw in https://github.com/lance-format/lance/pull/9526
• refactor!: vendor jsonb into the lance-jsonb crate by @Xuanwo in https://github.com/lance-format/lance/pull/9534
• perf(mem_wal): optimize Blob v2 writes and promotion by @jackye1995 in https://github.com/lance-format/lance/pull/9717
• feat(cleanup): add configurable delete concurrency by @majin1102 in https://github.com/lance-format/lance/pull/9774
• perf(linalg)!: select f32 l2/dot kernels once and fill batches via _into APIs by @wjones127 in https://github.com/lance-format/lance/pull/10086

Critical Fixes ‼️

• fix(encoding): use each layer's dimension when flattening nested FSL by @LuciferYang in https://github.com/lance-format/lance/pull/9760
• fix(scan): preserve candidates from inexact scalar index results by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9789
• fix: include overlay-stale rows in indexed merge insert by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9865
• fix(encoding): read RLE-compressed mini-block dictionary values by @lichuang in https://github.com/lance-format/lance/pull/10145

New Features 🎉

• feat(encoding): add range and delta block codecs by @Xuanwo in https://github.com/lance-format/lance/pull/8325
• feat(encoding): add bounded block dictionary by @Xuanwo in https://github.com/lance-format/lance/pull/8326
• feat(encoding): compose generic block codecs by @Xuanwo in https://github.com/lance-format/lance/pull/8327
• feat(encoding): select compressed offset blocks by @Xuanwo in https://github.com/lance-format/lance/pull/8330
• feat(index): support Hamming distance in streaming coreset k-means by @a-erofeev in https://github.com/lance-format/lance/pull/8610
• feat: add non-reusable field IDs by @Xuanwo in https://github.com/lance-format/lance/pull/8658
• feat(index): f64 centroids support in streaming coreset k-means by @a-erofeev in https://github.com/lance-format/lance/pull/8662
• feat(java): scan fragment physical row slices by @majin1102 in https://github.com/lance-format/lance/pull/8951
• feat(namespace): return inserted row count and version from insert_into_table by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/9229
• feat(blob): reuse independently addressed managed objects by @Xuanwo in https://github.com/lance-format/lance/pull/9293
• feat(format): add target_data_type to JsonIndexDetails by @wjones127 in https://github.com/lance-format/lance/pull/9303
• feat: support SQL IS [NOT] DISTINCT FROM as null-safe comparison by @jonasdedden in https://github.com/lance-format/lance/pull/9331
• feat(mem_wal): maintain every index by intent, replaceable on a live writer by @xuanyu-z in https://github.com/lance-format/lance/pull/9335
• feat(datafusion): estimate total_byte_size for Lance scans by @vivek-bharathan in https://github.com/lance-format/lance/pull/9430
• feat: allow FRI to be used with stable row IDs by @westonpace in https://github.com/lance-format/lance/pull/9514
• feat(format): define Arrow-ordered zone write seeds by @LuQQiu in https://github.com/lance-format/lance/pull/9564
• feat(python): expose the fragment ids of a dataset by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9592
• feat(python): expose the checked out version by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9594
• feat: record row-address moves for deferred compactions on stable-row-ID tables by @brendanclement in https://github.com/lance-format/lance/pull/9611
• feat(java): count indexed rows on explicit scalar segments by @hfutatzhanghb in https://github.com/lance-format/lance/pull/9644
• feat(java): open a dataset at a branch or tag by @zy-kkk in https://github.com/lance-format/lance/pull/9703
• feat(mem_wal): move a live writer onto a schema change instead of reopening it by @xuanyu-z in https://github.com/lance-format/lance/pull/9712
• feat(count): look through row-preserving custom nodes in CountPushdown by @wkalt in https://github.com/lance-format/lance/pull/9716
• feat(scanner): make Scanner::project_with_schema public by @amunra in https://github.com/lance-format/lance/pull/9735
• feat(exec): late-materialize non-sort columns under a top-k by @wkalt in https://github.com/lance-format/lance/pull/9750
• feat(search): add search effort and auto probing through k=100000 by @BubbleCal in https://github.com/lance-format/lance/pull/9798
• feat(dataset): support stable spilled row lineage by @BubbleCal in https://github.com/lance-format/lance/pull/9802
• feat(mem_wal): make memtable index kinds pluggable by @xuanyu-z in https://github.com/lance-format/lance/pull/9803
• feat(mem_wal): answer memtable filters with Lance's scalar expression pass by @xuanyu-z in https://github.com/lance-format/lance/pull/9804
• feat(mem_wal): pass writer settings and flush build parameters to memtable index plugins by @xuanyu-z in https://github.com/lance-format/lance/pull/9805
• feat(cache): add tier probes and selective entry reads by @BubbleCal in https://github.com/lance-format/lance/pull/9829
• feat(io): make file-read coalescing configurable by @BubbleCal in https://github.com/lance-format/lance/pull/9833
• feat(fts): allow external indexed input for hybrid queries by @BubbleCal in https://github.com/lance-format/lance/pull/9847
• feat(index): distributed execution of optimize indices by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9871
• feat(format): define provider-managed clustering metadata by @LuQQiu in https://github.com/lance-format/lance/pull/9878
• feat(blob): expose managed sharing in Python preparation API by @Xuanwo in https://github.com/lance-format/lance/pull/9880
• feat: synchronous FilteredReadExec encoding given a resolved table identifier by @LuQQiu in https://github.com/lance-format/lance/pull/9919
• feat(format): define independent planar metadata for v2.3 by @Xuanwo in https://github.com/lance-format/lance/pull/9920
• feat(encoding): implement planar page execution for Lance 2.3 by @Xuanwo in https://github.com/lance-format/lance/pull/9921
• feat(encoding): add indexed values to sparse planar pages by @Xuanwo in https://github.com/lance-format/lance/pull/9925
• feat(encoding): cap the dictionary size a page may be encoded with by @dshepelev15 in https://github.com/lance-format/lance/pull/9939
• feat(java): expose MinHash LSH indices to Java by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9965
• feat(python): expose the MinHash LSH index to python by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9966
• feat(index): add a PCA model and vector pre-transform by @beautygua in https://github.com/lance-format/lance/pull/9967
• feat(dataset): spill carried lineage in merge insert by @BubbleCal in https://github.com/lance-format/lance/pull/10062
• feat(blob): support managed data file parts and descriptor reuse by @Xuanwo in https://github.com/lance-format/lance/pull/10176
• feat(index): allow training IVF from supplied samples by @BubbleCal in https://github.com/lance-format/lance/pull/10188
• feat(namespace): add the spec operations LanceNamespace lacked by @twmb in https://github.com/lance-format/lance/pull/10192

Bug Fixes 🐛

• fix: assign fragment and row ids to new fragments in Merge commits by @wkalt in https://github.com/lance-format/lance/pull/7706
• fix(encoding): validate block bitpacking payloads by @Xuanwo in https://github.com/lance-format/lance/pull/8328
• fix(encoding): bound general block decompression by @Xuanwo in https://github.com/lance-format/lance/pull/8329
• fix: choose cheapest blanks when restoring deleted rows by @chenyu-x in https://github.com/lance-format/lance/pull/8946
• fix(file): verify 2.1+ nullability against name-selected columns by @jackylee-ch in https://github.com/lance-format/lance/pull/9247
• fix: prune overlays for dropped columns by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9314
• fix(datafusion): preserve boolean expression literal types by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9322
• fix(python): say that create_index ignores index_cache_size by @LuciferYang in https://github.com/lance-format/lance/pull/9384
• fix(index): store the converted metric in IvfTransformer::new_flat by @LuciferYang in https://github.com/lance-format/lance/pull/9399
• fix(index): report shuffle-stage failures instead of panicking by @LuciferYang in https://github.com/lance-format/lance/pull/9405
• fix(index): reject the hamming metric for IVF_RQ before training by @LuciferYang in https://github.com/lance-format/lance/pull/9423
• fix(blob): deduplicate managed reference scans during cleanup by @Xuanwo in https://github.com/lance-format/lance/pull/9435
• fix: avoid panic for missing duplicate field id by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9448
• fix: move inline transaction to a new manifest field by @wjones127 in https://github.com/lance-format/lance/pull/9482
• fix(index): carry merged segment coverage through compaction only while indexed data matches by @xuanyu-z in https://github.com/lance-format/lance/pull/9524
• fix: write nullable structs that contain a Null-typed field by @breken-ai in https://github.com/lance-format/lance/pull/9579
• fix: read back FixedSizeList columns of the Null type by @breken-ai in https://github.com/lance-format/lance/pull/9580
• fix(namespace): handle legacy auto-cleanup policy on directory manifest by @Xuanwo in https://github.com/lance-format/lance/pull/9600
• fix: preserve fragment reuse mappings across concurrent maintenance by @brendanclement in https://github.com/lance-format/lance/pull/9620
• fix(index): bound bitmap cache state lookup batches by @Xuanwo in https://github.com/lance-format/lance/pull/9622
• fix: start the commit retry budget after catching up by @Xuanwo in https://github.com/lance-format/lance/pull/9631
• fix(io): count HEAD requests without object body bytes by @geruh in https://github.com/lance-format/lance/pull/9648
• fix(scan): avoid exact counts when retaining deleted rows by @geruh in https://github.com/lance-format/lance/pull/9652
• fix(java): include baseId in DataFile/DeletionFile equals and add hashCode by @Zhuoxi2000 in https://github.com/lance-format/lance/pull/9686
• fix(mem_wal): rank a dot-product memtable index by distance, not by product by @xuanyu-z in https://github.com/lance-format/lance/pull/9689
• fix: prevent incomplete unordered listings after retry errors by @kamronis in https://github.com/lance-format/lance/pull/9691
• fix(datafusion): preserve regexp_match null checks by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9698
• fix(scanner): preserve requested projection in fast search by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9699
• fix: enforce AIMD delete request budget by @kamronis in https://github.com/lance-format/lance/pull/9707
• fix(cleanup): tolerate missing objects without error logs by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9709
• fix(io): resolve cloud option aliases deterministically by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9719
• fix(index): reject JSON array and object index paths by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9727
• fix(index): remap every segment of a tagged index instead of only the first by @LuQQiu in https://github.com/lance-format/lance/pull/9728
• fix(index): admit MemWAL state updates on tables with a tagged fragment reuse history by @LuQQiu in https://github.com/lance-format/lance/pull/9729
• fix(index): check rewrite groups one by one and decode the FRI ledger only when needed by @LuQQiu in https://github.com/lance-format/lance/pull/9731
• fix(mem_wal): full-text search across columns added or renamed since a source was written by @xuanyu-z in https://github.com/lance-format/lance/pull/9740
• fix(merge_insert): accept legacy blob input for Blob v2 columns by @wkalt in https://github.com/lance-format/lance/pull/9741
• fix(merge_insert): rewrite rows for partial blob sources on indexed joins by @wkalt in https://github.com/lance-format/lance/pull/9742
• fix(cleanup): reclaim abandoned temporary data files by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9747
• fix: make MultiMatchQuery serde lossless by @hamersaw in https://github.com/lance-format/lance/pull/9753
• fix(io): reject zero multipart upload concurrency by @kamronis in https://github.com/lance-format/lance/pull/9756
• fix(merge-insert): preserve join columns in horizontal updates by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9765
• fix: keep stable row IDs when cloning or branching an emptied table by @brendanclement in https://github.com/lance-format/lance/pull/9768
• fix(index): bound preflight scans and batch vector sampling by @BubbleCal in https://github.com/lance-format/lance/pull/9776
• fix(cleanup): share delete rate limit across cascaded branches by @majin1102 in https://github.com/lance-format/lance/pull/9782
• fix(mem_wal): cast a point lookup's base struct to the selected child order by @hamersaw in https://github.com/lance-format/lance/pull/9788
• fix(python): accept version 0 as the begin of an explicit delta range by @LuciferYang in https://github.com/lance-format/lance/pull/9794
• fix(index): refuse preprocessed training data in a shape the index does not train from by @xuanyu-z in https://github.com/lance-format/lance/pull/9795
• fix(mem_wal): keep null rows out of memtable B-tree range filters by @xuanyu-z in https://github.com/lance-format/lance/pull/9796
• fix(commit): let a deferred rewrite exempt a reindex across processes on untagged FRI tables by @xuanyu-z in https://github.com/lance-format/lance/pull/9808
• fix(mem_wal): report one column statistic per output column from memtable scans by @hamersaw in https://github.com/lance-format/lance/pull/9809
• fix: keep scan order when the optimizer adds parallelism by @vivek-bharathan in https://github.com/lance-format/lance/pull/9819
• fix(namespace): preserve unbounded IVF probe maximum by @jackye1995 in https://github.com/lance-format/lance/pull/9822
• fix(cache): count IndexMetadata::index_details in deep_size_of by @zehiko in https://github.com/lance-format/lance/pull/9828
• fix(mem_wal): skip already-flushed generations when replaying the WAL by @xuanyu-z in https://github.com/lance-format/lance/pull/9840
• fix(mem_wal): read each memtable's visible batches once per LSM query by @xuanyu-z in https://github.com/lance-format/lance/pull/9841
• fix(mem_wal): rank every vector search arm on one distance scale by @hamersaw in https://github.com/lance-format/lance/pull/9849
• fix: allow renames alongside column casts by @app/lance-gatefixer in https://github.com/lance-format/lance/pull/9854
• fix(mem_wal): keep the table's field ids in flushed generations by @xuanyu-z in https://github.com/lance-format/lance/pull/9857
• fix(index): write zone map seeds when a seeded column gets a new data file by @LuQQiu in https://github.com/lance-format/lance/pull/9868
• fix(blob): require explicit opt-in for managed storage by @jackye1995 in https://github.com/lance-format/lance/pull/9891
• fix(mem_wal): compare maintained indexes by resolved settings, not base details by @xuanyu-z in https://github.com/lance-format/lance/pull/9899
• fix(index): keep a vector index's details when remapping it by @xuanyu-z in https://github.com/lance-format/lance/pull/9900
• fix(encoding): size full-zip validity by decoded items, not rows by @dshepelev15 in https://github.com/lance-format/lance/pull/9906
• fix: count rows selected by row id or row address by @vivek-bharathan in https://github.com/lance-format/lance/pull/9907
• fix(encoding): do not materialize the constant scalar into null slots by @dsingh-exa in https://github.com/lance-format/lance/pull/9909
• fix(encoding): bounds-check the item payload in VariableFullZipDecoder::unzip by @dsingh-exa in https://github.com/lance-format/lance/pull/9910
• fix(io): call multipart complete once and retry it at the HTTP layer by @dsingh-exa in https://github.com/lance-format/lance/pull/9913
• fix(index): trace segment coverage across recorded compactions by @brendanclement in https://github.com/lance-format/lance/pull/9946
• fix(merge_insert): build update_if structs by name on indexed joins by @wkalt in https://github.com/lance-format/lance/pull/9950
• fix(lance-jsonb): escape object keys when displaying Value by @westonpace in https://github.com/lance-format/lance/pull/9951
• fix(mem_wal): make shard manifest versions safe to prune by @xuanyu-z in https://github.com/lance-format/lance/pull/9953
• fix: read struct children added below a list item at any depth by @wkalt in https://github.com/lance-format/lance/pull/9954
• fix(mem_wal): keep field ids on prepared blob v2 columns by @xuanyu-z in https://github.com/lance-format/lance/pull/9957
• fix(mem_wal): emit an LSM scan's sources in a fixed order by @hamersaw in https://github.com/lance-format/lance/pull/9983
• fix(namespace): use POST for REST count_rows, tags/list and stats by @twmb in https://github.com/lance-format/lance/pull/9993
• fix(namespace): tell tables and namespaces apart in the dir manifest by @twmb in https://github.com/lance-format/lance/pull/9998
• fix(index): harvest zone map seeds from the serving data file by @BubbleCal in https://github.com/lance-format/lance/pull/10003
• fix(index): reject mixed row domains in scalar index segments by @BubbleCal in https://github.com/lance-format/lance/pull/10004
• fix(io): stop ignoring storage configuration that lance accepts by @LuciferYang in https://github.com/lance-format/lance/pull/10008
• fix(io): resolve locations that the rest of the stack accepts by @LuciferYang in https://github.com/lance-format/lance/pull/10009
• fix(io): record the IO that currently goes uncounted by @LuciferYang in https://github.com/lance-format/lance/pull/10010
• fix(scanner): align row offset statistics with output schema by @BubbleCal in https://github.com/lance-format/lance/pull/10030
• fix(mem_wal): retry a failed memtable flush before any later generation by @hamersaw in https://github.com/lance-format/lance/pull/10031
• fix(mem_wal): reject unhashable primary key types at initialization by @hamersaw in https://github.com/lance-format/lance/pull/10040
• fix(compaction): remap address-domain indices eagerly on stable-row-id tables by @westonpace in https://github.com/lance-format/lance/pull/10041
• fix(fts): refine stop words across latin-script languages by @BubbleCal in https://github.com/lance-format/lance/pull/10045
• fix(fts): preserve Chinese and Japanese search terms by @BubbleCal in https://github.com/lance-format/lance/pull/10046
• fix(fts): preserve Korean content and numeric tokens by @BubbleCal in https://github.com/lance-format/lance/pull/10048
• fix(fts): preserve Arabic content and polarity by @BubbleCal in https://github.com/lance-format/lance/pull/10049
• fix(fts): preserve Greek content and polarity by @BubbleCal in https://github.com/lance-format/lance/pull/10050
• fix(fts): preserve Russian content and numeric terms by @BubbleCal in https://github.com/lance-format/lance/pull/10051
• fix(fts): preserve Tamil content and numeric terms by @BubbleCal in https://github.com/lance-format/lance/pull/10052
• fix(index): allow partition rebalancing within explicit merge windows by @BubbleCal in https://github.com/lance-format/lance/pull/10076
• fix(index): keep the earliest-created segment when every index segment is empty by @ump45nose in https://github.com/lance-format/lance/pull/10107
• fix(index): drop old fragment ids from stale indices on deferred overlay compaction by @ump45nose in https://github.com/lance-format/lance/pull/10109
• fix(fsst): bound sampling for sparse inputs by @LuQQiu in https://github.com/lance-format/lance/pull/10120
• fix(fts): size the staged finalize from the build's memory config and report one progress stage by @LuQQiu in https://github.com/lance-format/lance/pull/10141
• fix(encoding): admit repeated Planar dictionary candidates by @Xuanwo in https://github.com/lance-format/lance/pull/10184
• fix(namespace): conform the REST client and adapter to the spec by @twmb in https://github.com/lance-format/lance/pull/10190
• fix(io): poll every sibling file reader in MergeStream to avoid deadlock by @westonpace in https://github.com/lance-format/lance/pull/10191
• fix(blob): backport descriptor subfield queries to v14 by @lance-community in https://github.com/lance-format/lance/pull/10247

Documentation 📚

• docs(format): define generic mini-block offsets by @Xuanwo in https://github.com/lance-format/lance/pull/8332
• docs: specify JSONB logical and physical storage by @Xuanwo in https://github.com/lance-format/lance/pull/9194
• docs: apache ozone as object store configuration guide by @ChenSammi in https://github.com/lance-format/lance/pull/9601
• docs: document the managed blob table capability by @Xuanwo in https://github.com/lance-format/lance/pull/9778
• docs(format): stabilize spilled row lineage by @BubbleCal in https://github.com/lance-format/lance/pull/9800
• docs(format): stabilize spilled row lineage by @BubbleCal in https://github.com/lance-format/lance/pull/9886
• docs(format): allow variable-length zones in version 1 write seeds by @LuQQiu in https://github.com/lance-format/lance/pull/10155

Performance Improvements 🚀

• perf(scanner): renmove backtics from column names by @yuvalif in https://github.com/lance-format/lance/pull/7782
• perf(encoding): measure compressed mini-block offsets by @Xuanwo in https://github.com/lance-format/lance/pull/8333
• perf(index): merge Bitmap index segments instead of rescanning the dataset by @amunra in https://github.com/lance-format/lance/pull/9290
• perf(index): fetch the IVF and quantizer global buffers together on index open by @LuciferYang in https://github.com/lance-format/lance/pull/9378
• perf(index): bucket shuffle rows by partition id instead of sorting keys by @LuciferYang in https://github.com/lance-format/lance/pull/9382
• perf: prune CDF fragments using row-version metadata by @Abhisheklearn12 in https://github.com/lance-format/lance/pull/9409
• perf(index): skip empty ranges when planning a partition read by @LuciferYang in https://github.com/lance-format/lance/pull/9413
• perf(index): reduce RQ threshold search overhead by @BubbleCal in https://github.com/lance-format/lance/pull/9464
• perf(index): reuse bounded scratch without serializing RQ batches by @BubbleCal in https://github.com/lance-format/lance/pull/9465
• perf(index): fuse RQ rotation quantization and factor computation by @BubbleCal in https://github.com/lance-format/lance/pull/9466
• perf(table): keep clustered row ID ranges compact in index by @Abhisheklearn12 in https://github.com/lance-format/lance/pull/9501
• perf(filter): narrow exact double literals against Float32 columns by @jonasdedden in https://github.com/lance-format/lance/pull/9512
• perf(python): read the version number without building the summary by @zhangyue19921010 in https://github.com/lance-format/lance/pull/9593
• perf(mem_wal): build LSM source scans with bounded concurrency by @beinan in https://github.com/lance-format/lance/pull/9642
• perf(scan): reuse fragment lookup when decoding plans by @geruh in https://github.com/lance-format/lance/pull/9653
• perf(core): index column names in Schema::project by @LuciferYang in https://github.com/lance-format/lance/pull/9711
• perf(index): bound merge_segments prefetch to one scan's byte budget by @amunra in https://github.com/lance-format/lance/pull/9761
• perf(mem_wal): answer a filtered LSM scan of memtables from their filter indexes by @xuanyu-z in https://github.com/lance-format/lance/pull/9806
• perf(mem_wal): rank only prefiltered, newest memtable rows in a filtered vector search by @hamersaw in https://github.com/lance-format/lance/pull/9812
• perf(index): add four-chain native u2 and u6 ex-dot kernels by @BubbleCal in https://github.com/lance-format/lance/pull/9839
• perf(mem_wal): grow memtable B-tree arena chunks from 4 KiB by @xuanyu-z in https://github.com/lance-format/lance/pull/9848
• perf(index): record zone map seed null positions as runs by @LuQQiu in https://github.com/lance-format/lance/pull/9860
• perf(index): skip nested value traversal when collecting zone map statistics by @LuQQiu in https://github.com/lance-format/lance/pull/9874
• perf(transaction): handle Rewrite groups in O(n) and bound-check group ranges by @dsingh-exa in https://github.com/lance-format/lance/pull/9908
• perf(dataset): serve read_version_transaction from the held or cached manifest by @dsingh-exa in https://github.com/lance-format/lance/pull/9915
• perf(python): read the version id directly when committing a transaction by @dsingh-exa in https://github.com/lance-format/lance/pull/9916
• perf(fragment): read the version id directly when writing deletion files by @dsingh-exa in https://github.com/lance-format/lance/pull/9917
• perf(encoding): write zstd output into reserved capacity instead of zero-filling by @dsingh-exa in https://github.com/lance-format/lance/pull/9918
• perf(encoding): bound planar sequence and offset codec selection by @Xuanwo in https://github.com/lance-format/lance/pull/9923
• perf(encoding): reduce composite RLE selection and materialization costs by @Xuanwo in https://github.com/lance-format/lance/pull/9924
• perf(file): decode lazily loaded column metadata from the tail read by @dshepelev15 in https://github.com/lance-format/lance/pull/9938
• perf(index): read the manifest version without summarizing every fragment by @dshepelev15 in https://github.com/lance-format/lance/pull/9942
• perf(index): cache the ownership mask of a vector index segment by @dshepelev15 in https://github.com/lance-format/lance/pull/9943
• perf(index): train and encode PQ codebooks through an f32 SoA centroid table by @dshepelev15 in https://github.com/lance-format/lance/pull/9945
• perf(scanner): serve null-check filters from a narrow leaf by @wkalt in https://github.com/lance-format/lance/pull/9948
• perf(merge_insert): evaluate target-column conditions on the rows they gate by @wkalt in https://github.com/lance-format/lance/pull/9949
• perf(mem_wal): allocate HNSW neighbor lists per node and charge missed index memory by @xuanyu-z in https://github.com/lance-format/lance/pull/9990
• perf(dataset): sum manifest row counts in Dataset::count_rows(None) by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10016
• perf(scan): share one planned filter across all fragments of a scan by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10042
• perf(merge_insert): prune target columns from the join scan unless matched rows are updated by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10043
• perf(rowids): load fragment sequences on demand by @BubbleCal in https://github.com/lance-format/lance/pull/10070
• perf(fts): finalize staged partitions in place with budgeted parallel copies and consolidation by @LuQQiu in https://github.com/lance-format/lance/pull/10138
• perf(io): raise the default AIMD ceiling to 10k by @BubbleCal in https://github.com/lance-format/lance/pull/10153
• perf(dataset): take rows by id without reading deletion files by @dshepelev15 in https://github.com/lance-format/lance/pull/10163
• perf(dataset): open only the data files that hold a projected field by @dshepelev15 in https://github.com/lance-format/lance/pull/10178
• perf(encoding): dictionary-encode repeated fixed-size vectors by @Xuanwo in https://github.com/lance-format/lance/pull/10187
• perf(fts): load requested postings on cold searches by @BubbleCal in https://github.com/lance-format/lance/pull/10198
• perf(fts): pack postings during position prewarm by @BubbleCal in https://github.com/lance-format/lance/pull/10201

Other Changes

• refactor(mem-wal): finish the SSTable/frozen naming pass by @hamersaw in https://github.com/lance-format/lance/pull/8836
• refactor(core): share CloneableError via Arc<error> instead of rebuilding on clone by @LuciferYang in https://github.com/lance-format/lance/pull/9626
• refactor: stop using DataFusion APIs deprecated or changed in DataFusion 55 by @slachiewicz in https://github.com/lance-format/lance/pull/9640
• refactor(index): expose RaBitQ encoding and factor primitives by @BubbleCal in https://github.com/lance-format/lance/pull/9832
• refactor(mem_wal): drop a dead branch and repeated tests, correct stale docs by @xuanyu-z in https://github.com/lance-format/lance/pull/9838
• refactor(encoding): reuse generic blocks for sparse structural arrays by @Xuanwo in https://github.com/lance-format/lance/pull/9843
• revert: undo premature spilled row lineage stabilization by @BubbleCal in https://github.com/lance-format/lance/pull/9885
• Revert "docs: specify JSONB logical and physical storage" by @Xuanwo in https://github.com/lance-format/lance/pull/9889
• refactor(format): remove unstable declarations from stable encoding schema by @Xuanwo in https://github.com/lance-format/lance/pull/9922
• refactor: split dataset/blob.rs into submodules by stage by @wjones127 in https://github.com/lance-format/lance/pull/9989
• refactor(namespace): split lance-namespace-impls dir.rs into submodules by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10073
• refactor(encoding): split encodings/logical/primitive.rs into submodules by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10075

Full Changelog: https://github.com/lance-format/lance/compare/release-root/14.0.0-beta.N...v14.0.0