ZeusDB Vector Database Files

The filter language gains boolean composition, presence tests and columns for the fields you declare, and a filtered search returns the nearest of the records the filter matches rather than whatever survived an unfiltered page. The index gains the verbs a store expects, len(), in, count(), delete(), clear(), update_metadata() and rebuild() among them, and dot as a fourth metric. A save is atomic and verified, a quantized index is scored on the metric it declared, and an unquantized index holds about half the memory it did. dim is now required, add() raises on a malformed batch and l1 can no longer be quantized, so read the breaking list before upgrading. A directory saved by 0.7.0 opens unchanged unless it pairs l1 with quantization or was created outside the new bounds.

Breaking

• dim is required on create(). It defaulted to 1,536, which built an index sized for one vendor's model family and reported the mistake at the first add(), as a dimension mismatch on every vector of any other width. create("hnsw") without it raises TypeError. Pass the width your embedding model produces, for example dim=1536 for OpenAI text-embedding-3-small or dim=768 for most sentence-transformers models.
• add() raises ValueError when the parallel arrays of a batch dict disagree in length, naming both lengths and the short field. It used to insert a prefix in one direction and store the surplus records under generated ids in the other, so one id against two vectors reported 2 inserted, 0 errors. Make ids, metadatas and the vector array the same length. Omitting ids entirely is not a disagreement and still generates one per record.
• A parallel array that is not a list raises TypeError. A tuple or an ndarray of ids or metadatas was counted as one error per record on the list path and discarded whole on the NumPy path, where every record then took a generated id. Pass a list, which for an array is ids.tolist().
• A metadata filter decides which records are ranked, not which results survive. The traversal admits only matching records, so a filtered search returns the top_k nearest of the records the filter matches. It used to find the top_k nearest of every record and then discard the ones the filter rejected, so a selective filter returned a short or empty page. A filter matching 5 of 300 records at top_k=5 returns all 5 where it returned none. Code that raised top_k to compensate can stop, and a test that pinned a short page under a filter needs its expectation refreshed. Where the filter matches fewer than 5,000 records those records are scored exactly rather than traversed, and that page is ordered by distance and then by id.
• l1 cannot be quantized. create(space="l1", quantization_config=...) raises ValueError. A quantized graph scores every candidate from tables of squared L2 distances to the codebook, and Manhattan distance does not order the same way, so such an index returned the wrong records and reported a score on a quantity it never declared. L1 tables over the shipped codebook and over a k-medians codebook fitted to absolute error were measured by brute force on three corpora at 100,000 records, and ranked 0.03 to 0.08 below what quantized l2 reaches at its own game on the same records, and at the default subvectors below the squared L2 ordering they would replace, so the pair stays refused. Use space="l2" with the same configuration, or l1 without quantization.
• A directory pairing l1 with quantization no longer opens, whatever release saved it. load() raises the same ValueError, naming the path. Open it under the release that saved it, read the records back with get_records(), and add them to an index created under l2 or without quantization. Every other directory saved by 0.7.0 inside the bounds below opens without any action.
• A quantized cosine index ranks and scores by cosine distance to the reconstruction. It ranked and scored by the squared L2 distance to it and reported that sum as the score, which read 1.86 times the cosine distance at 25,000 records. Result order changes, in each measured case for the better. Recall at 10 rose by 0.0015 to 0.0518 across three corpora at two subvector counts each, with every 95 percent half width at or under 0.001. The exact scan path normalises the reconstruction before scoring, as add() does, where it scored against a reconstruction of length 0.90. A search costs 1 to 3 percent more and a build 3 to 8 percent more. A test that pinned scores or exact result sets from a quantized cosine index needs its expectations refreshed. A quantized cosine directory saved by an earlier release opens and is scored by cosine distance from then on. Its graph keeps the neighbour lists it was saved with. Loading it once with ZEUSDB_LOAD_REBUILD_GRAPH=1 set and saving it again wires it on the new ordering.
• A quantized l2 index reports the rooted distance on every path. The traversal returned the squared sum and the exact scan the rooted distance, so one index reported 159.416 and 12.626 for the same record depending on how selective the filter was. The page is rooted once per candidate, which does not move the order. A threshold on the score of a quantized l2 index that was derived from the traversal path needs re-deriving on the rooted scale. An unquantized index is unchanged.
• dim, ef_construction, top_k and ef_search have ceilings. dim above 65,536 and ef_construction above 4,096 raise at create(), rebuild() and load(), and top_k above 65,536 and ef_search above 131,072 raise at search(). None had one, and create(dim=2**40) ended the process with an allocation failure. A directory saved with a value outside these bounds does not open under this release. Open it under the release that saved it, read the records back with get_records(), and add them to an index created inside the bounds.
• save() and load() print nothing. Both wrote progress lines to stdout, ending in a line reporting that the save or load had completed. A script that read stdout for those lines has nothing to read. save() returns None and raises on failure, load() returns the index, and the log records are unchanged.

Fixed

• Records queued in the log file's writer at process exit were lost, because the appender's worker guard was never dropped and its flush() is a no-op. The guard is dropped at exit through an atexit hook registered on import, which waits for the writer to drain. os._exit() and an abort run no hook.
• A trained quantized_with_raw index saved while holding no records reopened without its raw vector store, so every record added after the load lost its raw vector permanently and get_records() returned a reconstruction. Reached by clear() or by removing every record before a save. The store is opened whenever the storage mode expects one.
• A quantized_with_raw directory whose vectors.bin was lost opened as a complete index built from reconstructions with the top page moved, one whose quantization.json was lost opened as unquantized, and one whose pq_codes.bin was lost reported itself quantized while holding no codes. A raw index saved over a quantized directory reopened carrying the previous save's codebook and codes. load() checks files_included in manifest.json against the directory in both directions before it parses anything, reads each quantization artefact only when the manifest names it, and refuses a directory with a file missing or with a file the manifest does not name, naming the file.
• total_size_mb in manifest.json counted the previous save's graph dump or none at all, training_progress in get_stats() read 0/1000 (100.0%) on a trained index, created_at recorded the load rather than the creation when a loaded index was saved again, and training_completed_at was stamped at save rather than when the codebook was fitted. All four are corrected, and the two timestamps survive a save and load.
• ZEUSDB_LOG_LEVEL=warning was accepted by the Python layer and refused by the Rust layer, which printed ignoring zeusdb_vector_database=warning to stderr and then filtered nothing, and warn failed the other way round. Both layers accept warn, warning, err, error, fatal and critical.
• A filter nested about four thousand levels deep overflowed the stack and ended the process. Conversion is capped at 128 levels and group nesting at 10, and either raises ValueError naming the depth.
• load() validates what it reads. config.json and quantization.json are held to the rules create() applies to dim, space, m, ef_construction, expected_size, bits and subvectors, where an unrecognised space used to fall back to cosine in silence, bits of 40 ended the process, bits of 64 masked every code to zero and subvectors of 0 divided by zero. Every container length in mappings.bin, vectors.bin, pq_codes.bin and pq_centroids.bin is decoded under a budget of 64 bytes per byte the file holds, where ten of them went to the allocator unexamined and a declared length of 2^40 ended the process. A graph dump naming an origin id above what config.json counts is refused, where it sized the graph's id table from that id and ended the process. A damaged or hand edited directory raises where it used to end the process or open wrongly.
• The quantized_with_raw warning, the create() docstring and the README said the mode held less memory than an unquantized index above a break even. It holds more at every record count, 1.08 times at 50,000 records of dimension 1,536 and 1.14 times at dimension 128, because it keeps every raw vector and adds the codes and the trained tables to it. The warning describes the mode as the accuracy mode and quotes only figures exact from the configuration.
• Eight error messages carried runs of spaces where a line continuation had been lost.

Added

• space="dot", the inner product, as a fourth metric. search() reports 1 - dot so that lower stays better, vectors are stored as given rather than normalised, and index.space reads the metric back. It cannot be combined with quantization_config, which raises ValueError at create(). The codebook is fitted by squared L2 and cannot rank by the inner product, with recall at 10 never above 0.37 by brute force over its own reconstructions across three corpora and stored length spreads up to three orders of magnitude, against an unquantized dot index at least 0.35 higher on the same data. Use cosine on normalised vectors where only direction should count, or dot without quantization where length must count.
• Boolean composition in the filter language. $and, $or and $not compose whole filters, where a mapping was only ever a conjunction of its fields. The three names are reserved as keys, so a field named $and, $or or $not raises ValueError naming the key, and any other name beginning with $ still filters. Groups nest to 10 levels.
• Filter operators nin, any, all, exists, is_missing and is_null. nin, any and all exclude a record lacking the field, as the existing operators do, and exists, is_missing and is_null take true or false and refuse anything else.
• create(indexed_fields=[...]) builds a column for each metadata field named, so a filter naming only declared fields is answered from the columns rather than by reading every record's metadata. Up to 32 names. At 100,000 records a filter matching one record costs 0.09 to 0.15 milliseconds where it cost 28 to 74, and eight declared columns cost 6.69 MB. A filter naming an undeclared field returns the same records, reads every record to find them and logs one warning naming the field, and one naming a declared field beside an undeclared one is bounded by the declared branch where that removes at least two thirds of the records. The declaration is carried in config.json and the columns are rebuilt from metadata.json on load, so no file is added and a directory saved before this opens unchanged. index.indexed_fields reads the declaration back.
• len(index), id in index, and count(filter=None), which walks every record with the interpreter lock released.
• remove_points(ids), returning the ids that were not present, remove_where(filter), returning how many it removed, and delete(ids=..., where=...), which dispatches to them, returns a count, and raises when given both arguments or neither. remove_where({}) is refused, because an empty filter matches every record everywhere else in the language and here that would empty the index.
• clear(), which replaces the graph rather than removing record by record, keeps the configuration including a fitted codebook, and returns the count removed. It does not reset the generated id counter, so ids generated after it continue the sequence.
• update_metadata(id, metadata), which replaces one record's metadata wholesale, as add(overwrite=True) does, leaves the vector, the codes and the graph node alone, and returns whether the id was present.
• rebuild(m=..., expected_size=..., ef_construction=...), which builds the graph again at a new configuration, in place. Every record keeps its vector, its metadata, its external id and its internal id, and a quantized index is rebuilt from its stored codes rather than re-encoded. The three arguments are held to the rules create() applies. m and ef_construction were fixed at construction before this.
• shrink_to_fit(), which returns the graph's spare buffer capacity to the allocator and reports the bytes released. compact() calls it.
• list(after=...), a cursor naming the last id of the previous page, beside offset. A deletion ahead of an offset shifts the next page by one and a cursor does not. The two cannot be combined, and a cursor naming a removed record raises KeyError rather than returning a page from somewhere else.
• get_records(strict=True), which raises KeyError naming every id the index does not hold. The default still skips them.
• AddResult.ids, every id the call put in the index, in insertion order.
• index.space, index.m, index.ef_construction and index.expected_size as read-only properties beside index.dim.
• shutdown_logging(), exported at package level, which drains the log file and closes it. It is registered with atexit on import, so a normally exiting process needs no call. Records emitted after an explicit call are discarded.
• centroid_norm_memory_mb in get_stats() on a quantized index, pricing the table of squared centroid norms the cosine scorer reads. It is folded into total_memory_mb.
• A CycloneDX SBOM of the crates the wheel links, attached to each GitHub Release, and dependency scanning with cargo-deny on pull requests touching the manifests and daily, reporting advisories to code scanning.
• A fuzzer over the graph dump reader, a randomised operation sequence checked after every step against a model of what the index holds, subprocess tests for every allocation bound, a comparison of every search page against a brute force ranking, and a lock rank registry that asserts the declared lock order on every acquisition in a debug build.

Changed

• An unquantized index holds about half the memory it did. Every raw vector was held twice, once in a map keyed by external id and once inside the graph, byte for byte. It is held once, in a store addressed by node index that the graph reads rather than owns. At 50,000 records of dimension 1,536 the index commits 325.3 MiB where it committed 632.4. Search latency did not move and the on-disk format is unchanged. graph_memory_mb reports everything the graph holds apart from the raw vectors, and total_memory_mb no longer counts a raw vector twice, so both read lower on an index that has not changed.
• A save writes a sibling directory and renames it into place, so a reader sees the previous index or the new one and never a mixture, and no artefact of an earlier save survives. manifest.json is written last and records a length and a SHA-256 digest for every other artefact, which load() checks before parsing them. A directory saved by an earlier release carries no digests and opens as before. format_version is unchanged at 1.1.0.
• list() returns records in arrival order, ascending by internal id, where it returned hash map iteration order, which differed from one process to the next. The order survives save() and load(), and offset pages over it.
• A 2-D NumPy array of float32 or float64 given to add() or to batch search(), and a 1-D float64 array given to single search(), is read directly rather than through the sequence protocol, which every such array fell through to before. The list path's empty batch check and dimension mismatch message apply on the array paths as well.
• Neighbour vectors are prefetched during a traversal on x86-64.
• The rebuild load() falls back to when a graph dump is absent or damaged runs inside the extension rather than through Python add() calls, and wires the same graph byte for byte.
• The quantized_with_raw warning is suppressed where the configuration never trains, the low dimension warning is removed, the quantized_only break even counts both codes a record carries, which moves the figure from 4,476 to 4,626 records at dimension 64, and every quantization warning attributes to the caller's create() line rather than to library internals.
• Every GitHub Actions step is pinned to a commit SHA, each wheel's SHA-256 is recorded at build and verified before publishing, a check-version job holds the tag, pyproject.toml, vdb-core/Cargo.toml and __version__ to one canonical version, every job carries a timeout, the lock order suite runs on a pull request touching a module that takes a lock, and the compiler is pinned to 1.97.1 in rust-toolchain.toml. The Miri job is removed, and the two tests comparing the AVX kernels against the scalar path bit for bit run on every push.
• The README documents the new verbs, the filter composition and columns, what a filtered search costs, what each storage mode holds, and is_training_ready, training_vectors_needed, rebuild_with_quantization, get_performance_info and benchmark_concurrent_reads, and corrects what it said about batch atomicity, cosine normalisation and quantized_only reconstruction.