Stoolap Files

What's New in v0.4.0

Immutable Volume Storage Engine

Stoolap now splits each table into a hot MVCC buffer (recent writes, WAL-backed) and cold frozen volumes (historical data, column-major). The query engine merges both sources transparently.

Cold volume format (STV4):

• Per-column per-row-group LZ4 compression with streaming CRC32 verification
• Zone maps (min/max per column per 64K row group) for scan pruning
• Bloom filters for point-lookup acceleration
• Dictionary encoding for text columns
• Deferred column loading with tiered eviction (hot/warm/cold memory tiers)

Lifecycle:

• PRAGMA CHECKPOINT seals hot rows into immutable .vol files, persists manifests, truncates WAL
• Bounded compaction: sub-target volumes merge, oversized volumes split, dirty volumes rewrite
• Background compaction thread (non-blocking checkpoint cycles)
• Cutoff-filtered seal and compaction during snapshot isolation transactions

Crash safety:

• Fsync-before-rename on all atomic writes (volumes, manifests, catalog)
• Two-phase WAL: only committed transactions applied during recovery
• Manifests loaded before WAL replay for idempotent recovery

Columnar Aggregate Pushdown

Filtered and grouped aggregates computed directly on raw column arrays without constructing Row objects:

• Filtered aggregates: SUM/COUNT/MIN/MAX/AVG with typed predicates on i64/f64/dictionary columns
• Grouped aggregates: Single-column GROUP BY with dictionary-indexed accumulators (zero hashing) or FxHashMap for numeric columns
• Dictionary DISTINCT: Extracts unique values from dictionary metadata without scanning rows
• Zone-map pruning: Volumes and row groups skipped when predicates prove no match
• IN list pruning: WHERE id IN (...) generates min/max bounds for zone-map elimination

Query Performance

• ORDER BY PK + LIMIT: K-way merge across sorted volume row_ids, stopping after limit rows
• MIN/MAX typed scan: Direct i64/f64/timestamp access with zone-map volume pruning
• OFFSET skip: Cold scan skips row materialization for offset rows
• Parallel cold scanning: Rayon-based parallel volume processing (4+ volumes, 100K+ rows threshold)

Foreign Key Improvements

• Recursive ON UPDATE CASCADE: Cascades through the full FK chain including grandchild tables, with depth limiting (16 levels)
• Referenced UNIQUE column cascade: FK cascade generalized from PK-only to any referenced UNIQUE column
• Pre-check RESTRICT: RESTRICT constraints checked before writing parent rows, preserving statement-level atomicity in explicit transactions
• SET NULL recursion: SET NULL arm recurses through descendants so deeper RESTRICT checks are enforced

Primary Key Update Protection

UPDATE on primary key columns is now rejected with a clear error message. The engine uses row_id == pk_value as a core invariant across ~50 code paths. This matches SQLite's behavior for rowid tables. Use DELETE + INSERT to change a row's primary key value.

Calendar-Aware INTERVAL Arithmetic

INTERVAL '1 month' and INTERVAL '1 year' now use proper calendar logic instead of 30-day/365-day approximations. Handles leap years, variable month lengths, and preserves nanosecond precision. Matches DATE_ADD behavior.

Schema Evolution Fixes

• CREATE INDEX after DROP+ADD COLUMN: validate_cold_unique and populate_index_from_cold now use column mapping to correctly translate schema indices to physical volume indices
• AS OF PK dedup: Resolves PK column through mapping for correct cold row dedup after schema evolution
• Partition grouping: Uses snapshotted cs.mapping instead of live lookup (immune to compaction races)

ALTER TABLE MODIFY COLUMN Validation

MODIFY COLUMN ... NOT NULL now validates existing data with a streaming IS NULL scan before applying the constraint. Returns a clear error if any NULL values exist.

Data Integrity

• Compaction TOCTOU fix: Snapshot sequence limit captured per-table (matches seal's per-table pattern)
• Manifest truncation errors: Tombstones, column renames, and dropped columns return errors instead of silent data loss on truncated manifests
• Volume corruption guards: Dictionary IDs and bytes offsets validated at deserialization, preventing panics on corrupted volume files
• Aggregation pushdown correctness: Bail on partial WHERE pushdown (prevents wrong results when memory filter is needed)
• Scanner column pruning: Materialize all columns when filter column indices can't be determined
• Seal race fix: collect_rows_with_limit uses collect_hot_row_ids_into instead of has_row_id point lookups

Configuration

New DSN parameters for volume storage tuning:

Parameter	Default	Description
checkpoint_interval	60	Seconds between checkpoint cycles
compact_threshold	4	Sub-target volumes per table before merging
target_volume_rows	1048576	Target rows per cold volume (min 65536)
checkpoint_on_close	on	Seal all hot rows on clean shutdown
volume_compression	on	LZ4 compression for cold volume files
sync_mode	normal	none/off, normal, full (or 0, 1, 2)

Invalid DSN parameter values now return errors instead of silently using defaults.

Migration from v0.3.7

Existing v0.3.7 databases are automatically migrated on first open:

1. Legacy snapshot .bin files loaded into hot buffer
2. WAL entries replayed
3. Hot data sealed into immutable .vol files
4. snapshots/ directory removed

Legacy DSN parameter names (snapshot_interval, snapshot_compression) are accepted for backward compatibility.

Other Changes

• Build timestamp embedded in version_info() output
• CLI error paths ensure database cleanup before exit
• Stale group cache cleared in volume scanner (prevents panic at row-group boundaries)
• Eviction epoch off-by-one corrected
• WASM binary rebuilt with warning-free compilation

Full Changelog: https://github.com/stoolap/stoolap/compare/v0.3.7...v0.4.0