QuestDB 9.3.4

QuestDB 9.3.4 delivers dynamic windows in WINDOW JOIN, Parquet row group pruning with bloom filters, new array functions, and significant performance improvements across ORDER BY, joins, and Parquet I/O.

For any questions or feedback, please join us on Slack or on Discourse.

See also our prettier release notes page.

Breaking changes 💥

• Parquet Varchar encoding default changed: Parquet export now uses RLE Dictionary encoding for Varchar columns instead of Delta Length Byte Array. Parquet files written by 9.3.4 use the new encoding by default. If downstream tools or pipelines depend on the previous encoding, use the new per-column encoding config to override.
• by @RaphDal in #6809
• Constant expression folding: The SQL engine now folds constant expressions at compile time, which aligned compile-time behaviour of NaN/Infinity values with their runtime behaviour. Infinity and -Infinity in constant float/double expressions are now collapsed to NULL at compile time, consistent with QuestDB's existing NULL convention. CASE/SWITCH expressions can no longer branch on Infinity or -Infinity as distinct values.
• by @RaphDal in #6828

Highlights

Dynamic window support in WINDOW JOIN

WINDOW JOIN now supports dynamic window ranges computed from column values or expressions:

:::sql
SELECT t.*,
       avg(p.mid) AS avg_mid
FROM trades t
WINDOW JOIN prices p
  ON p.sym = t.sym
  RANGE BETWEEN t.lookback PRECEDING
        AND CURRENT ROW
  INCLUDE PREVAILING;

Parquet row group pruning with bloom filters

Queries over Parquet files now leverage min/max statistics and bloom filters to skip entire row groups that cannot match the query predicate, dramatically reducing I/O for selective queries.

Array functions

New DOUBLE[] functions: array_sort(), array_reverse(), array_elem_min(), array_elem_max(), array_elem_avg(), and array_elem_sum().

Other

• arg_min() and arg_max() now support CHAR arguments.
• Per-column Parquet encoding/compression configuration.
• minTimestamp and maxTimestamp columns added to sys.telemetry_wal.

Performance

• ASOF and WINDOW JOIN: Faster execution for large right-hand-side tables.
• ORDER BY: Pre-computed sort keys for fixed-width types; dedicated fast path for SYMBOL columns.
• HORIZON JOIN: Better parallelization across various data distributions.
• Vectorized GROUP BY: More non-keyed queries now use SIMD computation.
• Parquet decoding: Faster row group decompression and column materialization.
• Parquet writing: Reduced write amplification and improved write speed.
• WAL writer: New cairo.wal.writer.madvise.mode config option for tuning memory access patterns.

Bug fixes

• Fixed data corruption in DECIMAL128 and DECIMAL256 columns.
• Fixed crash on LATEST BY ALL queries over large tables.
• Fixed crash when Parquet partition statistics are missing.
• Fixed crash in SAMPLE BY FILL with array column aggregates.
• Fixed read_parquet() crash on SYMBOL columns from native Parquet files.
• Fixed WINDOW JOIN INCLUDE PREVAILING dropping the prevailing row when the window had matches.
• Fixed WINDOW JOIN dropping the prevailing row on cross-partition boundaries.
• Fixed AssertionError triggered by certain JOIN queries.
• Fixed read_parquet() on Parquet files with stale QuestDB metadata.
• Fixed support for quoted column names in ALTER COLUMN.
• Fixed resource leaks and NPEs in the SQL engine.
• Disabled materialized view parallel SQL on low-core machines.

Changelist

• fix(core): fix crash on LATEST BY ALL queries over large tables by @ideoma in #6832
• feat(core): add minTimestamp and maxTimestamp columns to sys.telemetry_wal by @ideoma in #6779
• fix(parquet): fix crash when parquet partition statistics are missing by @kafka1991 in #6827
• feat(sql): add array_sort and array_reverse functions for double arrays by @javier in #6820
• perf(wal): add cairo.wal.writer.madvise.mode configuration option by @jerrinot in #6841
• perf(sql): speed up ASOF and WINDOW JOINs for large right-hand-side tables by @puzpuzpuz in #6822
• fix(sql): fix SAMPLE BY FILL crash with array column aggregates by @javier in #6811
• perf(parquet): improve decoding performance for parquet by @RaphDal in #6759
• feat(sql): array_elem_{min,max,avg,sum} functions by @amunra in #6829
• perf(sql): breaking change 💥 - Improve query execution time by folding constant expressions by @RaphDal in #6828
• fix(sql): fix AssertionError that could be triggered by JOIN SQL by @DHRUV6029 in #6824
• fix(sql): support quoted column names in ALTER COLUMN by @nwoolmer in #6842
• feat(core): parquet row group pruning with min/max statistics and bloom filters by @kafka1991 in #6739
• feat(sql): use vectorized computation in more non-keyed GROUP BY queries by @puzpuzpuz in #6805
• fix(sql): fix WINDOW JOIN INCLUDE PREVAILING dropping prevailing row when window has matches by @RaphDal in #6868
• fix(core): fix read_parquet() crash on SYMBOL columns from native parquet files by @ideoma in #6865
• perf(parquet): breaking change 💥 - improve strings decoding performance for parquet files by @RaphDal in #6809
• perf(sql): improve ORDER BY performance for fixed-width column types by @kafka1991 in #6862
• feat(sql): arg_min and arg_max CHAR functions by @nwoolmer in #6730
• perf(sql): speed up ORDER BY on SYMBOL columns by @kafka1991 in #6870
• fix(sql): WINDOW JOIN dropping prevailing row on cross-partition by @RaphDal in #6871
• perf(sql): optimize parallel HORIZON JOIN for various data distributions by @puzpuzpuz in #6867
• fix(sql): fix bugs, resource leaks and NPE in SQL engine by @bluestreak01 in #6874
• feat(sql): add dynamic window support in WINDOW JOIN by @puzpuzpuz in #6859
• fix(core): fix data corruption in DECIMAL128 and DECIMAL256 columns by @jerrinot in #6873
• feat(core): control parquet writing file size growth, reduce write amplification, improve write speed by @ideoma in #6819
• feat(sql): add per-column parquet encoding/compression config by @RaphDal in #6843
• fix(core): disable materialized view parallel SQL on low-core machines by @nwoolmer in #6891
• fix(sql): fix read_parquet on Parquet files with stale QDB metadata by @nwoolmer in #6885

Full Changelog: 9.3.3...9.3.4