Dragonfly v1.37.0

Important stability fixes and changes:

• Significant memory reduction around json documents [#6511]
• Vector search memory usage is 40% less with hash map documents: [#6436] [#6437]
• Support of SORT command - including BY and GET options.
• Several bug fixes around streams: [#6506] [#6492] [#6532]
• Added support for DIGEST and DELEX commands: [#6328]

What's Changed

• feat(search): Use stateless allocator for strings by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6285
• chore: Refactor squashed command execution to avoid stub ConnectionContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6312
• refactor: add checks to investigate the race condition in slots migration by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6331
• fix(helm): correct security context field placement and disable strict schema validation by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6213
• fix(tests): Use promtool for metrics sanity check by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6318
• fix(search): support parenthesized star in FT.SEARCH queries by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6343
• fix: helm push issue by @romange in https://github.com/dragonflydb/dragonfly/pull/6339
• chore: move ReplyBuilder calls in generic_family.cc to new API by @romange in https://github.com/dragonflydb/dragonfly/pull/6305
• Add checks for expiration inconsistency by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6346
• feat(hscan): add NOVALUES flag to omit values by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6325
• chore: refactor SortGeneric function by @romange in https://github.com/dragonflydb/dragonfly/pull/6347
• feat: add pagination to cross-shard search by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6352
• refactor: address comments by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6350
• fix(set): SPOP random member even if it contains null bytes by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6361
• fix(server): Set error type when returning -NOGROUP error by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6358
• chore: improvements around sorting algorithm by @romange in https://github.com/dragonflydb/dragonfly/pull/6360
• facade: adopt RespSrvParser in server connections by @romange in https://github.com/dragonflydb/dragonfly/pull/6356
• feat(server): Support squashing of global eval with single shard by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6321
• fix: memory calculation for OAHSet by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6359
• fix: regtest failure by @romange in https://github.com/dragonflydb/dragonfly/pull/6367
• test: use hiredis for reply_builder_test by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6362
• ci: use OIDC for AWS authentication instead of access keys by @AdiCloudbit-DevOps in https://github.com/dragonflydb/dragonfly/pull/6158
• refactor: improve OAHSet::Find() performance by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6373
• chore(server): Simplify InvokeCmd by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6363
• chore(cmake): Update jsoncons to 1.5.0 by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6349
• chore: encapsulate tx object within CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6384
• fix(server): support "sort by" option by @romange in https://github.com/dragonflydb/dragonfly/pull/6370
• refactor(acl): from facade to server by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6340
• chore: add copilot env setup steps by @romange in https://github.com/dragonflydb/dragonfly/pull/6387
• fix(ci): add AWS env fallbacks for OIDC in composite actions by @AdiCloudbit-DevOps in https://github.com/dragonflydb/dragonfly/pull/6385
• chore: move CommandContext cid to private scope by @romange in https://github.com/dragonflydb/dragonfly/pull/6386
• chore(server): Updates around DispatchMC/DispatchCommand by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6351
• fix: compatibility with cpp20 by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6395
• chore: Update CommandContext c'tor by @romange in https://github.com/dragonflydb/dragonfly/pull/6393
• feat(server): Add GET option support to SORT command by @romange in https://github.com/dragonflydb/dragonfly/pull/6388
• chore: update AGENTS.md by @romange in https://github.com/dragonflydb/dragonfly/pull/6392
• Switch to C++20 by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6379
• chore: Clean up DispatchCommand a bit more. by @romange in https://github.com/dragonflydb/dragonfly/pull/6397
• fix(core): Remove direct access to RObjWrapper by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6140
• fix: copilot-setup-steps.yml by @romange in https://github.com/dragonflydb/dragonfly/pull/6405
• Implement DIGEST and DELEX commands (Redis 8.4.0 compatible) by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6328
• chore: tx clean-ups by @romange in https://github.com/dragonflydb/dragonfly/pull/6407
• chore: remove quicklist from the codebase by @romange in https://github.com/dragonflydb/dragonfly/pull/6415
• chore(core): CompactObj is_key field, separate encoding by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6253
• chore: wrap list operations within ListWrapper by @romange in https://github.com/dragonflydb/dragonfly/pull/6417
• fix: s3 snapshot saving by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6400
• chore(facade): Refactor IoLoop to new interface by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6383
• chore: clarify IterateList interface by @romange in https://github.com/dragonflydb/dragonfly/pull/6425
• chore: show slave_repl_offset on replicas when master link is down by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6353
• fix: reloading vector index by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6422
• chore(search): Simplify SFINAE usage with concepts in defragment-map by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6428
• fix: static assert on macos by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6430
• docs: add async-tiering.md by @romange in https://github.com/dragonflydb/dragonfly/pull/6427
• chore(core): Move expire flag to key by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6420
• chore: add ListPack class that wraps c listpack by @romange in https://github.com/dragonflydb/dragonfly/pull/6432
• chore: factor out ContainerEntry and QList::Entry into a common class by @romange in https://github.com/dragonflydb/dragonfly/pull/6426
• refactor: InvokeCmd by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6399
• fix: partial sync from same source master with different shard sizes by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6381
• feat: add hnsw metadata into index serialization by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6429
• fix: QList::iterator semantics by @romange in https://github.com/dragonflydb/dragonfly/pull/6439
• tests: Check if tx is initialized before accessing its state by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6446
• chore: support listpack with lists by @romange in https://github.com/dragonflydb/dragonfly/pull/6424
• Use async tx for ioloopv2/set by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6433
• feat(server): add protected mode to bind localhost when no password is set by @rafaelperoco in https://github.com/dragonflydb/dragonfly/pull/6442
• tests: Replace pytest with gh workflow by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6453
• CI: remove faulty condition by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6458
• chore(server): Try to simplify build by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6455
• chore: create smalls lists as listpacks by @romange in https://github.com/dragonflydb/dragonfly/pull/6450
• chore(tiering): polish type system for better clarity and safety by @romange in https://github.com/dragonflydb/dragonfly/pull/6465
• chore: unify auth errors under the same type by @romange in https://github.com/dragonflydb/dragonfly/pull/6449
• fix(server): ignore SIGHUP to prevent termination on session closing by @boomballa in https://github.com/dragonflydb/dragonfly/pull/6463
• chore(tiering): refactor stashing to determine serialization upfront by @romange in https://github.com/dragonflydb/dragonfly/pull/6470
• Add index serialization by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6461
• docs: add code review instructions for Copilot agents by @romange in https://github.com/dragonflydb/dragonfly/pull/6471
• feat(hnswlib): Support using pointer to vector data by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6436
• chore(tiering): refactor Read/Modify to free functions by @romange in https://github.com/dragonflydb/dragonfly/pull/6469
• design: stateful async handlers by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6462
• chore: better naming for compression functions in qlist.cc by @romange in https://github.com/dragonflydb/dragonfly/pull/6474
• feat(search): Build vector index by using data pointer for HSET objects by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6437
• refactor(facade): decouple pipeline traffic from administrative command queue (phase 1) by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6423
• chore: bump helio to fix TLS socket crash by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6480
• feat(search): Omit key defrag is vector data is used as reference by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6454
• chore: sync rax and stream code with Valkey by @romange in https://github.com/dragonflydb/dragonfly/pull/6473
• feat(cluster): add range syntax support to DFLYCLUSTER GETSLOTINFO by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6485
• chore: add tiering params and stats to qlist by @romange in https://github.com/dragonflydb/dragonfly/pull/6476
• feat(json): Adds a ref-counted blob by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6456
• fix(server): Fix concurrent access to now_streamed_ by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6486
• chore: add more invariant checks for streams (#6459) by @romange in https://github.com/dragonflydb/dragonfly/pull/6488
• fix(search): GEO units shouldn't be tokens by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6489
• fix: prevent connected_clients counter leak on replica connection migration by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6491
• feat(json): Adds a reference counted string class by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6497
• fix(tiering): implement qlist symmetric cooloff and fix memory handling by @romange in https://github.com/dragonflydb/dragonfly/pull/6496
• chore: Allow JustReply() in cmd-support by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6478
• fix(server): Async index builder, part 1 by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6504
• feat: add shrink to OAHSet by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6513
• feat: send index definition for every thread by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6505
• fix(cmd): Ensure arg slice lifetime by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6514
• fix(hset): delete key if empty after HEXPIRE by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6503
• chore: Add MinIO support for S3 snapshot tests by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6501
• fix: xreadgroup is a write command by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6492
• fix: add missing memory tracking on streams by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6506
• fix: xreadgroup read_history should not stick by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6523
• feat(search): Async index building (part 2) by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6517
• refactor(facade): separate redis data/control paths (phase 2, final) by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6477
• feat: add serialization for shard DocId mapping by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6519
• refactor: simplify OAHSet code by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6522
• fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6526
• feat(hnswlib): Use __builtin_prefetch by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6500
• fix: QUIT pipeline flush by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6518
• fix: allow lazy expiry during global EVAL transactions by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6524
• tests: filter up BullMQ failed tests. by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6533
• fix(tiering): Handle string that are encoded as LONG LONG objects by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6530
• feat(core/search): Range Tree builder by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6520
• fix: always replicate new consumers in xreadgroup by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6532
• fix(facade): fix migration stats accounting crash by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6538
• fix(search): Fix search test after async index construction by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6541
• feat(json): Use reference counted strings for json keys by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6511
• fix: search and dfly lib circular dep by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6543
• fix(search): Avoid second traversal without vector fields by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6539
• chore: reduce includes by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6464
• fix(search): Handle missing fields with GeoIndex by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6550
• feat(server): Plug in RangeTree::Builder by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6537
• fix: hnsw global serialization and add flag for hnsw save\load by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6553
• json: Add test for pool shrinking by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6544
• Remove redundant includes to reduce compilation time by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6548
• fix(search): fix more tests by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6547
• test: search with replication by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6563
• feat(hnsw): Use cosine distance by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6559
• chore: supress maint_notifications client errors by @romange in https://github.com/dragonflydb/dragonfly/pull/6569
• Reduce compilation time by removing unnecessary header includes by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6562
• Migrate INCR/DECR/INCRBY/DECRBY command family to AsyncCmd architecture by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6568
• feat(search): Handle GEO index fields by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6566
• fix(search): Missing vector field should not be zero vector by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6565
• refactor(string): migrate SET, APPEND, PREPEND to async SimpleContext by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6580
• core/server: Compute memory size of json object using jsoncons API by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6572
• fix: validate interleaved-keys argument count by stored step by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6574
• build: add optional ASAN support for release builds by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6590
• refactor: store hnsw metadata in separate AUX field by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6587
• chore(server): Move vector index construction to builder by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6571
• chore: get rid of deprecated DFLY_PASSWORD env checks by @romange in https://github.com/dragonflydb/dragonfly/pull/6586
• fix: reject GAT command sent via Redis protocol instead of crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6592
• fix: handle MEMORY USAGE without key argument instead of crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6597
• server: Change metric type to counter to match redis exporter by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6581
• fix: reject SCRIPT FLAGS with invalid SHA length instead of crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6603
• feat(server): add CLIENT INFO command by @romange in https://github.com/dragonflydb/dragonfly/pull/6576
• fix(tls): ub in HandleRequests after first socket read by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6594
• Consolidate SlowLog tests in server_family_test by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6585
• Refactor server/common.h - Phase 1: Extract common types and statistics by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6582
• test: wait for async indexing completion in test_replicate_all_index_types by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6593
• fix(hnsw): Avoid possible HSET object memory misalignment by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6577
• chore: simplify snapshotting serialization callback by @romange in https://github.com/dragonflydb/dragonfly/pull/6595
• server: Record values in histogram atomically by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6583
• Refactor server/common.h - Phase 2: Extract execution state types by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6605
• Extract RespExprBuilder to shared test utils and migrate test_utils to RESPParser with RESP3 support by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6588
• fix: mimalloc page reclamation is too slow by @romange in https://github.com/dragonflydb/dragonfly/pull/6575
• feat: add HNSW index deserialization by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6606
• Refactor server/common.h - Phase 3: Extract synchronization primitives by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6612
• Refactor bitops/stream updater storage by @romange in https://github.com/dragonflydb/dragonfly/pull/6613
• fix: make ChangeMonitor idempotent to prevent double-add in MULTI/EXEC by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6615
• test: Improve AFL++ fuzzing infrastructure by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6618
• fix: crash on EVALSHA with empty sha by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6620
• fix: string corruption during rdb load by @romange in https://github.com/dragonflydb/dragonfly/pull/6621
• make migration more stable by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6616
• chore(regtests): replication missmatching dbnum by @ashotland in https://github.com/dragonflydb/dragonfly/pull/6607
• fix(server): fix OpenSSL shutdown race with mimalloc by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6626
• fix(fuzz): fix false-green CI and simplify fuzzing workflow by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6632
• fix(server): guard ReadProcStats with linux for macOS compatibility by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6633
• fix: do not overrun stack for asan enabled builds by @romange in https://github.com/dragonflydb/dragonfly/pull/6635
• fix(tiering): Fix replication serialization order by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6610
• fix(memcache): reject gat/gats without key instead of crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6639
• feat(server): Extensive slowlog stats for exec/eval by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6584
• fix(facade): Fix missing DFLY_USE_SSL defines by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6636
• fix(memcache): cap store value size with max_bulk_len by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6644
• chore: move simple types to common_types.h and reduce tx_base.h header includes by 52.9% by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6641
• fix(facade): Disable v2 loop for redis by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6643
• test: Add memcache protocol fuzzing support by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6638
• fix(rdb): fix chunked SBF loading for filters >= 4 GiB by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6649
• chore: journal clean-ups by @romange in https://github.com/dragonflydb/dragonfly/pull/6622
• fix: test_redis_om by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6655
• fix(tiering): Reduce load on tiering test by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6650
• chore: add logs to rdb_load by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6642
• Load hnsw index by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6647
• fix(facade): use WriteRef for large values in meta get response by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6657
• refactor: include what you use by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6651
• skip tiering test by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6658
• fix(server): Wait for index load on replication by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6645
• refactor: include what you use by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6660
• feat(tiering): SSD tiering support during slot migration by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6542
• refactor: remove extra includes from headers by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6663
• chore: introduce heavy tests by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6659
• chore: Mark cluster tiering test as heavy by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6671
• fix: backward compatibility for replication regarding HNSW index seri… by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6664

Huge thanks to all the contributors! ❤️

New Contributors

• @AdiCloudbit-DevOps made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6158
• @rafaelperoco made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6442

Full Changelog: https://github.com/dragonflydb/dragonfly/compare/v1.36.0...v1.37.0