Dragonfly v1.32.0

Important stability fixes and changes:

• Lots of performance improvements around Search [#5341]#5327 [#5350] [#5545] [#5533]
• Add optional latency percentile stats [#5372] and [#5428]
• Update mimalloc version to 2.24
• Added BZMPOP and BLMPOP commands [#5436] and [#5370]
• Added CONFIG REWRITE command [#5471]

What's Changed

• test(search): add unified benchmarks for prefix/suffix/infix search by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5319
• chore: get rid of quicklist encoding by @romange in https://github.com/dragonflydb/dragonfly/pull/5282
• feat: add flag pause_wait_timeout by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5328
• test: enable test_multidim_knn python test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5329
• test: make random determenistic in test_multidim_knn python test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5335
• fix: data race when doing a snapshot in cache mode by @romange in https://github.com/dragonflydb/dragonfly/pull/5336
• chore(search_family): Minor performance improvements to GetAllResults methods. FIRST PR by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5341
• fix: JSON.DEL to return 0 for non-existing keys instead of error by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5343
• chore: improve test_full_sync test by @romange in https://github.com/dragonflydb/dragonfly/pull/5340
• chore: Refactor CompactObject encoding by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5342
• feat: Added Sort_RO Command. by @H4R5H1T-007 in https://github.com/dragonflydb/dragonfly/pull/5332
• chore: avoid deduplicating docs ids when not needed by @romange in https://github.com/dragonflydb/dragonfly/pull/5344
• chore: add hdr_histogram to the project by @romange in https://github.com/dragonflydb/dragonfly/pull/5330
• chore(tiering_test): tune memory settings to be below 6GB by @romange in https://github.com/dragonflydb/dragonfly/pull/5345
• chore: suffix search by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5327
• test: Vector operations benchmark was added by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5353
• feat(replication): Aggregate data until threshold before replication by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5308
• SimSIMD integration for Vector operations by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5348
• feat (server) : introduce no point in time replication by @adiholden in https://github.com/dragonflydb/dragonfly/pull/5103
• chore: move 3rd party libraries into a separate cmake file by @romange in https://github.com/dragonflydb/dragonfly/pull/5362
• fix(pytest): Increase allowed diff in test_partial_replication_on_same_source_master by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5363
• fix(replication): reduce ERROR logging during normal shutdown by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5366
• chore: remove unneeded mimalloc-specific code by @romange in https://github.com/dragonflydb/dragonfly/pull/5367
• chore: add mimalloc2 third_party library by @romange in https://github.com/dragonflydb/dragonfly/pull/5364
• fix(replication): reduce ERROR to WARNING for connection issues during sync by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5369
• feat(search): Suffix/Infix search for Tags by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5350
• fix(zset): expand parsing of rank arguments beyond int32 by @romange in https://github.com/dragonflydb/dragonfly/pull/5377
• test: replication_test.py::test_bug_in_json_memory_tracking enabled by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5375
• chore: Disable test_replication_all cache mode by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5385
• chore: Make fakeredis part of the core CI checks by @romange in https://github.com/dragonflydb/dragonfly/pull/5380
• fix(zset_family): Return early if range is expected to be zero by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5386
• fix: JSON.STRLEN behavior for non-existent keys by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5389
• chore: add psync metrics for replica by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5365
• fix(server): fix serialization on move with big values by @adiholden in https://github.com/dragonflydb/dragonfly/pull/5390
• feat(pytest): Set and log random seed used in pytest by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5391
• chore: remove zset/t_zset files by @romange in https://github.com/dragonflydb/dragonfly/pull/5387
• feat: add support for BLMPOP command by @EricHayter in https://github.com/dragonflydb/dragonfly/pull/5370
• feat(pytest): Initialize random seed for each test by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5395
• fix(values): fix typos in Helm README and values file by @adimoragbloch in https://github.com/dragonflydb/dragonfly/pull/5400
• fix: no set/hmap expiration during serialization by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5349
• feat(search_family): Add SortedVector<pair\<DocId, double>> by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5405
• feat: add snapshot-id field into dfs files by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5399
• chore: Enable point_in_time_snapshot by default by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5403
• chore: Increase BLMPopBlocking sleep duration by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5408
• chore: skip flaky test_pipeline_cache_size by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5410
• chore: introduce class scope fixtures by @romange in https://github.com/dragonflydb/dragonfly/pull/5409
• fix(pytest): Increase REPLTAKOVER time in test_take_over_seeder by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5415
• server_family: Expose command latency stats as part of info by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5372
• fix(search): Cut off results before serializing by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5412
• test: Add Dragonfly search benchmark tests by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5406
• feat(search_family): Introduce RangeTree by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5414
• fix: search_benchmark_utils on python 3.12 by @romange in https://github.com/dragonflydb/dragonfly/pull/5422
• feat(search_family): Add RangeTree to NumericIndex by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5416
• chore: switch to mimalloc2 by @romange in https://github.com/dragonflydb/dragonfly/pull/5368
• feat(search_family): Add optimized merging logic for one and two blocks in RangeResult by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5417
• chore(range_tree): Simplify RangeTree code by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5418
• Feat config management by @boomballa in https://github.com/dragonflydb/dragonfly/pull/5388
• Lua library Integer overflow can cause the DragonFly crash. (CVE-2020-14147) by @KIMDONGYEON00 in https://github.com/dragonflydb/dragonfly/pull/5421
• chore(search): LIMIT=0 Fast path and remove LOAD parameter from FT.SEARCH by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5419
• server_family: Support resetting latency histogram by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5428
• test: search benchmark updated to use class fixture for multiple tests by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5425
• chore: harden checks in _mi_page_malloc_zero by @romange in https://github.com/dragonflydb/dragonfly/pull/5427
• fix(pytest): Wait replication to be in sync in test_stream_approximate_trimming by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5430
• chore: avoid squashing when squashing_current_reply_size crosses limit by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/4924
• fix: acl log missing return after reply by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5454
• chore: add acl help by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5456
• chore: allow initializing a separate huffman table for string values by @romange in https://github.com/dragonflydb/dragonfly/pull/5470
• feat(search_family): Introduce SingleBlockRangeResult and TwoBlocksRangeResult by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5424
• chore: add tls_caching flags by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5485
• chore: add more logs during replication by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5489
• chore(set_family): Fast path SPOP with small count by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5491
• fix: add a sleep to relax slot migration process by @romange in https://github.com/dragonflydb/dragonfly/pull/5478
• chore: add tls_socket accept error metric by @romange in https://github.com/dragonflydb/dragonfly/pull/5492
• chore: add io_read_cnt metric by @romange in https://github.com/dragonflydb/dragonfly/pull/5494
• chore: add tls handshake statistics by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5493
• feat: add more auxillary strings to the snapshot by @romange in https://github.com/dragonflydb/dragonfly/pull/5411
• chore: log frequency for IsReplySizeOverLimit by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5501
• fix: adjust BLMPOP command options and implementation for proper journaling by @EricHayter in https://github.com/dragonflydb/dragonfly/pull/5503
• chore: add more metrics by @romange in https://github.com/dragonflydb/dragonfly/pull/5504
• fix: allow eviction of zero-allocation entries in cache mode by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5500
• server: Make memory defrag blocking, accept optional threshold by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5498
• feat: add debug segments command by @romange in https://github.com/dragonflydb/dragonfly/pull/5505
• fix: prevent idle disconnects for replica flow connections by @romange in https://github.com/dragonflydb/dragonfly/pull/5506
• chore(search): Simplify return field logic by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5486
• fix: lua_undeclared_keys_shas updated for additional sha by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5511
• chore: replace size() > 0 with !empty() by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5499
• chore: add additional runtime configuration knobs for connections by @romange in https://github.com/dragonflydb/dragonfly/pull/5520
• fix: reduce cardinality of the huffman histogram by @romange in https://github.com/dragonflydb/dragonfly/pull/5518
• fix: enable keep alive on replica side of replication connections by @romange in https://github.com/dragonflydb/dragonfly/pull/5519
• chore: flush after serializing big string by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5401
• refactor: debug populate stream generation by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5407
• chore(search_family): Simplify logic for numeric indexes. FIRST PR by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5522
• chore: skip test_replication_onmove_flow by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5526
• fix: Prevent INFO command deadlock during S3 save operations by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5528
• chore: measure p99.9 in the replayer by @romange in https://github.com/dragonflydb/dragonfly/pull/5530
• chore: -time-limit option was added to record traffic tool by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5532
• chore: add p999 metric for realtime tables as well by @romange in https://github.com/dragonflydb/dragonfly/pull/5531
• chore: fix monitor preemption in atomic section by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5523
• feat(search_family): Add BLOCKSIZE option for FT.CREATE by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5423
• fix(search_test): Add kMaxRangeBlockSize to the SeveralNumericAndTagIndexes benchmark by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5541
• feat(search_family): Speed up merging for numeric indexes by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5533
• chore: lower mem usage of test_big_strings by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5540
• chore(tiering): Push error handling one more level up by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5351
• chore: Add cmake build options to exclude code [#1] by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5536
• feat: add RESP3 support for JSON.TYPE and JSON numeric operations by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5517
• fix(search): Fix nullable sort out of bounds by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5539
• core,server: collect page usage stats during defragmentation task by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5502
• chore: Add more stats to CLIENT LIST by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5553
• chore: Refactor JournalReader::ReadString by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5550
• chore(server): Some memory code updates by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5379
• fix: FT.CREATE first-query initialization penalty (slowdown) by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5473
• feat: allow migrating client connections via CLIENT MIGRATE by @romange in https://github.com/dragonflydb/dragonfly/pull/5551
• fix(acl): preemptive TlsSocket::Shutdown in atomic section by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5525
• feat(config): Implement CONFIG REWRITE command by @boomballa in https://github.com/dragonflydb/dragonfly/pull/5471
• feat(traffic_replay): Add compare option by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5555
• feat(cluster): migrate thread on same shard config by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5554
• chore: use stringmatchlen in GlobMatcher by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5547
• chore: excessive logging in dragonfly connection by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5558
• feat(search): Add IP (Inner Product) distance metric support for vector similarity by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5559
• fix(tiering): Fix bug in TryGrow by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5563
• fix: Allow CLUSTER commands during LOADING state by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5566
• server: Check result of hdr_init by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5568
• chore: pull helio by @romange in https://github.com/dragonflydb/dragonfly/pull/5557
• fix: Add enhanced error logging and TCP diagnostics for JournalStreamer disconnects by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5572
• Feat: Added BZMPOP Command by @H4R5H1T-007 in https://github.com/dragonflydb/dragonfly/pull/5436
• chore: time sensitive sscan by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5527
• chore: do not shutdown master on takeover in cluster mode by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5544
• feat: remove crc calculation for migration by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5577
• chore: Don't use point_in_time_replication = False in test_replication_all by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5574
• fix(server): Make max_memory_flag atomic by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5576
• fix(tiering): Fix rdb load backpressure by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5578
• docker: Remove cmake dir from dockerfile which no longer exists, add patches dir by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5575
• feat(tiering): Error propagation to client (Part 3) by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5573
• chore: pull helio by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5567
• fix: temporary set ignore_crc=false during serialization for backward compatibility by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5583
• feat(tiering): Error propagation to client (Part 2) by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5565
• chore: do not enter loading state during partial sync by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/5579
• chore: Refactor CmdSerializer by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/5584
• chore(cluster): Decommit memory only under flag by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5588
• chore: update helio to fix build by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5593
• feat: Add LinearSearchMap by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5591
• core,server: report defrag metrics by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5560
• fix: streamer.cc build on macos by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5594
• docs(build): fix missing -D flag in minimal debug build command by @guozhihao-224 in https://github.com/dragonflydb/dragonfly/pull/5600
• chore: update daily builds notifications by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/5599
• feat: introduce a CPU budget parameter to control slot migration pacing by @romange in https://github.com/dragonflydb/dragonfly/pull/5598
• chore: extend DEBUG TOPK/KEYS to show total samples count by @romange in https://github.com/dragonflydb/dragonfly/pull/5605
• feat(search_family): Speed up merging of index results by @BagritsevichStepan in https://github.com/dragonflydb/dragonfly/pull/5545
• fix(tiering): Fix MacOs build by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5602
• chore: add support for value length distribution sampling by @romange in https://github.com/dragonflydb/dragonfly/pull/5606
• feat: fix lua memory deallocation and lua_mem_gc_threshold=0 disable lua force GC calls by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/5587

Huge thanks to all the contributors! ❤️

New Contributors

• @EricHayter made their first contribution in https://github.com/dragonflydb/dragonfly/pull/5370
• @adimoragbloch made their first contribution in https://github.com/dragonflydb/dragonfly/pull/5400
• @boomballa made their first contribution in https://github.com/dragonflydb/dragonfly/pull/5388
• @guozhihao-224 made their first contribution in https://github.com/dragonflydb/dragonfly/pull/5600

Full Changelog: https://github.com/dragonflydb/dragonfly/compare/v1.31.0...v1.32.0