Major Updates

📊 Stats: 18 PRs merged, from 5 contributors

📈 Code diff: 84 files changed, with 11,991 insertions and 389 deletions

🧩 External operator plugins — Support loading external operators via Python entry points, allowing third-party OPs to be shipped as independent pip packages and used with zero extra config. [#1026]

🗄️ HDFS protocol support — Data-Juicer can now read datasets from and export processed results to HDFS (hdfs://), consistent with the existing S3 remote-storage workflow. [#1014]

⚡ Leaner & faster Ray Data — Removed eager count()/columns()/schema() actions that triggered premature Ray Dataset execution, replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs, and made PartitionedRayExecutor run partitions concurrently. [#1022] [#1024] [#1025] [#1032] [#1036]

🌐 Elastic multi-node sharding — Available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor, including retries, stale-claim recovery, status inspection, and ordered merge. [#1015]

📊 RayAnalyzer for distributed analysis — New RayAnalyzer computes filter stats via Ray map_batches and aggregates overall statistics (count/mean/std/min/max) using Ray native aggregation operators, without pandas materialization. It is the distributed counterpart to the local Analyzer. [#1016]

Enhancements

• External operator plugins via entry points: third-party OPs can be shipped as independent pip packages and are loaded into the global OPERATORS registry at import data_juicer.ops time. [#1026]
• HDFS protocol support for dataset loading and exporting. [#1014]
• Public Ray compute strategies: replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs. [#1024]
• Reduced eager Ray dataset actions: removed premature count(), switched to take(k)/take_all(), and avoided repeated columns()/schema() calls. [#1025]
• Elastic multi-node sharding: available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor (retries, stale-claim recovery, status inspection, and ordered merge). [#1015]
• RayAnalyzer for distributed data analysis: computes filter stats via Ray map_batches and aggregates overall statistics without pandas materialization. [#1016]
• Membership operators (in/not in) in general_field_filter for expressions like lang in ['en', 'zh']. [#1000]
• Streamed n-gram frequency counting: count character and word n-grams as they are generated instead of retaining an occurrence-sized intermediate list, cutting peak memory in character_repetition_filter and word_repetition_filter while preserving ratios, caching, and filtering decisions. [#1029]
• Bounded MinHash permutation workspace: reduce token-by-permutation matrices in 8 MiB blocks and fold exact minima in document_minhash_deduplicator, cutting RSS from 768.0 to 271.9 MiB at 4x while preserving byte signatures. [#1035]
• Test refactor & coverage: enhanced tests for base_op, job utils, and config. [#1018]

Fixed Bugs

• Concurrent partitioned Ray execution: PartitionedRayExecutor now drives partitions concurrently via a driver-side ThreadPoolExecutor instead of serially, preserving partition order. [#1022]
• Elastic actor pool concurrency: ActorPoolStrategy now honors (min, max) num_proc ranges instead of collapsing them to a fixed size. [#1032]
• Partition checkpoint preservation: partition checkpoints now survive Ray block-layout changes, so resumed runs no longer reprocess completed partitions. [#1036]
• text_chunk_mapper visible delimiters: split_pattern no longer emits visible delimiters (e.g. _@@@_) as standalone chunks. [#999]
• calibrate_response_mapper output pattern: now honors output_pattern for regex extraction instead of ignoring it. [#1017]
• image_diffusion_mapper null captions: handles missing or empty caption fields gracefully instead of crashing. [#1020]
• Silent no-op on empty OP list: DefaultExecutor and RayExecutor now warn when the process operator list is empty. [#1020]
• pandas 3 incompatibility with Ray Data: pinned pandas<3 in the distributed extra to avoid AttributeError during JSON export. [#1023]
• pyarrow>=17 compatibility: bumped fsspec accordingly. [#1020]
• Python 3.13+ dependency resolution: removed the numpy<2.0 upper bound (blocked tf-keras/ml-dtypes needing numpy>=2.1.0) and upgraded label-studio to >=1.23.0. [#1027]

Acknowledgements

• @LiuGuH contributed HDFS protocol support for dataset loading and exporting. [#1014]
• @macroguo-ghy contributed streamed n-gram frequency counting and a bounded MinHash permutation workspace to cut peak memory. [#1029] [#1035]

New Contributors

• @LiuGuH made their first contribution in [#1014]

Full Changelog: https://github.com/datajuicer/data-juicer/compare/v1.5.4...v1.5.5