Major Updates

📊 Stats: 23 PRs merged, from 9 contributors

📈 Code diff: 197 files changed, with 9,415 insertions and 1,741 deletions

🧃 Juicer model release — Released Juicer-35B-A3B (ModelScope), a locally deployable data-refinement model that follows natural-language instructions for text cleaning, filtering, and semantic labeling. English and Chinese guides and the Juicer Playground provide deployment instructions and example recipes. [#1060]

🧮 Cluster-aware partitioning — Automatic partition counts now use live Ray cluster resources. Manual partition.size targets split data at row boundaries, including inputs with fewer blocks than partitions. [#1044] [#1045] [#1063]

✅ Config validation — Pipeline preflight catches invalid operator settings and executor/schema mismatches before processing. Reader defaults now apply consistently across execution and analysis. [#1049] [#1061]

🔌 LiteLLM backend — New api_backend="litellm" support in prepare_api_model routes chat, embedding, and Responses requests through LiteLLM providers while retaining the existing OpenAI-compatible backend as the default. [#1062]

📚 Documentation refresh — Rewritten English and Chinese guides cover installation, processing, analysis, configuration, export, and the playground. Added documentation for 28 existing operators, corrected examples, and separated guide/API navigation with incremental versioned builds. [#1048]

🗄️ Unified remote export — Local, S3, and HDFS export share filesystem dispatch, and JSONL output now serializes Python dates and datetimes in ISO format. [#1021] [#1065]

🖼️ Image OHEM selector — New image_ohem_selector selects high-loss image samples using a user-supplied scoring function and a top-k or ratio budget. [#1038]

⚡ Bounded tokenizer batches — Token-count filters limit tokenizer batch sizes to reduce peak memory on long inputs. [#1040]

🔧 Robustness fixes — Fixed fused-filter cache isolation, MinHash state reuse and empty inputs, deduplicator execution-mode declarations, text chunking, compressed HPO sampling, and pandas extension-dtype handling. [#1059] [#1043] [#1051] [#1057] [#1046] [#1055] [#1047]

Enhancements

• Cluster-aware auto partition counts: partition.num_of_partitions: auto derives partition counts from live Ray cluster resources. [#1045]
• Single Ray topology source: consolidated resource detection and removed guessed node counts and driver-local clamping in partitioned execution. [#1044]
• Manual partition sample targets: partition.size specifies an approximate number of rows per partition; explicit count and sample-target settings are validated as alternatives. [#1063]
• Pipeline preflight validation: validate operator parameters, executor compatibility, and required dataset fields before processing across default, Ray, and partitioned Ray execution. [#1049]
• Consistent reader defaults: dataset loading and analysis share reader configuration, with explicit load arguments taking precedence over recipe defaults and np. Exposed the existing load_jsonl_lenient and use_dag controls in YAML/CLI. [#1061]
• LiteLLM provider integration: chat, embedding, and Responses wrappers support provider-specific routing and credentials through the opt-in LiteLLM backend. [#1062]
• Unified remote filesystem dispatch: Exporter and RayExporter share path resolution for local files, S3, and HDFS. [#1021]
• Bounded tokenizer batches: cap token-count tokenizer batch sizes while preserving filtering behavior. [#1040]
• Bilingual guides and operator reference: rewrite task guides, document 28 existing operators, fix examples and links, and update guide/API navigation and versioned documentation builds. [#1048]
• Juicer model documentation: added English and Chinese guides for the separately announced Juicer model (ModelScope) and playground. [#1060]
• Maintenance forks: updated recognize-anything and transformers-stream-generator references to maintained personal forks. [#1056]
• Dependency and CI updates: bumped NLTK to 3.10.3 and updated checkout/setup-python action majors in the pre-commit workflow. [#1066] [#1067]

New OPs

• Image OHEM selector: image_ohem_selector keeps the highest-loss image samples, with callable or file-based scoring functions, an optional model factory, and top-k or ratio selection. [#1038]

Fixed Bugs

• Fused-filter context cache isolation: cache keys now include the text key and tokenizer configuration, preventing statistics from being reused across different columns or tokenization settings. [#1059]
• MinHash stale hash tables: reset hash tables between process() calls so an operator instance can process independent datasets without carrying over previous state. [#1043]
• MinHash empty token sets: return an all-maximum signature instead of crashing on an empty reduction; zero-token samples share a signature and are deduplicated together. [#1051]
• Deduplicator execution modes: correctly declare local-only and Ray support so incompatible executor/operator combinations are rejected. [#1057]
• JSONL date serialization: Python date and datetime values are serialized with isoformat() instead of raising TypeError. [#1065]
• Extensionless export paths: report how to provide a file suffix or explicit export_type when the output path has no extension. [#1053]
• Empty text chunks: recursively_chunk no longer emits empty fragments. [#1046]
• HPO gzip JSONL sampling: read compressed statistics through gzip and clean up temporary samples even when analysis fails. [#1055]
• pandas extension dtypes: use pandas' numeric-dtype predicate for analysis compatibility. [#1047]
• Ray JSON reader options: forward configured read options consistently, preserve disabled reader threading by default, and fix the older-PyArrow fallback. [#1061]
• Global configuration consistency: remove unused intermediate_storage.*, preserve_intermediate_data, resource_optimization.auto_configure, max_log_size_mb, and backup_count declarations; remove the stale add_suffix example; reject unsupported checkpoint strategies and nonpositive intervals. [#1061]
• Partition-size configuration: connect the sample target to row-based splitting, handle single-partition and remainder cases, and remove the unused max_partition_size_mb field. The legacy partition_size remains accepted with a deprecation warning; partition.target_size_mb is a planning estimate rather than a hard size limit. [#1063]
• Import-safe download and HPO tooling: correct package imports in the arXiv downloader and HPO entry point; importing the W&B HPO module no longer starts a sweep. [#1048]
• Regression test discovery and assertions: remove the preflight test's import-path override and align MinHash/Ray regression cases with current behavior, including runtime-environment restoration after failed retries.

Acknowledgements

• @nofear-spec contributed the image OHEM selector for hard-example mining. [#1038]
• @macroguo-ghy contributed bounded tokenizer batching to reduce peak memory. [#1040]
• @mameikagou fixed gzip JSONL statistics sampling for HPO. [#1055]
• @arkrolin fixed fused-filter context cache isolation. [#1059]
• @prodmanpd contributed LiteLLM provider integration. [#1062]
• @JohnGiorgi fixed date and datetime serialization in JSONL exports. [#1065]
• @katsugtgz updated NLTK and the pre-commit workflow actions. [#1066] [#1067]

New Contributors

• @nofear-spec made their first contribution in [#1038]
• @mameikagou made their first contribution in [#1055]
• @arkrolin made their first contribution in [#1059]
• @prodmanpd made their first contribution in [#1062]
• @katsugtgz made their first contribution in [#1066]

Full Changelog: https://github.com/datajuicer/data-juicer/compare/v1.5.5...v1.6.0