Major Updates

📊 Stats: 15 PRs merged, from 11 contributors

📈 Code diff: 217 files changed, with 18,972 insertions and 1,685 deletions

🧠 Semantic OPs MVP shipped: Introduced a new semantic operations framework with extract and condition filter capabilities, laying the groundwork for join/aggregation/top-k in future releases. [#948]

🤖 Agent interaction quality evaluation: New OPs and recipe for assessing interaction quality, with bad-case HTML report generation, and robust JSONL / HuggingFace meta loading support. [#957]

📄 Cross-document line-level deduplication: A new DocumentLineDeduplicator enables deduplication at the line level across multiple documents — useful for removing boilerplate or repeated content shared between files. [#961]

⚡ Lighter default installation: Reduced the size of default dependencies to significantly speed up first-time installation. [#959]

New OPs

• DocumentLineDeduplicator: Performs cross-document line-level deduplication — ideal for removing shared boilerplate text, repeated headers/footers, or duplicated content across large corpora. [#961]
• semantic_extract_mapper & semantic_condition_filter (MVP): Extract structured fields from unstructured text and filter based on semantic conditions, powered by the new semantic ops framework. [#948]
• Interaction quality OPs (via dj-agents): Evaluate the quality of agent interactions with dedicated scoring operators and a recipe that generates HTML bad-case reports for human review. [#957]

Enhancements

• O(n²) → O(n) in FrequencySpecifiedFieldSelector: Replaced repeated sum() accumulation with itertools.chain, eliminating quadratic behavior on large batches. [#975]
• Exclude work_dir from fingerprint hashing: Cache fingerprints no longer change when work_dir differs, preventing unnecessary recomputation while preserving correct pickling behavior. [#967]
• Reduce default dependency size: Trimmed the default install footprint so new users can get started faster without pulling in heavyweight optional packages. [#959]
• Replace bs4 stub with beautifulsoup4: Switched to the canonical package name to prevent installation conflicts and deprecation warnings. [#977]

Fixed Bugs

• Invalid max_new_tokens injected into API chat completions: The parameter was being incorrectly passed to chat completion API calls that don't support it, causing request failures. Now properly excluded. [#983]
• initialize_ray crash without config: Ray initialization failed when no config was provided. Added proper fallback handling. [#981]
• exit(1) replaced with raise in dataset processing: Using exit() in a library context kills the entire process. Errors now raise proper exceptions for library-safe error handling. [#974]
• PyArrow 20.0.0+ batch JSON reading compatibility: Fixed an issue where open_json batch reading broke under PyArrow 20+. [#942]
• text_keys not propagated to ops in service mode: The DJ service was not passing text_keys through to operators via get_init_configs, causing ops to use wrong keys. [#960]
• Model utils sampling params & API client initialization: Fixed incorrect sampling parameter handling and optimized API client setup in model_utils.py. [#962]
• Aesthetics-predictor normalization check too strict: The substring match for model normalization detection now supports local model paths, not just HuggingFace hub identifiers. [#946]
• Temp dir removal failure in RayExecutor: Added fallback handling when temporary directory cleanup fails (e.g., due to permission issues or concurrent access). [#943]

Acknowledgements

• @dubin555 delivered key performance optimizations and library-safety improvements. [#974] [#975]
• @amanyara contributed the new DocumentLineDeduplicator. [#961]
• @ideal added robustness to RayExecutor temp dir cleanup. [#943]
• @HunterLine fixed PyArrow 20+ batch JSON reading compatibility. [#942]
• @justinwolfington corrected the beautifulsoup4 package dependency. [#977]
• @macroguo-ghy fixed Ray initialization without config. [#981]

New Contributors

• @justinwolfington made their first contribution in [#977]
• @macroguo-ghy made their first contribution in [#981]
• @amanyara made their first contribution in [#961]
• @ideal made their first contribution in [#943]

Full Changelog: https://github.com/datajuicer/data-juicer/compare/v1.5.1...v1.5.2