Download Latest Version 6.1.1 source code.tar.gz (254.0 MB)
Email in envelope

Get an email when there's a new version of Spark NLP

Home / 6.1.0
Name Modified Size InfoDownloads / Week
Parent folder
6.1.0 source code.tar.gz 2025-07-23 253.3 MB
0
6.1.0 source code.zip 2025-07-23 434.9 MB
0
README.md 2025-07-23 7.0 kB
0
Totals: 3 Items   688.3 MB 0

📢 Spark NLP 6.1.0: State-of-the-art LLM Capabilities and Advancing Universal Ingestion

We are excited to announce Spark NLP 6.1.0, another milestone for building scalable, distributed AI pipelines! This major release significantly enhances our capabilities for state-of-the-art multimodal and large language models and universal data ingestion. Upgrade Spark NLP to 6.1.0 to improve both usability and performance across ingestion, inference, and multimodal processing pipelines, all within the native Spark ecosystem.

🔥 Highlights

  • Upgraded llama.cpp Integration: We've updated our llama.cpp backend to tag b5932 which supports inference with the latest generation of LLMs.
  • Unified Document Ingestion with Reader2Doc: Introducing a new annotator that streamlines the process of loading and integrating diverse file formats (PDFs, Word, Excel, PowerPoint, HTML, Text, Email, Markdown) directly into Spark NLP pipelines with a unified and flexible interface.
  • Support for Phi-4: Spark NLP now natively supports the Phi-4 model, allowing users to leverage its advanced reasoning capabilities.

🚀 New Features & Enhancements

Large Language Models (LLMs)

  • llama.cpp Upgrade: Our llama.cpp backend has been upgraded to version b5932. This update enables native inference for the newest LLMs, such as Gemma 3 and Phi-4, ensuring broader model compatibility and improved performance.
  • NOTE: We are still in the process of upgrading our multimodal AutoGGUFVisionModel annotator to the latest backend. This means that this annotator will not be available in this version. As a workaround, please use version 6.0.5 of Spark NLP.
  • Phi-4 Model Support: Spark NLP now integrates the Phi-4 model, an advanced open model trained on a blend of synthetic data, filtered public domain content, and academic Q&A datasets. This integration enables sophisticated reasoning capabilities directly within Spark NLP. (Link to notebook)

Document Ingestion

  • Reader2Doc Annotator: This new annotator provides a simplified, unified interface for integrating various Spark NLP readers. It supports a wide range of formats, including PDFs, plain text, HTML, Word (.doc/.docx), Excel (.xls/.xlsx), PowerPoint (.ppt/.pptx), email files (.eml, .msg), and Markdown (.md).
  • Using this annotator, you can read all these different formats into Spark NLP documents, making them directly accessible in all your Spark NLP pipelines. This significantly reduces boilerplate code and enhances flexibility in data loading workflows, making it easier to scale and switch between data sources.

Let's use a code example to see how easy it is to use:

:::python
reader2doc = Reader2Doc() \
    .setContentType("application/pdf") \
    .setContentPath("./pdf-files") \
    .setOutputCol("document")

# other NLP stages in `nlp_stages`

pipeline = Pipeline(stages=[reader2doc] + nlp_stages)
model = pipeline.fit(empty_df)
result_df = model.transform(empty_df)

Check out our full example notebook to see it in action.

🐛 Bug Fixes

  • HuggingFace OpenVINO Notebook for Qwen2VL: Addressed and fixed issues in the notebook related to the OpenVINO conversion of the Qwen2VL model, ensuring smoother functionality.

❤️ Community Support

  • Slack: For live discussion with the Spark NLP community and the team.
  • GitHub: Bug reports, feature requests, and contributions.
  • Discussions: Engage with other community members, share ideas, and show off how you use Spark NLP!
  • Medium: Spark NLP articles.
  • JohnSnowLabs official Medium
  • YouTube: Spark NLP video tutorials.

Installation

Python

:::shell
pip install spark-nlp==6.1.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

:::shell
spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.0

GPU

:::shell
spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.0

Apple Silicon (M1 & M2)

:::shell
spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.0

AArch64

:::shell
spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

:::xml
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-gpu:

:::xml
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-silicon:

:::xml
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-aarch64:

:::xml
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

FAT JARs

What's Changed

  • Update HuggingFace_OpenVINO_in_Spark_NLP_Qwen2VL.ipynb [#14631] by @AbdullahMubeenAnwar
  • Sparknlp 1189 Introducing Phi4 [#14606] by @prabod
  • SPARKNLP-1259 Introducing Reader2Doc Annotator [#14632] by @danilojsl
  • [SPARKNLP-1194] Upgrade jsl-llamacpp to newest version [#14633] by @DevinTDHa
  • Add telemetry to github actions [skip-test] [#14568] by @KshitizGIT

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.5...6.1.0

Source: README.md, updated 2025-07-23