Corpus redundancy manager download

Redundancy due to cut-paste operations in text creates bias in machine learning for NLP.
This module takes a directory and produces a subset of the files in that directory (in a list) with an upper bound on similarity between two files.

Features

Identify copy paste redundancy in a document corpus
Input: a folder with text documents and similarity threshold
Output (a) a list of non-redundant documents (a non-redundant subset of the corpus)
Output (b) list of document pairs found to be redundant with the amount of redundancy for the pair
Python script (2.6) - tested on various Linux flavours + Windows XP/7

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Corpus redundancy manager

Corpus redundancy manager Web Site

Other Useful Business Software

Demo Series - Small Business Backup By Veeam

Learn how to protect your Microsoft 365 data, with simple, actionable tips today.

Watch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.

Watch Demo Series

Rate This Project

User Reviews

Be the first to post a review of Corpus redundancy manager!

Additional Project Details

Intended Audience

Science/Research

User Interface

Console/Terminal

Programming Language

Python

Related Categories

Python Linguistics Software, Python Natural Language Processing (NLP) Tool

Registered

2011-05-09

Similar Business Software

LM-Kit.NET

LM-Kit.NET is a complete local AI runtime for .NET that lets engineering teams ship AI-powered features without cloud dependencies, per-token costs, or data leaving the network. Most .NET AI integrations stop at inference. LM-Kit.NET covers the full range of capabilities production...

See Software
kama.ai

The Responsible AI-Agent for organizations where the answer HAS TO BE RIGHT. kama.ai answers from knowledge your team approves, not from what a model guesses. Every response is accurate, consistent, and traceable to its source. Governed by design. How it works: kama.ai is built on a...

See Software
QBench

QBench is a cloud-based Laboratory Information Management System (LIMS) designed to help laboratories manage samples, workflows, data, inventory, reporting, quality processes, and client interactions in one platform. Labs use QBench to manage laboratory operations from order placement and...

See Software
Google AI Studio

Google AI Studio is a unified development platform that helps teams explore, build, and deploy applications using Google’s most advanced AI models. It brings text, image, audio, and video models together in one interactive playground. With vibe coding, developers can use natural language to...

See Software
Lockbox LIMS

A sample tracking, test result capture, and inventory management cloud LIMS for life science research, biotech/NGS, and industrial QC labs. Includes regulatory support for CLIA, HIPAA, Part 11, and ISO 17025. Nothing is more critical to a lab’s success than the quality, security, and...

See Software
SAP S/4HANA Cloud Public Edition

SAP Cloud ERP is the premier ERP solution for growth-focused organizations. Seamlessly integrating AI, and predictive analytics, it empowers businesses to digitally transform and streamline processes end to end. Leveraging built-in industry best practices, SAP Cloud ERP accelerates...

See Software

Report inappropriate content