Corpus redundancy manager download

Redundancy due to cut-paste operations in text creates bias in machine learning for NLP.
This module takes a directory and produces a subset of the files in that directory (in a list) with an upper bound on similarity between two files.

Features

Identify copy paste redundancy in a document corpus
Input: a folder with text documents and similarity threshold
Output (a) a list of non-redundant documents (a non-redundant subset of the corpus)
Output (b) list of document pairs found to be redundant with the amount of redundancy for the pair
Python script (2.6) - tested on various Linux flavours + Windows XP/7

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Corpus redundancy manager

Corpus redundancy manager Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

The database for AI-powered applications.

MongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.

Start Free

Rate This Project

User Reviews

Be the first to post a review of Corpus redundancy manager!

Additional Project Details

Intended Audience

Science/Research

User Interface

Console/Terminal

Programming Language

Python

Related Categories

Python Linguistics Software, Python Natural Language Processing (NLP) Tool

Registered

2011-05-09

Similar Business Software

ActCAD Software

ActCAD is a native dwg/dxf cad software suitable for professional 2D drafting and 3D modeling projects. ActCAD is trusted by over 30000 users in over 103 countries for more than 10 years. The interface, commands, icons, dialogs, shortcuts etc. are very much similar to other popular cad software...

See Software
LM-Kit.NET

LM-Kit.NET is a cutting-edge, high-level inference SDK designed specifically to bring the advanced capabilities of Large Language Models (LLM) into the C# ecosystem. Tailored for developers working within .NET, LM-Kit.NET provides a comprehensive suite of powerful Generative AI tools, making...

See Software
Dronedesk

Are you wasting hours on drone flight planning? Still using spreadsheets, doc templates, and paper checklists? If so, it's time to switch to Dronedesk, the web-based drone operations management application that makes planning safe drone flights super-efficient. Dronedesk does all the...

See Software
AvPro Software

AvPro Software is comprehensive and easy-to-use. It's perfect for Aircraft MRO, Certified Repair Station (CRS), Aircraft Operators, and parts brokers. You can track Aircraft Parts I(nventory, Work Orders, and much more. Modular in nature and specifically designed for aircraft maintenance...

See Software
Houzz Pro

Houzz Pro is the #1 construction management solution for residential contractors and designers. Get an all-in-one solution that spans the full customer lifecycle, including marketing, CRM, estimates, takeoffs, 3D floor plans, project management, selections, online invoicing & payments,...

See Software
Altium Develop

Altium Develop is a multidisciplinary product creation platform that breaks down silos and empowers teams to design collaboratively without limits. Built on Altium Designer and Altium 365, it unifies electrical, mechanical, software, sourcing, and manufacturing teams in a shared environment....

See Software