Redundancy due to cut-paste operations in text creates bias in machine learning for NLP.
This module takes a directory and produces a subset of the files in that directory (in a list) with an upper bound on similarity between two files.

Features

  • Identify copy paste redundancy in a document corpus
  • Input: a folder with text documents and similarity threshold
  • Output (a) a list of non-redundant documents (a non-redundant subset of the corpus)
  • Output (b) list of document pairs found to be redundant with the amount of redundancy for the pair
  • Python script (2.6) - tested on various Linux flavours + Windows XP/7

Project Activity

See All Activity >

Categories

Linguistics, Natural Language Processing (NLP)

License

GNU General Public License version 3.0 (GPLv3)

Follow Corpus redundancy manager

Corpus redundancy manager Web Site

Other Useful Business Software
Stop Storing Third-Party Tokens in Your Database Icon
Stop Storing Third-Party Tokens in Your Database

Auth0 Token Vault handles secure token storage, exchange, and refresh for external providers so you don't have to build it yourself.

Rolling your own OAuth token storage can be a security liability. Token Vault securely stores access and refresh tokens from federated providers and handles exchange and renewal automatically. Connected accounts, refresh exchange, and privileged worker flows included.
Try Auth0 for Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Corpus redundancy manager!

Additional Project Details

Intended Audience

Science/Research

User Interface

Console/Terminal

Programming Language

Python

Related Categories

Python Linguistics Software, Python Natural Language Processing (NLP) Tool

Registered

2011-05-09
Report inappropriate content