Search Results for "corpora"

...The repository aims to make datasets easy to inspect and transform, with scripts for downloading, deduping, cleaning, and converting to formats like JSONL that slot into training pipelines. It highlights instruction-tuning and conversation-style corpora while also pointing to code, math, or domain-specific sets for targeted capabilities. Quality is a recurring theme: examples and utilities help filter low-value samples, enforce length limits, and split train/validation consistently so results are comparable. Licensing and provenance are surfaced to encourage compliant usage and to guide dataset selection in commercial settings. ...

...The repository gathers several major datasets, including Chinese Wikipedia entries, news articles, encyclopedia-style question answering data, community question answering data, and Chinese-English translation sentence pairs. Each dataset includes descriptions, download links, structure notes, and examples to help users understand how the data is formatted. The corpora can support tasks such as language model pretraining, word vector training, question answering, title generation, keyword generation, translation, and sentence representation learning. Overall, it is a practical resource hub for building or testing Chinese NLP models with larger and more varied datasets.

...This project now contains fully functional Concordia search library. In the near future, it will be extended by concordia-server: ligthweight, robust web server providing corpora search functionalities

...It was created to make multilingual vector representations easier to obtain, especially for languages with fewer readily available resources than English. The repository provides models trained using both Word2Vec and fastText. Training corpora are constructed from Wikipedia database dumps using language-specific preprocessing when necessary. Scripts are included for corpus creation and for training new embeddings with either supported algorithm. Available languages include Chinese, Japanese, Korean, Spanish, French, German, Hindi, Russian, Vietnamese, Thai, and many others, with model metadata covering vector, corpus, and vocabulary sizes.

This repository is a curated collection of Chinese poems and poets organized into catalogs, metadata, and text representations suitable for research, creative and cultural use. It includes major dynastic corpora, such as Tang and Song poems, as well as biographical and categorization data. Each poem entry is structured with fields like author, dynasty, title, content, and sometimes annotations or alternate versions. Developers and scholars can build tools that query by author, era, keyword, or poetic form using the standardized data structure. ...

RC-Data is a dataset generation framework created by Google DeepMind to produce large-scale reading comprehension question-answer pairs from CNN and Daily Mail news articles. The dataset, introduced in the 2015 paper “Teaching Machines to Read and Comprehend” (Hermann et al., NIPS 2015), was among the first large corpora designed to train and evaluate machine reading and comprehension models. The repository provides scripts for downloading archived CNN and Daily Mail articles from the Wayback Machine and automatically generating cloze-style questions where entities in the text are replaced with placeholders. Each data instance consists of a news article (context), a generated question, and its corresponding answer, making it suitable for supervised machine learning setups. ...

...Also, it comes with a WordNet integration. If you only intend to use TextBlob’s default models (no model overrides), you can pass the lite argument. This downloads only those corpora needed for basic functionality. TextBlob is also available as a conda package.