Search Results for "deduplication"

DOLMA (Data Optimization and Learning for Model Alignment) is a framework designed to manage large-scale datasets for training and fine-tuning language models efficiently.

AutoCVE is an agent-driven platform for automating authorized source-code auditing and CVE discovery. It covers the workflow from project selection and repository import to vulnerability verification and submission-ready report generation. An orchestrator coordinates specialized agents for reconnaissance, scanning, triage, deep analysis, verification, and final consolidation. Users can choose enhanced scanning, intelligent auditing, or comprehensive auditing based on speed and depth...

NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline...