QSV Files

[20.1.0] - 2026-05-18 🤖 The "Synthetic Data" Release 🎲

A feature-packed minor release headlined by a brand-new synthesize command for generating realistic fake CSV data, a much smarter describegpt that can now describe what your columns mean (not just their data types), and new "approximate stats" modes that let stats and frequency keep working on files that are bigger than your computer's memory. No breaking changes — pipelines built on 20.0.0 will upgrade in place.

Highlights

• 🆕 synthesize — generate realistic fake CSVs from a real one. Point it at a source file and it produces a new CSV of any size whose columns look and behave like the original — same value mix, same distribution shape, same null rate — but without any of the original records. Useful for sharing test data, populating staging environments, or building demos without leaking real customer data.
• Categorical columns (e.g. country, status) are rebuilt by sampling the real values in the same proportions they appear.
• Numeric and date columns preserve the shape of the distribution, not just the min/max — so the synthetic data has realistic clusters, not a flat random spread.
• Null rates are matched per column.
• --seed makes output fully reproducible — same seed, same file, every time.
• --dictionary / --infer-content-type plugs in the new describegpt Content Types (see next bullet) so columns recognized as e.g. email, phone, city, or credit_card are filled with realistic-looking fakes instead of generic random strings. --locale picks from 14 regional flavors (US, FR, JP, etc.) so the fakes match your audience.

• Cross-column correlations (e.g. keeping city ↔ zip_code consistent within a row) aren't modeled by default — but turning on describegpt's --two-pass option (see next bullet) lets the LLM detect related fields, and synthesize will then keep those relationships consistent in the generated rows.

• 🧠 describegpt got a lot smarter — it can now label what your columns mean. In addition to qsv's existing type detection (Integer, Float, Date, etc.), describegpt can now ask an LLM to classify each column with a semantic label from a 47-token vocabulary covering people, addresses, companies, technical identifiers, and more — so a column of strings isn't just "String", it's email, street_address, job_title, or credit_card. These labels are what powers synthesize's realistic fakes, but they're also useful on their own as auto-generated data dictionaries.

• --two-pass runs the LLM a second time over the ENTIRE Data Dictionary so it can spot relationships between columns (e.g. "this is a state_abbr because the next column is a zip_code") and fix sloppy first-pass labels. This is also what unlocks cross-column consistency in synthesize (see previous bullet).
• Deterministic unique_id tag — columns where every value is unique (like IDs and UUIDs) are tagged by qsv directly, before the LLM ever sees them. That means the label is 100% reproducible and doesn't drift between LLM versions.
• Smarter time/duration handling — duration columns can carry a realistic upper bound (e.g. "0–1 hour") so synthetic latency or TTL values stay believable instead of ranging out to absurd numbers.
• --markdown-template lets you customize the generated Data Dictionary's Markdown output — add your team's review checklist, restructure the per-column layout, whatever fits your docs.

• Lower LLM costs — the default prompts were restructured to stop re-sending the dictionary on every step, measurably cutting token usage on multi-phase runs.

• 📊 Approximate stats for huge files — stats and frequency no longer give up when a file is bigger than your RAM. New opt-in modes use Apache DataSketches algorithms that compute approximate-but-bounded-error answers in a tiny fraction of the memory. Three new modes across two commands:

• For stats: --quantile-method tdigest for approximate percentiles (t-digest) and --cardinality-method hll for approximate distinct counts (HyperLogLog).
• For frequency: --sketch-method misra-gries for approximate top-K most-frequent values (Misra-Gries Frequent Items).
• Automatic when you'd otherwise OOM: if qsv detects the file is too big to fit in memory, it now auto-switches to the approximate modes (and tells you which ones), instead of failing. Pass --quantile-method exact (etc.) to force the precise calculation regardless.
• Cache stays correct: the stats cache key now includes the chosen mode, so switching between exact and approximate modes won't accidentally return stale results.
• Note: these modes require a "little-endian" CPU, which covers all common hardware (Intel, AMD, Apple Silicon, ARM, etc.). Exotic platforms like IBM s390x get a clear error message instead.

Detailed MCP Server and Cowork Plugin changes are documented in the MCP Server/Cowork Plugin CHANGELOG.

Added

• synthesize: new top-level command (see Headline) #3854
• synthesize: --consistent-fakes for stable source→fake mapping #3865
• synthesize: --locale option for 14 fake-rs locales #3860
• describegpt: --two-pass cross-field Data Dictionary refinement #3863
• describegpt: deterministic unique_id Content Type for ALL_UNIQUE fields #3862
• describegpt,synthesize: infer Content Type for temporal fields with LLM-hinted duration cap #3861
• describegpt,synthesize: 5 new Content Type tokens — street_name, license_plate, industry, profession, ipv6_address
• describegpt: --markdown-template for customizable Markdown output #3834
• pivotp: --agg quantile@<p> (alias q@<p>) with linear interpolation #3842
• stats/frequency: opt-in Apache DataSketches modes — HLL cardinality, Frequent Items top-K #3840
• stats: widened BLAKE3 fingerprint to cover all streaming stats #3824

Changed

• stats/frequency: auto-enable Apache DataSketches estimators (t-digest + HyperLogLog for stats; Misra-Gries Frequent Items for frequency) when util::mem_file_check reports OOM, in addition to the existing auto-index fallback. A wwarn! is emitted listing the auto-enabled estimators; explicit --quantile-method exact / --cardinality-method exact / --sketch-method exact still suppresses the auto-enable #3843
• stats: three opt-in micro-optimizations — simdutf8 output, t-digest quantiles, mode-cardinality cap #3839
• synthesize: use string-length stats for unstructured text columns #3864
• describegpt: inline {{ dictionary }} in default description/tags prompts; skip redundant chat-message dictionary injection when the template already inlines it
• synthesize: handle both describegpt-wrapped and raw dictionary JSON
• refactor: adopt Rust 1.95 cfg_select! macro at platform-conditional sites #3846
• perf: promote bytes_to_cow_str helper to util and sweep callsites
• perf(moarstats): hint rare branches with core::hint::cold_path() #3823
• perf(stats): mark non-UTF-8 branch cold
• perf(frequency): hint UTF-8 failure as cold in the ignore-case hot loop #3821
• refactor(stats): shrink and tidy WhichStats #3822
• refactor(publish): fetch tags and enforce SemVer for debian package releases
• refactor(benchmarks): harden benchmarks.sh error handling and cross-platform support #3814
• deps: bump polars (latest upstream), calamine 0.34→0.35, csvlens fork with bumped arrow, sysinfo 0.38.4→0.39.2, rust_decimal 1.41→1.42, tokio 1.52.1→1.52.3, filetime 0.2.27→0.2.29, jsonschema 0.46.4→0.46.5, rand_xoshiro 0.8.0→0.8.1, redis 1.2.0→1.2.1, qsv-dateparser 0.14→0.15 (adds support for ISO 8601 T-separated datetimes without a timezone suffix — e.g. 2020-01-15T08:00:00, the form produced by Python's datetime.isoformat() without astimezone(); previously misclassified by qsv stats --infer-dates as String)
• assorted clippy cleanups across stats, frequency, pivotp, partition

Fixed

• stats: preserve length & lex stats when column type widens to String #3856
• stats: remove duplicate big-endian TDigestStub/HllSketchStub defs #3857
• stats: restore big-endian build by giving slot fallbacks an accessible .0 #3850
• stats/frequency: gate Apache DataSketches behind little-endian targets #3847
• apply/applydp: thousands negative fractions; scope <NULL> to regex_replace #3845
• moarstats: retry on stats coverage mismatch + fsync joined CSV parent dir #3838
• moarstats: close fsync race that silently dropped joined columns on macOS #3830
• util: open subprocess output with write access for fsync (Windows) #3831
• qsvdp: only list commands actually compiled into the binary #3816 #3819
• help-md-gen: infer real argument type for Options "Type" column #3858 #3859
• synthesize: Date/DateTime columns now always use build_date() with the source's real min/max bounds — previously, if the LLM (incorrectly) tagged a date column with a faker-mapped content_type like time (which is time-of-day, e.g. 14:30:45) or any other temporal token, either faker branch in ColumnGenerator::build() could fire before the type-based match and emit a time-of-day string for a date column. Suppress any LLM-emitted content_type for Date/DateTime columns at function entry so both faker branches fall through to the type-based fallback (regression test in tests/test_synthesize.rs::synthesize_date_column_ignores_time_content_type)
• test(fetch,sample): bind to ephemeral port to fix flaky macOS CI #3827
• test(moarstats): serially execute some flaky CI tests; add missing serial_test::serial import
• test(stats): fix s390x big-endian quantile-method rejection test

Docs

• AI Policy section added to README.md and CONTRIBUTING.md, with cross-links and contributor attribution guidance
• docs-drift CI check added; audit-detected drift fixed #3868
• README emoji legend audited and normalized; help docs regenerated #3832
• "Processing Very Large Files" guidance added; large-file recipe inline-comment fixes
• stats: explicit Count Reference tables for 47 stats / 55 moarstats measures; count conventions clarified
• STATS_DEFINITIONS audit; stats DEVELOPER NOTE wordsmithed
• features: corrected self_update probability + nightly sub-features documentation #3833
• Test count updated to the verified exact total (3,094)

Full Changelog: https://github.com/dathere/qsv/compare/20.0.0...20.1.0