Download Latest Version 5.0.0 source code.zip (28.6 MB)
Email in envelope

Get an email when there's a new version of Datasets

Home / 5.0.0
Name Modified Size InfoDownloads / Week
Parent folder
5.0.0 source code.tar.gz 2026-06-05 28.5 MB
0
5.0.0 source code.zip 2026-06-05 28.6 MB
3 3 weekly downloads
README.md 2026-06-05 7.5 kB
0
Totals: 3 Items   57.1 MB 3

Datasets Features

Agent traces

  • Parse Agent traces messages for SFT using teich by @lhoestq in https://github.com/huggingface/datasets/pull/8232
  • Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset
  • Using the teich library (new optional dependency), traces are parsed to messages to enable training on traces using e.g. trl
  • Load the data:

```python

from datasets import load_dataset ds = load_dataset("lhoestq/agent-traces-example", split="train") ds[0]["messages"] [{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...' ...] ```

  • Train on agent traces:

bash trl sft --dataset-name lhoestq/agent-traces-example ...

Next-level shuffling in streaming mode

  • Use multiple input shards for shuffle buffer by @lhoestq in https://github.com/huggingface/datasets/pull/8194

    ```python ds = load_dataset(..., streaming=True) ds = ds.shuffle(seed=42)

    or configure local buffer shuffling manually, default is:

    ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10) ```

    befoređź‘Ž: image

    after✨: image

    toy example comparison

    ```python from datasets import IterableDataset

    ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024) ds = ds.shuffle(seed=42)

    print("Cold start ids:") print(list(ds.take(10)["i"])) print("Nominal regime ids:") print(list(ds.skip(10_000).take(10)["i"])) ```

    befoređź‘Ž:

    Cold start ids: [6148853, 6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858] Nominal regime ids: [6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858, 6149290]

    after✨:

    Cold start ids: [7836668, 9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871] Nominal regime ids: [9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871, 16758448]

    Note: ds.state_dict() and ds.load_state_dict() are still supported for this improved shuffling :) enabling dataset checkpointing

    Note 2: it uses threads to fetch the first examples in parallel from the input shards

    Note 3: This is a BREAKING CHANGE: the default shuffling mechanism now uses multiple input shards. You can get the old mechanism by passing max_buffer_input_shards=1 to IterableDataset.shuffle()

New batching features for robotics datasets

  • Add batch(by_column=...) by @lhoestq in https://github.com/huggingface/datasets/pull/8172

    ```python from datasets import Dataset

    ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2})

    ds = ds.to_iterable_dataset()

    ds = ds.batch(by_column="episode") for x in ds: print(x)

    {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}

    {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}

    ```

New supported formats

Other improvements and bug fixes

New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.5...5.0.0

Source: README.md, updated 2026-06-05