Datasets Files

Bug fixes

• Fix version string in init.py by @qgallouedec in https://github.com/huggingface/datasets/pull/8244
• fix conda build by @lhoestq in https://github.com/huggingface/datasets/pull/8250
• Fix JSON loader schema inference for files starting with a UTF-8 BOM (#8241) by @archievi in https://github.com/huggingface/datasets/pull/8243
• Support hermes traces by @lhoestq in https://github.com/huggingface/datasets/pull/8255
• Fix batch(by_column=...) crashing after shard/shuffle/split by @pkooij in https://github.com/huggingface/datasets/pull/8259
• fix traces streaming by @lhoestq in https://github.com/huggingface/datasets/pull/8277
• support droid agent traces by @cfahlgren1 in https://github.com/huggingface/datasets/pull/8263
• Fix CI: commit operation equality (hfh 1.20.0) and pytest parametrize collection error by @Wauplin in https://github.com/huggingface/datasets/pull/8283
• Fix lance auth by @lhoestq in https://github.com/huggingface/datasets/pull/8301
• Fix symlink-following arbitrary file write in archive extraction by @AAtomical in https://github.com/huggingface/datasets/pull/8303
• Bug Fix: Resuming Twice Resets the Dataloader by @francesco-bertolotti in https://github.com/huggingface/datasets/pull/8295
• Bump fsspec and simpler wds compr by @lhoestq in https://github.com/huggingface/datasets/pull/8337
• fix: validate Arrow IPC record batches by @XciD in https://github.com/huggingface/datasets/pull/8350
• Make the dataset fingerprint independent of Arrow chunking by @SuryanshSS1011 in https://github.com/huggingface/datasets/pull/8339
• Fix casting a nullable LargeList to a different inner type by @vineethsaivs in https://github.com/huggingface/datasets/pull/8346
• docs: replace AutoFeatureExtractor with AutoImageProcessor in image preprocessing docs by @gautamkishore in https://github.com/huggingface/datasets/pull/8326
• Fix column drop in Arrow path of axis=1 concatenation by @ebarkhordar in https://github.com/huggingface/datasets/pull/8342
• Raise on length mismatch in batched IterableDataset.map by @sohumt123 in https://github.com/huggingface/datasets/pull/8332
• Fix require_storage_embed recursing into require_storage_cast by @vineethsaivs in https://github.com/huggingface/datasets/pull/8349
• Fix path traversal via metadata file_name in folder-based builders by @Kaif10 in https://github.com/huggingface/datasets/pull/8325
• Support batched=True in Dataset.to_dict by @vineethsaivs in https://github.com/huggingface/datasets/pull/8333
• Fix hdf5 external files by @lhoestq in https://github.com/huggingface/datasets/pull/8355
• remove bad require_storage test by @lhoestq in https://github.com/huggingface/datasets/pull/8357
• ensure fiels are in repo by @lhoestq in https://github.com/huggingface/datasets/pull/8356
• Keep flat numeric columns with nulls numeric in numpy format by @ebarkhordar in https://github.com/huggingface/datasets/pull/8352
• Fix bucket dataset card handling and push metadata accounting by @pjh4993 in https://github.com/huggingface/datasets/pull/8354
• Fix CSV loader dropping on_bad_lines/encoding_errors on pandas 2.0-2.2 by @ebarkhordar in https://github.com/huggingface/datasets/pull/8358
• Keep integers on the python read path for fixed-shape ArrayXD columns with nulls by @ebarkhordar in https://github.com/huggingface/datasets/pull/8363
• Rebatch arrow source before formatting in IterableDataset.filter to fix resume data loss by @ebarkhordar in https://github.com/huggingface/datasets/pull/8360
• Decode Json() columns in Dataset.to_pandas() by @ebarkhordar in https://github.com/huggingface/datasets/pull/8344
• fix buckets on windows by @lhoestq in https://github.com/huggingface/datasets/pull/8369
• Fix DatasetDict.push_to_hub leaving removed splits in the dataset card by @pjh4993 in https://github.com/huggingface/datasets/pull/8367
• Preserve nullable integer columns in to_json/to_csv/to_sql by @ebarkhordar in https://github.com/huggingface/datasets/pull/8366

Docs

• docs: fix duplicate "to" in IterableDataset push-to-hub example by @DaoyuanLi2816 in https://github.com/huggingface/datasets/pull/8252
• Clarify dataset creation vs loading workflows in create_dataset tutorial by @zanvari in https://github.com/huggingface/datasets/pull/8235

New Contributors

• @DaoyuanLi2816 made their first contribution in https://github.com/huggingface/datasets/pull/8252
• @zanvari made their first contribution in https://github.com/huggingface/datasets/pull/8235
• @archievi made their first contribution in https://github.com/huggingface/datasets/pull/8243
• @pkooij made their first contribution in https://github.com/huggingface/datasets/pull/8259
• @AAtomical made their first contribution in https://github.com/huggingface/datasets/pull/8303
• @francesco-bertolotti made their first contribution in https://github.com/huggingface/datasets/pull/8295
• @XciD made their first contribution in https://github.com/huggingface/datasets/pull/8350
• @SuryanshSS1011 made their first contribution in https://github.com/huggingface/datasets/pull/8339
• @vineethsaivs made their first contribution in https://github.com/huggingface/datasets/pull/8346
• @gautamkishore made their first contribution in https://github.com/huggingface/datasets/pull/8326
• @ebarkhordar made their first contribution in https://github.com/huggingface/datasets/pull/8342
• @sohumt123 made their first contribution in https://github.com/huggingface/datasets/pull/8332
• @Kaif10 made their first contribution in https://github.com/huggingface/datasets/pull/8325
• @pjh4993 made their first contribution in https://github.com/huggingface/datasets/pull/8354

Full Changelog: https://github.com/huggingface/datasets/compare/5.0.0...5.0.1