Daft Files

What's Changed 🚀

✨ Features

• feat: add File.to_tempfile method and optimize range requests @universalmind303 (#5226)
• feat: migrate nested constructors, get, slice, to_unix_epoch, partitioning functions @kevinzwang (#5233)
• feat: implement Lance filter+count pushdown optimization @huleilei (#5152)
• feat: migrate remaining basic expressions @kevinzwang (#5234)
• feat: Add regexp_count @srilman (#5191)
• feat: retry eof error in S3 multipart upload @kevinzwang (#5179)
• feat(flotilla): Distributed Pivot @srilman (#5199)
• feat: allow for functions module to be accessed from daft import @kevinzwang (#5219)
• feat: migrate first batch of daft functions @kevinzwang (#5086)
• fix: remove io_config from global scan operator @Jay-ju (#4676)
• feat: implement cross join in distributed flotilla engine @ohbh (#5180)
• feat: make multipart part size configurable @stayrascal (#5051)

🐛 Bug Fixes

• fix: write_iceberg should handle write.data.path @eric-maynard (#5153)
• fix: Cast to table schema in lance write @colin-ho (#5221)
• fix: rename index function to find @kevinzwang (#5218)
• fix: remove unnecessary field renaming in ListMap scalar UDF @ltdthanhdat (#5205)
• fix: remove io_config from global scan operator @Jay-ju (#4676)

🚀 Performance

• perf: Drop local limit tasks global limit reached @colin-ho (#5194)

📖 Documentation

• docs: add file datatype to urls\&files docs page @universalmind303 (#5240)
• docs: add pipeline visualization and fix grammar in batch inference docs @ykdojo (#5239)
• docs: add Examples link to getting started sections @ykdojo (#5202)
• docs: fix additional incorrect references to Iceberg as a catalog @ykdojo (#5198)
• docs: clarify data catalog vs table format in README @ykdojo (#5197)
• docs: remove outdated engine comparison page @ykdojo (#5196)
• docs: Add embed_image to the user guide @desmondcheongzx (#5195)
• docs: update minhash dedupe example and tutorial @everettVT (#5192)
• docs: Add Minhash Dedupe Example @everettVT (#5165)

👷 CI

• ci: enable AI integration tests in PR @kevinzwang (#5217)
• ci: move transformers tests to integration tests @rchowell (#5186)

🔧 Maintenance

• chore: Don't log failures in url download if on_error="null" @colin-ho (#5231)
• chore: Make the runner a separate global singleton @srilman (#5185)
• chore: remove expr.str.regexp_split since we will immediately deprecate it @kevinzwang (#5216)
• chore: Rewrite the dashboard server to use Axum @srilman (#5212)
• chore: Remove reset_runner option @srilman (#5184)
• chore: Split .str.split into .str.split and .str.regexp_split @srilman (#5211)
• chore: enable logging configuration during launch udf worker @stayrascal (#5168)

⏪ Reverts

• revert: "fix: remove io_config from global scan operator" @universalmind303 (#5203)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.6.1...v0.6.2