Daft Files

What's Changed 🚀

💥 Breaking Changes

• feat!: add iceberg table support with the gravitino catalog @rchowell (#6509)

✨ Features

• feat!: add iceberg table support with the gravitino catalog @rchowell (#6509)
• feat(swordfish): plan caching @colin-ho (#6278)
• feat: add daft-checkpoint crate with CheckpointStore trait and in-mem… @rohitkulshreshtha (#6511)
• feat: optionally keep the native progress bar visible after job completion @BABTUNA (#6449)
• feat(build): add experimental cross-wheel build target @cckellogg (#6360)
• feat: DataFrame.from_files @rchowell (#6507)
• feat: Add Apache Paimon Lake Format Support @chenghuichen (#6450)
• feat: add current_date, current_timestamp, current_timezone and SQL aliases @BABTUNA (#6495)
• feat(subscribers): add typed event for operator and stats callbacks @cckellogg (#6479)
• feat: support to_ray_dataset() from the native runner @desmondcheongzx (#6486)
• feat: async data sources and tasks @rchowell (#6492)
• feat: Add hex and unhex @Lucas61000 (#6373)
• feat: implement _repr_mimebundle_ for DataFrame @Abyss-lord (#6385)
• feat: Add string strip function @Lucas61000 (#6372)
• feat: add plan fingerprinting for LocalPhysicalPlan @colin-ho (#6276)
• feat: Add metrics for join operators @srilman (#6391)
• feat(observability): add process-level memory and CPU monitoring @desmondcheongzx (#6428)
• feat: define a single DataSource trait matching our python interface. @rchowell (#6427)
• feat(io): Propagate storage_options to Lance backend to enable Apache Gravitino (gvfs://) support @qingfeng-occ (#6430)
• feat(subscribers): add JSONL event log subscriber @cckellogg (#6420)
• feat: Add support for reading whole text files to read_text @plotor (#6354)
• feat(expressions): support count(mode='all') without expr @Abyss-lord (#6358)
• feat: kafka bounded datasource @everySympathy (#5970)
• feat(udf): support ray_options and resource overrides in UDF v2 @Jay-ju (#5982)
• feat: Add global agg support for Dataframe select @caican00 (#6067)

🐛 Bug Fixes

• fix: correct grid card indentation on modalities overview page @everettVT (#6530)
• fix(series): unable to cast directly from string to fixed_size_binary @Abyss-lord (#6521)
• fix: respect scheme aliases in Iceberg writes @rchowell (#6512)
• fix(series): cast should not change input argument name @Abyss-lord (#6515)
• fix: preserve show options without setting format @aaron-ang (#6102)
• fix: strict inequalities must NOT be relaxed for Identity transform @gavin9402 (#6484)
• fix: fix column missing issue for jsonl record schema when reading json/jsonl files @caican00 (#5681)
• fix: remove use_safetensors=True to allow model fallback @rchowell (#6489)
• fix: close connections in io tests and address other warnings @rchowell (#6470)
• fix: deflake a probabilistic test that is causing release uploads to fail @rchowell (#6475)
• fix(io): fix JSONL byte-range reading and make chunk_size byte-based @everySympathy (#6374)
• fix(sql): fall back to single scan when partition bounds are degenerate @qingfeng-occ (#6482)
• fix: clippy errors on iterators in tests @rchowell (#6476)
• fix: handles schema inference errors when partition columns are NULL @rchowell (#6467)
• fix: keep field metadata when converting series to ffi schema @universalmind303 (#6471)
• fix: Track durations for all local operators in distributed @srilman (#6438)
• fix(optimizer): allow column-alias projections in join reordering @desmondcheongzx (#6412)
• fix: Expose json_target_filesize execution config @plotor (#6382)
• fix: fix for bad merge @universalmind303 (#6419)
• fix(sql): Fix error in TPC-DS q23 @Lucas61000 (#6317)
• fix: Preserve materialized stats when splitting granular projections @plotor (#6376)
• fix(dataframe): preserve field order in from_pylist @Abyss-lord (#6369)
• fix: report bytes.read for local CSV scans @Abyss-lord (#6380)

🚀 Performance

• perf: Optimize list_* aggregation functions @srilman (#6494)
• perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch @universalmind303 (#6444)

♻️ Refactor

• refactor: implement generic shuffle interface for flight shuffle @ohbh (#6456)
• refactor(ext): use single, publishable crate for daft-ext types @rchowell (#6452)
• refactor(optimizer): introduce bitmask-based RelationSet for join reordering @desmondcheongzx (#6421)
• refactor(io): merge conditional arms in opendal_backends ls operator @qingfeng-occ (#6461)
• refactor(io): move common source fields to own type @rchowell (#6451)
• refactor: remove EmptyScan plan node @colin-ho (#6275)
• refactor(swordfish): Streaming flight shuffle read @colin-ho (#6269)

📖 Documentation

• docs: clean up roadmap wording @ykdojo (#6517)
• docs: update roadmap for 2026 @everettVT (#6480)
• docs: fix daft dashboard command example @qingfeng-occ (#6483)
• docs: document get_or_infer_runner_type and get_or_create_runner @ykdojo (#6453)
• docs(kafka): add dedicated connector docs for kafka @everySympathy (#6433)
• docs: improve skills install instructions @ykdojo (#6422)
• docs: add Daft Skills page and Claude Code plugin config @ykdojo (#6368)

👷 CI

• ci: disable HF xet storage backend in unit tests @rchowell (#6493)
• ci: refresh apt index before installing deps @rchowell (#6491)
• ci: remove broken links @rchowell (#6490)

🔧 Maintenance

• chore: exclude symlink files from end-of-file-fixer pre-commit hook @chenghuichen (#6531)
• chore(ci): migrate BuildJet runners to Blacksmith @madvart (#6416)
• chore: use dedicated OSS AWS account @jeevb (#6442)
• chore: Collection of random Swordfish cleanups @srilman (#6454)
• chore(swordfish): Small swordfish cleanups @colin-ho (#6488)
• chore: use concurrent broken link checker @rchowell (#6474)
• chore: remove _FixEmptyStructArrays workaround @universalmind303 (#6478)
• chore: deprecate all extra lance functions in favor of new daft-lance @universalmind303 (#6487)
• chore: remove deprecated catalog modules @rchowell (#6457)
• chore: Disable cargo check in local dev @srilman (#6439)
• chore: Abstract metrics meters for execution @srilman (#6384)
• chore: Remove some extra \& duplicate Rust packages @srilman (#6435)
• chore: Make the partition threshold to use pre_shuffle_merge configurable @plotor (#6356)
• chore: rename DataSource enum to ScanSource @rchowell (#6417)
• chore: separates source types from file format configuration @rchowell (#6406)

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.7.5...v0.7.6