Sail Files

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: rustup profile for Dockerfile by @linhr in https://github.com/lakehq/sail/pull/2042
• docs: update contact info by @linhr in https://github.com/lakehq/sail/pull/2044
• chore: upgrade Codecov action by @linhr in https://github.com/lakehq/sail/pull/2045
• docs: remove AI writing patterns from prose by @pomykalakyle in https://github.com/lakehq/sail/pull/1980
• feat: catalog managed Delta Lake table by @lonless9 in https://github.com/lakehq/sail/pull/1957
• feat: catalog managed Iceberg table by @lonless9 in https://github.com/lakehq/sail/pull/1929
• refactor: move data source options by @linhr in https://github.com/lakehq/sail/pull/2047
• deps(npm): bump the all group with 4 updates by @dependabot[bot] in https://github.com/lakehq/sail/pull/2048
• refactor: logical planning for data source writer by @linhr in https://github.com/lakehq/sail/pull/2050
• chore: upgrade to DataFusion 54.0.0 and Arrow 58.3.0 by @davidlghellin in https://github.com/lakehq/sail/pull/1956
• feat: initial support for named window clause in SQL by @ctydw123 in https://github.com/lakehq/sail/pull/2039
• feat: add window function by @radmirnovii in https://github.com/lakehq/sail/pull/2006
• fix: parity with various funcs by @shehabgamin in https://github.com/lakehq/sail/pull/2054
• chore: use #!/usr/bin/env bash shebang in shell scripts by @davidlghellin in https://github.com/lakehq/sail/pull/2062
• fix: skip Spark Connect tests for DF ops unavailable in Spark 3.5 by @shehabgamin in https://github.com/lakehq/sail/pull/2064
• chore: clean up downcast_ref usage by @linhr in https://github.com/lakehq/sail/pull/2065
• fix: minor issues after DataFusion 54 upgrade by @linhr in https://github.com/lakehq/sail/pull/2066
• feat: support various sink modes for file writes by @linhr in https://github.com/lakehq/sail/pull/2053
• fix: align file/catalog write existence checks with Spark by @shehabgamin in https://github.com/lakehq/sail/pull/2068
• feat: implement to_xml Spark function by @anarefolio in https://github.com/lakehq/sail/pull/2032
• feat: add window_time function by @radmirnovii in https://github.com/lakehq/sail/pull/2063
• refactor: logical planning for DELETE by @linhr in https://github.com/lakehq/sail/pull/2069
• fix: align bin with Spark ANSI cast semantics by @davidlghellin in https://github.com/lakehq/sail/pull/2019
• refactor: logical planning for MERGE by @linhr in https://github.com/lakehq/sail/pull/2070
• deps(cargo): bump the all group with 12 updates by @dependabot[bot] in https://github.com/lakehq/sail/pull/2072
• deps(npm): bump the all group with 9 updates by @dependabot[bot] in https://github.com/lakehq/sail/pull/2071
• chore: move HMS codegen to a separate crate by @linhr in https://github.com/lakehq/sail/pull/2076
• fix: data source temporary views by @shehabgamin in https://github.com/lakehq/sail/pull/2059
• refactor: physical planner for row-level operations by @linhr in https://github.com/lakehq/sail/pull/2077
• fix: docker moby error by @twsl in https://github.com/lakehq/sail/pull/2080
• fix: align Spark parity for to_number format parsing by @davidlghellin in https://github.com/lakehq/sail/pull/2074
• perf: single-buffer concat_ws kernel by @davidlghellin in https://github.com/lakehq/sail/pull/2040
• fix: support array indexing and bracket paths in get_json_object by @davidlghellin in https://github.com/lakehq/sail/pull/2028
• feat: enable ANSI mode for negative by @davidlghellin in https://github.com/lakehq/sail/pull/2031
• perf: implement simplify hook on concat by @davidlghellin in https://github.com/lakehq/sail/pull/1891
• fix: don't fail a pending task while workers are still launching by @benzsevern in https://github.com/lakehq/sail/pull/2081
• fix: handle regexp_extract optional capture groups by @pomykalakyle in https://github.com/lakehq/sail/pull/2082
• fix: column default metadata handling for Hive Metastore catalog writes by @shehabgamin in https://github.com/lakehq/sail/pull/2058
• feat: introduce catalog execution context by @lonless9 in https://github.com/lakehq/sail/pull/2078
• chore: vendor arrow-pyarrow for PyO3 security fix by @linhr in https://github.com/lakehq/sail/pull/2088
• deps(npm): bump the all group with 3 updates by @dependabot[bot] in https://github.com/lakehq/sail/pull/2091
• fix: type casting issue for map_entries by @linhr in https://github.com/lakehq/sail/pull/2095
• feat: support filter array higher-order function with lambda by @davidlghellin in https://github.com/lakehq/sail/pull/2057
• chore: improve test selection logic by @linhr in https://github.com/lakehq/sail/pull/2102
• chore: move integration tests by @linhr in https://github.com/lakehq/sail/pull/2105
• fix: array_position by @shehabgamin in https://github.com/lakehq/sail/pull/2104
• chore: reorganize Python unit tests by @linhr in https://github.com/lakehq/sail/pull/2108
• fix: format sketch function default arguments by @lonless9 in https://github.com/lakehq/sail/pull/2030
• test: consolidate session management logic by @linhr in https://github.com/lakehq/sail/pull/2111
• test: clean up BDD step modules by @linhr in https://github.com/lakehq/sail/pull/2112
• test: rewrite test classes as module-level tests by @linhr in https://github.com/lakehq/sail/pull/2113
• refactor: simplify query planning and codec for lambda functions by @shehabgamin in https://github.com/lakehq/sail/pull/2106
• refactor: minor codec cleanup by @linhr in https://github.com/lakehq/sail/pull/2115
• fix: align lakehouse CREATE and REPLACE TABLE semantics by @lonless9 in https://github.com/lakehq/sail/pull/2097
• chore: use hashmap instead of array traversal for merge plan by @ctydw123 in https://github.com/lakehq/sail/pull/2110
• fix: align EXPLAIN EXTENDED, and COST output with Spark by @nis12ram in https://github.com/lakehq/sail/pull/2114
• feat: support transform array higher-order function with lambda by @davidlghellin in https://github.com/lakehq/sail/pull/2103
• feat: support exists and forall array higher-order function with lambda by @davidlghellin in https://github.com/lakehq/sail/pull/2109
• chore: update docs and tests for parity by @linhr in https://github.com/lakehq/sail/pull/2124
• fix: support Windows local paths for Iceberg by @lonless9 in https://github.com/lakehq/sail/pull/2119
• feat: support Parquet content-defined chunking options by @linhr in https://github.com/lakehq/sail/pull/2125
• fix: support timestampadd by @pomykalakyle in https://github.com/lakehq/sail/pull/2086
• feat: lambda aggregate by @shehabgamin in https://github.com/lakehq/sail/pull/2123
• chore: upgrade action versions by @linhr in https://github.com/lakehq/sail/pull/2126
• feat: support array_sort array higher-order function with comparator lambda by @davidlghellin in https://github.com/lakehq/sail/pull/2121
• fix: validate aggregate/reduce lambda arity by @davidlghellin in https://github.com/lakehq/sail/pull/2127
• fix: case insensitive function match by @shehabgamin in https://github.com/lakehq/sail/pull/2131
• refactor: use datafusion-spark UDF for xxhash64 by @davidlghellin in https://github.com/lakehq/sail/pull/2130
• test: skip Spark 3 DataFrame doctests unsupported by Spark Connect by @shehabgamin in https://github.com/lakehq/sail/pull/2133
• deps(cargo): bump the all group across 1 directory with 6 updates by @dependabot[bot] in https://github.com/lakehq/sail/pull/2132
• refactor: reorganize Delta Lake module layout by @lonless9 in https://github.com/lakehq/sail/pull/2122
• feat: add pivot (rewrite to aggregate with per-value FILTER) by @davidlghellin in https://github.com/lakehq/sail/pull/2120
• fix: make TableUpdate/ViewUpdate discriminated enums in Iceberg REST catalog by @alexy in https://github.com/lakehq/sail/pull/2134
• refactor: pivot by @shehabgamin in https://github.com/lakehq/sail/pull/2138
• feat: improve catalog providers by @linhr in https://github.com/lakehq/sail/pull/2141
• feat: misc parity by @shehabgamin in https://github.com/lakehq/sail/pull/2142
• fix: delta lake legacy protocol check blocking writes to delta-spark-created column-mapping tables by @zemin-piao in https://github.com/lakehq/sail/pull/2148
• chore: update OpenTelemetry dependencies by @linhr in https://github.com/lakehq/sail/pull/2149
• feat: improve schema_of_json function by @davidlghellin in https://github.com/lakehq/sail/pull/1225
• feat: unify to_timestamp/try_to_timestamp, honor ANSI, drop legacy UDF by @davidlghellin in https://github.com/lakehq/sail/pull/2100
• fix: restore SPARK_REMOTE="local" JVM validation under PySpark 4.1 by @davidlghellin in https://github.com/lakehq/sail/pull/2153
• feat: add try_to_time, unify to_time under SparkTime by @davidlghellin in https://github.com/lakehq/sail/pull/2094
• feat: enable ANSI mode for percentile_disc by @davidlghellin in https://github.com/lakehq/sail/pull/1962
• chore: prepare v0.6.5 by @linhr in https://github.com/lakehq/sail/pull/2156

New Contributors

• @ctydw123 made their first contribution in https://github.com/lakehq/sail/pull/2039
• @radmirnovii made their first contribution in https://github.com/lakehq/sail/pull/2006
• @benzsevern made their first contribution in https://github.com/lakehq/sail/pull/2081

Full Changelog: https://github.com/lakehq/sail/compare/v0.6.4...v0.6.5