Ibis Files

9.4.0 (2024-09-03)

Features

• api: add approx_quantiles for computing approximate quantiles (dcdb7a7)
• api: add DateValue.epoch api for computing days since epoch (#9856) (8b0fb66)
• api: make the null function deferrable (0613ef1)
• api: support SchemaLike in Backend.create_table() (#9885) (949fbea)
• api: support deferred objects in literal (#9904) (0a07906)
• clickhouse: partition kwargs for compile and execution in to_pyarrow and to_pandas (2dd2c3f)
• clickhouse: support ms/us/ns truncate units (9881edb)
• decompile: make the decompiler run on TPCH query 1 (#9779) (0268044)
• exasol: implement approx_nunique, std, var (d9c3daa)
• exasol: implement approx_nunique, std, var (63c20c0)
• exasol: implement cov/corr (24f41b2)
• exasol: implement median and approx_median (3cfc344)
• exasol: implement quantile (ecbef94)
• exasol: implement Table.nunique (a24200c)
• exasol: implement Table.nunique (7ead7c7)
• flink: array sort (ca85ae2)
• flink: support ArrayValue.collect (eb857e6)
• impala: add tbl_properties to create_table (#9839) (e3d02bd)
• mssql: support connecting with a url (#9894) (8bb12e1), closes #9856
• oracle: implement mode aggregation (#9914) (9ee910d)
• output-formats: add support for to_parquet_dir (#9781) (80dfbe2)
• polars: array sort (9a2563b)
• polars: implement approx_nunique (3f3738d)
• pyspark: support quantile (26d8516)
• selectors: support naming deferreds in across (de1595c)
• snowflake: implement interval arithmetic (#9794) (41e10ca), closes #9783
• sql: enable cross-database joins (#9849) (c3ff6ae)
• sql: fuse distinct with other select nodes when possible (c31412b)
• sqlite: support most date/timestamp interval arithmetic (75f594d)
• sql: load parsed but unsupported types as unknown (#9868) (a76acfc)
• sql: support inserts with default constraints (#9844) (86a3c06)
• timestamps: add support for timestamp/date +/- intervals for additional backends (#9799) (79cef68)
• trino: support years and months in datetime arithmetic (1133973)
• trino: wrap auth strings with BasicAuthentication (#9960) (e0f54c9)

Bug Fixes

• bigquery: disallow column names longer than 300 characters (#9916) (ea97794), closes #8931
• clickhouse: workaround EXCEPT and INTERSECT generation in sqlglot; add tpcds query 87 (#9959) (910b8f5)
• datafusion: fix creation of SessionContext in datafusion 40.1.0 (eec5328)
• datafusion: handle NULLs in array flatten (ecc199f)
• deps: update dependency datafusion to v40 (4aa402a)
• deps: update dependency sqlglot to >=23.4,<25.11 (#9805) (84bfeb5)
• deps: update dependency sqlglot to >=23.4,<25.12 (#9834) (69a10d9)
• deps: update dependency sqlglot to >=23.4,<25.13 (#9851) (6780a6b)
• deps: update dependency sqlglot to >=23.4,<25.15 (#9864) (d182e9e)
• deps: update dependency sqlglot to >=23.4,<25.16 (#9875) (0a6765b)
• deps: update dependency sqlglot to >=23.4,<25.17 (#9907) (9e52edb)
• deps: update dependency sqlglot to >=23.4,<25.18 (#9935) (ee5116d)
• deps: update dependency sqlglot to >=23.4,<25.19 (#9962) (4c136d8)
• dot-sql: ensure that CTEs can be used in .sql (b63e0fd)
• duckdb: fix create_table() in databases with spaces in the name (#9817) (9da3c9f)
• exasol: properly handle returning BIGINT values (e20bdad)
• ir: convert analytic functions to window functions in filters (31295dd)
• mssql: remove sort key to keep order (#9848) (3780a13)
• mssql: support .cache() for caching tables (1de2f45)
• oracle: avoid double cursor closing by removing unnecessary close in _fetch_from_cursor (#9913) (a402095)
• oracle: implement current_catalog and current_database correctly (#9918) (4fdb707)
• pickle: make Parameter instances pickleable (#9798) (d772c80), closes #9793
• pivot-wider: handle the case of empty id_cols (#9912) (4a4bc64)
• polars: use drop_table when cleaning the cache and remove duplicated _remove_table method (#9922) (ce51941)
• polars: use flatten API for ArrayFlatten implementation to avoid large string upcast (#9997) (7a6af8d), closes #9995
• pyspark: suppress errors for unsupported Databricks serverless compute properties (#9830) (57f5ff6)
• repr: format scalar values with same logic as columnar values (5cd58fa)
• rewrite: avoid accumulating context state during rewriting (#9814) (9165255)
• strings: correct a typo in startswith docstring (#9994) (f98aec5), closes #1000 #2000 #3000
• trino: remove hack that silently breaks join transpilation (#9941) (c842453)
• ux: get rid of duplicated tracebacks (#10002) (7df4bdd)

Documentation

• add aggregate-udfs api page to index (#9789) (01dc81e)
• blog: ibis + clickhouse + shiny for better pypi stats (#9880) (4d8d352)
• blog: kaggle competition using IbisML (#9505) (7320c18)
• blog: mention dask in the pandeprecation post (#9899) (2f2c3ed)
• blog: minor edits to pandas blog (#9920) (69a44c4)
• blog: post on why we are dropping the pandas backend (#9896) (95104e5)
• datatypes: install rich traceback handler for prettier exception tracebacks (#10004) (4b77e4f)
• fix doctests for postgres backend (#9964) (497df15)
• presentations: positconf 2024 talk (#9822) (e8b89e7)
• snowflake: document using private key to connect to snowflake (c70f55a)
• snowflake: fix the syntax for passing schema (#9831) (c99cb4b)
• snowflake: remove duplicate schema docstring (#9829) (3183bef)

Refactors

• aliasing: remove the need for renaming after execution (#9996) (a0d7237)
• api: make unit required in IntegerValue.as_timestamp (7fa7395)
• internals: don't cache table accessor to avoid a circular reference (fb604a5)
• intervals: conslidate interval conversion under _make_interval base compiler implementation (fe29210)
• make approximate ops subclasses of their non-approximate variants (9d218d1)
• remove base implementation for quantile (3c49c6a)
• selectors: remove janky Predicate class and unify Selectors under a single interface (#9917) (c15a229)
• simplify caching implementation (afba988)

Deprecations

• api: deprecate StructValue.destructure API (#9824) (98c5c76)
• api: deprecate type coercion to_* methods in favor of as_* methods (fb22e20), closes #9788 /github.com/ibis-project/ibis/pull/9843#discussion_r1717545190
• dask,pandas: deprecate the dask and pandas backends (910fa5c)
• expr-api: deprecate useless has_name method (#9901) (e0436aa)