Name | Modified | Size | Downloads / Week |
---|---|---|---|
Parent folder | |||
README.md | 2025-07-08 | 6.2 kB | |
v0.25.0_ Support Ibis table validation source code.tar.gz | 2025-07-08 | 539.5 kB | |
v0.25.0_ Support Ibis table validation source code.zip | 2025-07-08 | 725.3 kB | |
Totals: 3 Items | 1.3 MB | 1 |
⭐️ Highlight
Pandera now supports Ibis 🦩! You can now validate data on all available ibis backends using the pandera.ibis
module.
In-memory table example:
:::python
import ibis
import pandera.ibis as pa
class Schema(pa.DataFrameModel):
state: str
city: str
price: int = pa.Field(in_range={"min_value": 5, "max_value": 20})
t = ibis.memtable(
{
'state': ['FL','FL','FL','CA','CA','CA'],
'city': [
'Orlando',
'Miami',
'Tampa',
'San Francisco',
'Los Angeles',
'San Diego',
],
'price': [8, 12, 10, 16, 20, 18],
}
)
Schema.validate(t).execute()
Sqlite example:
:::python
con = ibis.sqlite.connect()
t = con.create_table(
"table",
schema=ibis.schema(dict(state="string", city="string", price="int64"))
)
con.insert(
"table",
obj=[
("FL", "Orlando", 8),
("FL", "Miami", 12),
("FL", "Tampa", 10),
("CA", "San Francisco", 16),
("CA", "Los Angeles", 20),
("CA", "San Diego", 18),
]
)
Schema.validate(t).execute()
What does this mean?
This release unlocks in database validation in some of the most widely used data platforms, including PostGres, Snowflake, BigQuery, MySQL, and more ✨. It means that you can validate data at scale, on your database/data framework of your choice, before fetching it for downstream analysis/modeling work.
Naturally, this also means that you can develop your schemas locally on a duckdb
or sqlite
backend and then use the same schemas in production on a remote database like postgres
.
Learn more about the integration here.
What's Changed
- Add Polars pydantic integration with format support and native JSON schema generation by @halicki in https://github.com/unionai-oss/pandera/pull/1979
- exclude python 3.12 and pyspark combo in ci by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2005
- Delete previously-added foo.txt and new_example.py by @deepyaman in https://github.com/unionai-oss/pandera/pull/2013
- Pin PySpark due to test failures/incompatibilities by @deepyaman in https://github.com/unionai-oss/pandera/pull/2010
- Temporarily pin
polars
due to test failure in CI by @deepyaman in https://github.com/unionai-oss/pandera/pull/2011 - Replace
event_loop
removed in pytest-asyncio 1.0 by @deepyaman in https://github.com/unionai-oss/pandera/pull/2014 - Fix typehint in unique_values_eq (issue [#1492]) by @AhmetZamanis in https://github.com/unionai-oss/pandera/pull/2015
- fix pyarrow string issue, fix docs failing issues by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2026
- bugfix: PANDERA_VALIDATION_ENABLED=False should disable validation by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2028
- Expect Python slice index errors after Python 3.10 by @deepyaman in https://github.com/unionai-oss/pandera/pull/2033
- Ibis dev by @deepyaman in https://github.com/unionai-oss/pandera/pull/2040
- handle dataframe-level failure cases: convert row to dict by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2050
- bugfix/1927 by @Jarek-Rolski in https://github.com/unionai-oss/pandera/pull/2019
- [🐻❄️ polars] Limit reported failure cases if Check.n_failure_cases is defined. by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2055
- [🦩 ibis] Limit reported failure cases if Check.n_failure_cases is defined. by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2056
- Add link to the documentation about Ibis datatypes by @deepyaman in https://github.com/unionai-oss/pandera/pull/2057
- Test column presence, mark other features not impl by @deepyaman in https://github.com/unionai-oss/pandera/pull/2060
- Run
pre-commit
on all files to fix linter issues by @deepyaman in https://github.com/unionai-oss/pandera/pull/2063 - Implement
regex
option and add additional checks by @deepyaman in https://github.com/unionai-oss/pandera/pull/2061 - Implement binary and boolean types (and test them) by @deepyaman in https://github.com/unionai-oss/pandera/pull/2064
- Add unit test suite for Ibis components, fix a bug by @deepyaman in https://github.com/unionai-oss/pandera/pull/2065
- bugfix: fix
format_vectorized_error_message
to properly format nested pyarrow failed cases by @AndrejIring in https://github.com/unionai-oss/pandera/pull/2036 - handle empty dataframes with PydanticModel: show warning by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2066
- bugfix/2031: Allow strict='filter' and coerce='True' at the same time for PySpark schemas by @gfilaci in https://github.com/unionai-oss/pandera/pull/2032
- Set validation scope for pandas run_checks methods by @amerberg in https://github.com/unionai-oss/pandera/pull/2003
- DataFrameSchema.update_index correctly sets title, description, and metadata by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2067
- [ibis 🦩] remove inplace=True in column validate call by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2068
- [ibis 🦩] check backend: use positional join for duckdb and polars, fix ibis DataFrameModel.validate types by @cosmicBboy in https://github.com/unionai-oss/pandera/pull/2071
New Contributors
- @halicki made their first contribution in https://github.com/unionai-oss/pandera/pull/1979
- @AhmetZamanis made their first contribution in https://github.com/unionai-oss/pandera/pull/2015
- @AndrejIring made their first contribution in https://github.com/unionai-oss/pandera/pull/2036
- @gfilaci made their first contribution in https://github.com/unionai-oss/pandera/pull/2032
- @amerberg made their first contribution in https://github.com/unionai-oss/pandera/pull/2003
Full Changelog: https://github.com/unionai-oss/pandera/compare/v0.24.0...v0.25.0