Kotlin Dataframe Files

We're one step closer to DataFrame 1.0! We've been working hard to iron out bugs and improve documentation and examples throughout.

Try out this release by updating to 1.0.0-Beta3 in your project or use %use dataframe in your notebook!

📚 Learn more: https://kotlin.github.io/dataframe/

🌟 Examples: https://github.com/Kotlin/dataframe/tree/master/examples

See below for a complete list of changes in this release grouped by category.

1.0.0-Beta3 Highlights

• NEW: Reading data from Parquet files
• NEW: Reading data from DuckDB databases
• Our KDocs and website have received many, many additions, as you can see below, but some that stand out are:
• Docs about the compiler plugin: https://kotlin.github.io/dataframe/compiler-plugin.html
• A setup guide with instructions for each supported platform: https://kotlin.github.io/dataframe/setup.html
• (Un)supported Data sources overview: https://kotlin.github.io/dataframe/data-sources.html
• DataFrame Concepts and Principles: https://kotlin.github.io/dataframe/concepts.html
• Frequently Asked Questions: https://kotlin.github.io/dataframe/faq.html
• Migration away from Gradle/KSP Plugin: https://kotlin.github.io/dataframe/migration-from-plugins.html
• New examples:
• Using the compiler plugin in a project: https://github.com/Kotlin/dataframe/tree/master/examples/kotlin-dataframe-plugin-example
• DataFrame on Android: https://github.com/Kotlin/dataframe/tree/master/examples/android-example
• "Unsupported data sources" examples, including Exposed, Spark, Hibernate, and Multik: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/examples
• DataFrame with Spark and Parquet files: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/spark-parquet-dataframe
• format to HTML has had several improvements and bug fixes; though, the API might have breaking changes.
• We continue to improve the compiler plugin to better reflect the schemas of dataframes in compile time. You can see the compile time schema by hovering over a function call in the IDE — the schema will be displayed in the popup. If there are any unexpected results, feel free to open an issue or contact us!

Known Issues

• IN NOTEBOOKS use version 1.0.0-Beta3n instead of 1.0.0-Beta3. This uses the patch of [#1435] for issue [#1116] avoiding DefinitelyNotNullable errors. This is the version that will be provided when writing %use dataframe. Check dataFrameConfig.version in your notebook to check which version you are using.
• median and percentile require explicit type arguments for non-numeric columns
• https://github.com/Kotlin/dataframe/issues/1189
• FormattedFrame does not render formatted in notebooks
• This has since been fixed but might require a yet to be released version of IntelliJ.
• See https://github.com/Kotlin/dataframe/pull/1405

Features

• DataFrameSchema.generateCode() by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1230
• add nulls parse option in readExcel by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1247
• Kotlin 2.2 by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1286
• Parse String to UUID [#1006] by @EmmanuelBerkowicz in https://github.com/Kotlin/dataframe/pull/1287
• Makes prev()?.newValue() work in Add DSL by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1303
• Added readOnly mode by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1325
• Add support for validating SQL queries with the WITH clause by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1327
• Add deprecated filterNotNull for discoverability of dropNulls by @koperagen in https://github.com/Kotlin/dataframe/pull/1334
• parseExperimentalUuid in ParserOptions by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1306
• Add support for reading parquet file thanks to arrow-dataset [#576] by @fb64 in https://github.com/Kotlin/dataframe/pull/577
• duckdb support by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1366
• Format colgroups by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1374
• Add support for read-only mode in DuckDB and SQLite by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1383
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1368
• Make AnyFrame.renderToString public by @bwjohnson92 in https://github.com/Kotlin/dataframe/pull/1395

Docs, Examples, and KDocs

• Website docs landing by @AndrewKis in https://github.com/Kotlin/dataframe/pull/1170
• Website docs examples dev by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1179
• fix snippets iframes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1190
• Website docs kotlin notebook by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1180
• Generate docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1194
• Rename WS docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1191
• Website docs dev by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1196
• fix kodex iframes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1201
• Number unification docs by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1200
• Add initial compiler plugin documentation by @koperagen in https://github.com/Kotlin/dataframe/pull/1202
• Add few more pages about compiler plugin by @koperagen in https://github.com/Kotlin/dataframe/pull/1204
• Remove some of the outdated info from README.md by @koperagen in https://github.com/Kotlin/dataframe/pull/1205
• Minor fixes for the docs by @koperagen in https://github.com/Kotlin/dataframe/pull/1208
• Update README.md by @koperagen in https://github.com/Kotlin/dataframe/pull/1209
• Update Gradle plugin docs and fix hardcoded versions by @koperagen in https://github.com/Kotlin/dataframe/pull/1211
• Update documentation for "create dataframe" by @koperagen in https://github.com/Kotlin/dataframe/pull/1212
• Remove examples for deprecated overloads from tests by @koperagen in https://github.com/Kotlin/dataframe/pull/1213
• Notebook updates by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1224
• Adding links to column selectors in docs by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1226
• fix: fixed German companies notebook sample by @kikoso in https://github.com/Kotlin/dataframe/pull/1234
• fix kodex docs iframes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1233
• fix escaping in iframes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1237
• add pictures for guides by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1239
• various docs fixes to lower the number of errors by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1238
• Plugin example by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1241
• KTNB-1041: Fix resource loading by @ileasile in https://github.com/Kotlin/dataframe/pull/1250
• Extension properties docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1246
• Readme update by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1251
• IDE sample of "unsupported sources"->DataFrame by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1231
• add kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1261
• Gather kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1272
• Corr kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1275
• filter kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1288
• columns selector type by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1274
• explode docs and tests by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1291
• improve concept docs structure by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1305
• Simplify compiler plugin setup in documentation by @koperagen in https://github.com/Kotlin/dataframe/pull/1317
• modules topic by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1314
• Flatten navigation of Modify operations by @koperagen in https://github.com/Kotlin/dataframe/pull/1312
• faq topic by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1316
• Docs small fixes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1323
• generateCode.kt refactor, KDocs, documentation fixes, and tests by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1311
• Data schemas docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1339
• Data sources docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1352
• Setup docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1357
• format kdocs++ by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1346
• count and countDistinct kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1301
• clarify how to create BaseColumn by @koperagen in https://github.com/Kotlin/dataframe/pull/1371
• add spelling conventions and name fixes in a whole project by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1364
• android example by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1349
• Android example: removing arrow from the examples by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1426
• Add Hibernate example with H2 in-memory database to idea-examples by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1367
• Extension property name generation fix & docs & tests by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1380
• Add SQL-to-Kotlin DataFrame transition guide for backend developers by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1377
• [Junie]: docs: update ColumnGroup description with relevant links by @jetbrains-junie[bot] in https://github.com/Kotlin/dataframe/pull/1404
• Added Spark-Parquet-KDF example by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1391
• Rename kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1402
• Insert kdocs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1406
• Util functions docs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1414

Fixes

• Bump apache poi to 5.4.1 by @Badya in https://github.com/Kotlin/dataframe/pull/1198
• fixed compilation on android by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1203
• Android compilation fix by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1218
• IDEA goodness by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1235
• Use DayOfWeek.isoDayNumber instead of DayOfWeek.value in DateTests.kt. by @chaoren in https://github.com/Kotlin/dataframe/pull/1232
• Run configs by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1236
• Reusage schemas fix by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1252
• Remove mandatory mariadb dependency from jdbc by @koperagen in https://github.com/Kotlin/dataframe/pull/1267
• GroupBy Jupyter codegen by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1263
• Add more specific explodeLists overload that helps to avoid unnecessary cast by @koperagen in https://github.com/Kotlin/dataframe/pull/1269
• Bump gradle to 8.14.2 by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1285
• Removing recursively() remnants from public api by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1302
• Replace static initialization of defaultTimeZone with getter by @koperagen in https://github.com/Kotlin/dataframe/pull/1315
• Improve SQL validation logic and add related test by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1324
• Migrate publishing by @koperagen in https://github.com/Kotlin/dataframe/pull/1331
• Hide mapToColumn in CS DSL in favor of expr by @koperagen in https://github.com/Kotlin/dataframe/pull/1360
• Adds tableTypes to DbType by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1283
• format kdocs++ by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1346
• skipKodex gradle parameter in dataframe-csv by @koperagen in https://github.com/Kotlin/dataframe/pull/1370
• Fix jupyter logger by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1361
• Initialize JetBrains Junie 🚀 by @jetbrains-junie[bot] in https://github.com/Kotlin/dataframe/pull/1379
• Fixed reading Parquet file on Windows by @zaleslaw in https://github.com/Kotlin/dataframe/pull/1381
• Extension property name generation fix & docs & tests by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1380
• Migrate usages of deprecated API in implementation & warning fixes by @koperagen in https://github.com/Kotlin/dataframe/pull/1382
• [Junie]: fix: update deprecation message for DataRow.get method by @jetbrains-junie[bot] in https://github.com/Kotlin/dataframe/pull/1399
• Removed deprecatedSchemaGeneratorPlugin Gradle plugin publication by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1400
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1368
• Columngroup map fix by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1396
• Adds is_formatted to json encoding for DF by @Jolanrensen in https://github.com/Kotlin/dataframe/pull/1405
• Util functions fixes by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1412
• Add API that debugger renderer can rely on to avoid breaking changes by @koperagen in https://github.com/Kotlin/dataframe/pull/1421
• Update Gradle to 9.0 by @koperagen in https://github.com/Kotlin/dataframe/pull/1428
• What's the version 1727 and 1548? Fixed by @Jolanrensen in https://github.com/Kotlin/dataframe/issues/436

Compiler Plugin

Note that many changes to the compiler plugin are made in Kotlin, as this is where the compiler plugin now lives. * Prepare Split for compiler plugin support by @koperagen in https://github.com/Kotlin/dataframe/pull/1197 * Prepare update and implode operations for compiler plugin support by @koperagen in https://github.com/Kotlin/dataframe/pull/1207 * Deprecate predicate argument in single, colsInGroups, colsAtAnyDepth by @koperagen in https://github.com/Kotlin/dataframe/pull/1176 * Add missed compiler plugin annotations for Split by @koperagen in https://github.com/Kotlin/dataframe/pull/1262 * add compiler annotations for dropNA by @koperagen in https://github.com/Kotlin/dataframe/pull/1330 * add missed compiler annotations for update by @koperagen in https://github.com/Kotlin/dataframe/pull/1333 * Add plugin annotations for a new batch of functions by @koperagen in https://github.com/Kotlin/dataframe/pull/1362 * Rollback parameter name to keep compatibility with the compiler plugin by @koperagen in https://github.com/Kotlin/dataframe/pull/1358 * cols with default argument deprecation by @AndreiKingsley in https://github.com/Kotlin/dataframe/pull/1369 * Add basic tests that API exposed for compiler plugin works without excluded dependencies by @koperagen in https://github.com/Kotlin/dataframe/pull/1397 * Remove annotations dependency from compiler-plugin-core by @koperagen in https://github.com/Kotlin/dataframe/pull/1409 * Add annotations to support types extending DataRow in the compiler plugin by @koperagen in https://github.com/Kotlin/dataframe/pull/1410 * [Compiler plugin] Support more functions in toDataFrame DSL and AddDsl by @koperagen in https://github.com/Kotlin/dataframe/pull/1425 * Prepare utility to assert compile time column order by @koperagen in https://github.com/Kotlin/dataframe/pull/1427

New Contributors

• @Badya made their first contribution in https://github.com/Kotlin/dataframe/pull/1198
• @kikoso made their first contribution in https://github.com/Kotlin/dataframe/pull/1234
• @chaoren made their first contribution in https://github.com/Kotlin/dataframe/pull/1232
• @EmmanuelBerkowicz made their first contribution in https://github.com/Kotlin/dataframe/pull/1287
• @jetbrains-junie[bot] made their first contribution in https://github.com/Kotlin/dataframe/pull/1379
• @bwjohnson92 made their first contribution in https://github.com/Kotlin/dataframe/pull/1395

Full Changelog: https://github.com/Kotlin/dataframe/compare/v1.0.0-Beta2...v1.0.0-Beta3