Bytewax Files

Overview

• Multiple operators have been reworked to avoid taking and releasing Python's global interpreter lock while iterating over multiple items. Windowing operators, stateful operators and operators like branch will see significant performance improvements.

Thanks to @damiondoesthings for helping us track this down!

• Breaking change FixedPartitionedSource.build_part, DynamicSource.build, FixedPartitionedSink.build_part and DynamicSink.build now take an additional step_id argument. This argument can be used when labeling custom Python metrics.

• Custom Python metrics can now be collected using the prometheus-client library.

• Breaking change The schema registry interface has been removed. You can still use schema registries, but you need to instantiate the (de)serializers on your own. This allows for more flexibility. See the confluent_serde and redpanda_serde examples for how to use the new interface.

• Fixes bug where items would be incorrectly marked as late in sliding and tumbling windows in cases where the timestamps are very far from the align_to parameter of the windower.

• Adds stateful_flat_map operator.

• Breaking change Removes builder argument from stateful_map. Instead, the initial state value is always None and you can call your previous builder by hand in the mapper.

• Breaking change Improves performance by removing the now: datetime argument from FixedPartitionedSource.build_part, DynamicSource.build, and UnaryLogic.on_item. If you need the current time, use:

  :::python from datetime import datetime, timezone

  now = datetime.now(timezone.utc)

• Breaking change Improves performance by removing the sched: datetime argument from StatefulSourcePartition.next_batch, StatelessSourcePartition.next_batch, UnaryLogic.on_notify. You should already have the scheduled next awake time in whatever instance variable you returned in {Stateful,Stateless}SourcePartition.next_awake or UnaryLogic.notify_at.

What's Changed

• Add Kafka concept section to metadata.json by @whoahbot in https://github.com/bytewax/bytewax/pull/373
• Fixed split_demo example by @Psykopear in https://github.com/bytewax/bytewax/pull/371
• Prevent dataflow hang if next_awake is far in future by @davidselassie in https://github.com/bytewax/bytewax/pull/374
• Adds a basic stub file generator by @davidselassie in https://github.com/bytewax/bytewax/pull/369
• Update metrics and observability guides by @Psykopear in https://github.com/bytewax/bytewax/pull/372
• Fixes pyright errors by @davidselassie in https://github.com/bytewax/bytewax/pull/378
• Shuffles around Kafka objects and updates docstrings by @davidselassie in https://github.com/bytewax/bytewax/pull/382
• Fixes collect and branch operator test file names by @davidselassie in https://github.com/bytewax/bytewax/pull/385
• Using MyST + Sphinx for API docs by @davidselassie in https://github.com/bytewax/bytewax/pull/383
• Update README.md by @jonasbest in https://github.com/bytewax/bytewax/pull/388
• All docs to Sphinx and RTD by @davidselassie in https://github.com/bytewax/bytewax/pull/386
• Update logo in README.md by @konradsienkowski in https://github.com/bytewax/bytewax/pull/389
• Removes stateful_map builder function and adds stateful_flat_map by @davidselassie in https://github.com/bytewax/bytewax/pull/387
• Re-enables doctests via Sybil by @davidselassie in https://github.com/bytewax/bytewax/pull/390
• Removes now and sched arguments in input partitions and unary logic by @davidselassie in https://github.com/bytewax/bytewax/pull/391
• fix backup interval of zero raising an exception by @damiondoesthings in https://github.com/bytewax/bytewax/pull/393
• Remove GLIBC 2.27 builder by @whoahbot in https://github.com/bytewax/bytewax/pull/395
• Fix recovery store garbage collection by @davidselassie in https://github.com/bytewax/bytewax/pull/394
• Customize Sphinx docs theme by @konradsienkowski in https://github.com/bytewax/bytewax/pull/400
• Cleanup docs by @whoahbot in https://github.com/bytewax/bytewax/pull/401
• [Docs]: Fix path to Slack icon by @konradsienkowski in https://github.com/bytewax/bytewax/pull/402
• Updates release instructions with Read the Docs stuff by @davidselassie in https://github.com/bytewax/bytewax/pull/403
• Additional recovery tests by @davidselassie in https://github.com/bytewax/bytewax/pull/407
• Adds warnings about using session windows by @davidselassie in https://github.com/bytewax/bytewax/pull/408
• Update max window documentation by @awmatheson in https://github.com/bytewax/bytewax/pull/410
• Windowing concept doc by @davidselassie in https://github.com/bytewax/bytewax/pull/412
• Don't call time_for twice by @whoahbot in https://github.com/bytewax/bytewax/pull/414
• Refactors window boundary calculation to avoid overflow by @davidselassie in https://github.com/bytewax/bytewax/pull/415
• Unified Redpanda and Confluent schema registries by @Psykopear in https://github.com/bytewax/bytewax/pull/399
• Fix inconsistent window boundaries panic due to microseconds in timestamps by @davidselassie in https://github.com/bytewax/bytewax/pull/416
• Makes stubgen deterministic by @davidselassie in https://github.com/bytewax/bytewax/pull/417
• Don't take the GIL when iterating over items by @whoahbot in https://github.com/bytewax/bytewax/pull/418
• Start working on custom metrics for Kafka by @whoahbot in https://github.com/bytewax/bytewax/pull/404
• Take 2 on inconsistent boundaries by @davidselassie in https://github.com/bytewax/bytewax/pull/419
• Add benchmarks for core operators by @whoahbot in https://github.com/bytewax/bytewax/pull/420
• Refactor operators by @whoahbot in https://github.com/bytewax/bytewax/pull/422
• Prepare for v0.19.0 release by @whoahbot in https://github.com/bytewax/bytewax/pull/423

New Contributors

• @jonasbest made their first contribution in https://github.com/bytewax/bytewax/pull/388
• @damiondoesthings made their first contribution in https://github.com/bytewax/bytewax/pull/393

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.18.2...v0.19.0