Bytewax Files

Overview

• Adds a dataflow structure visualizer. Run python -m bytewax.visualize.

• Breaking change The internal format of recovery databases has been changed from using JsonPickle to Python's built-in {py:obj}pickle. Recovery stores that used the old format will not be usable after upgrading.

• Breaking change The unary operator and UnaryLogic have been renamed to stateful and StatefulLogic respectively.

• Adds a stateful_batch operator to allow for lower-level batch control while managing state.

• StatefulLogic.on_notify, StatefulLogic.on_eof, and StatefulLogic.notify_at are now optional overrides. The defaults retain the state and emit nothing.

• Breaking change Windowing operators have been moved from bytewax.operators.window into bytewax.operators.windowing.

• Breaking change ClockConfigs have had Config dropped from their name and are just Clocks. E.g. If you previously from bytewax.operators.window import SystemClockConfig now from bytewax.operators.windowing import SystemClock.

• Breaking change WindowConfigs have been renamed to Windowers. E.g. If you previously from bytewax.operators.window import SessionWindow now from bytewax.operators.windowing import SessionWindower.

• Breaking change All windowing operators now return a set of streams {py:obj}~bytewax.operators.windowing.WindowOut. {py:obj}~bytewax.operators.windowing.WindowMetadata now is branched into its own stream and is no longer part of the single downstream. All window operator emitted items are labeled with the unique window ID they came from to facilitate joining the data later.

• Breaking change {py:obj}~bytewax.operators.windowing.fold_window now requires a merge argument. This handles whenever the session windower determines that two windows must be merged because a new item bridged a gap.

• Breaking change The join_named and join_window_named operators have been removed because they did not support returning proper type information. Use {py:obj}~bytewax.operators.join or {py:obj}~bytewax.operators.windowing.join_window instead, which have been enhanced to properly type their downstream values.

• Breaking change {py:obj}~bytewax.operators.join and {py:obj}~bytewax.operators.windowing.join_window have had their product argument replaced with mode. You now can specify more nuanced kinds of join modes.

• Python interfaces are now provided for custom clocks and windowers. Subclass {py:obj}~bytewax.operators.windowing.Clock (and a corresponding {py:obj}~bytewax.operators.windowing.ClockLogic) or {py:obj}~bytewax.operators.windowing.Windower (and a corresponding {py:obj}~bytewax.operators.windowing.WindowerLogic) to define your own senses of time and window definitions.

• Adds a {py:obj}~bytewax.operators.windowing.window operator to allow you to write more flexible custom windowing operators.

• Session windows now work correctly with out-of-order data and joins.

• All windowing operators now process items in timestamp order. The most visible change that this results in is that the {py:obj}~bytewax.operators.windowing.collect_window operator now emits collections with values in timestamp order.

• Adds a {py:obj}~bytewax.operators.filter_map_value operator.

• Adds a {py:obj}~bytewax.operators.enrich_cached operator for easier joining with an external data source.

• Adds a {py:obj}~bytewax.operators.key_rm convenience operator to remove keys from a {py:obj}~bytewax.operators.KeyedStream.

What's Changed

• Little docs cleanups by @davidselassie in https://github.com/bytewax/bytewax/pull/424
• Add builds for Python 3.12 by @whoahbot in https://github.com/bytewax/bytewax/pull/425
• Adds async connector guide by @davidselassie in https://github.com/bytewax/bytewax/pull/428
• refactor orderbook example to remove ordereddict by @lfunderburk in https://github.com/bytewax/bytewax/pull/429
• Fix an issue with session windows by @whoahbot in https://github.com/bytewax/bytewax/pull/430
• Prepare 0.19.1 by @whoahbot in https://github.com/bytewax/bytewax/pull/431
• Adds a simple Mermaid diagram encoder by @davidselassie in https://github.com/bytewax/bytewax/pull/432
• Forgot to change docs sidebar globs when moving dirs by @davidselassie in https://github.com/bytewax/bytewax/pull/435
• Update 301/404 links in README.md by @konradsienkowski in https://github.com/bytewax/bytewax/pull/434
• Move windowing operators into Python by @davidselassie in https://github.com/bytewax/bytewax/pull/433
• System clock must advance watermark on EOF by @davidselassie in https://github.com/bytewax/bytewax/pull/437
• Upgrade pyo3 to 0.21.1 by @whoahbot in https://github.com/bytewax/bytewax/pull/439
• Allow jsonpickle to serialize numpy/pandas by @whoahbot in https://github.com/bytewax/bytewax/pull/440
• Simplifies PyO3 extensions by @davidselassie in https://github.com/bytewax/bytewax/pull/441
• Add a performance guide. by @whoahbot in https://github.com/bytewax/bytewax/pull/442
• Slightly better typing on joins by @davidselassie in https://github.com/bytewax/bytewax/pull/445
• Fixes up mypy typing in Kafka connectors by @davidselassie in https://github.com/bytewax/bytewax/pull/447
• "Reproducible" dev and CI envs by @davidselassie in https://github.com/bytewax/bytewax/pull/438
• More Deployment in Documentation by @awmatheson in https://github.com/bytewax/bytewax/pull/443
• Fix prometheus scrape endpoint by @whoahbot in https://github.com/bytewax/bytewax/pull/451
• Fix Mac CI by @davidselassie in https://github.com/bytewax/bytewax/pull/450
• Re-enables doctests by @davidselassie in https://github.com/bytewax/bytewax/pull/448
• Adds enrich_cached operator by @davidselassie in https://github.com/bytewax/bytewax/pull/449
• Make public visualization script by @davidselassie in https://github.com/bytewax/bytewax/pull/455
• Run Vermin twice, once for lib code 3.8- and once for dev code 3.12- by @davidselassie in https://github.com/bytewax/bytewax/pull/457
• Substitute in the current Bytewax version into docs by @davidselassie in https://github.com/bytewax/bytewax/pull/456
• Renames operators.window to operators.windowing by @davidselassie in https://github.com/bytewax/bytewax/pull/453
• More type annotation fixups by @davidselassie in https://github.com/bytewax/bytewax/pull/458
• Serde changes by @whoahbot in https://github.com/bytewax/bytewax/pull/446
• Fix docs typos by @hcs42 in https://github.com/bytewax/bytewax/pull/459
• Windowing operators process items in timestamp order by @davidselassie in https://github.com/bytewax/bytewax/pull/461
• Tiny doc formatting fix by @davidselassie in https://github.com/bytewax/bytewax/pull/463
• join and join_window operators now have modes by @davidselassie in https://github.com/bytewax/bytewax/pull/462
• Adds a way to link to files in the public repo from docs retaining the current version by @davidselassie in https://github.com/bytewax/bytewax/pull/465
• Join mode default now typechecks by @davidselassie in https://github.com/bytewax/bytewax/pull/468
• Fixes for re-joining WindowMetadata by @davidselassie in https://github.com/bytewax/bytewax/pull/469
• Prepare 0.20.0 by @whoahbot in https://github.com/bytewax/bytewax/pull/466

New Contributors

• @lfunderburk made their first contribution in https://github.com/bytewax/bytewax/pull/429
• @hcs42 made their first contribution in https://github.com/bytewax/bytewax/pull/459

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.19.0...v0.20.0