Option to export compiled code including procedural steps
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer
Brought to you by:
slothflowlabs
Menu
▾
▴
#7 Option to export compiled code including procedural steps
Originally created by: add944
Originally owned by: SouravRoy-ETL
Hello,
SQL export feature is great and the workflow control components further extend the functionality!
Is there a way to implement the full pipeline code export ?
Originally posted by: SouravRoy-ETL
Thanks so much, glad the SQL export and the workflow-control components are landing well for you! The export now includes the procedural steps too: driver sources and sinks (Oracle, REST, Kafka, and so on) and the ctl.* control steps show up inline as descriptive comments, so the exported script reflects the whole pipeline order instead of only the parts that lower to DuckDB SQL.
Please try it on the new build and tell me if it captures what you were after: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.1.0-hotfix2
Originally posted by: add944
Apologies I might've not been super clear with this one! I was referring to a functionality whereby you can export the full pipeline code that can run anywhere. E.g. Amphi-etl has such functionality where everything is python under the hood. I understand in your project these procedural steps (e.g. wait, iteratate, if swtiches) are rust based? https://github.com/amphi-ai/amphi-etl
Originally posted by: SouravRoy-ETL
Fixed in v0.5.0. The SQL/code export now documents the control-flow steps (ctl.die, ctl.warn, run-job etc.) instead of emitting empty stages, so the exported script reflects the procedural parts of the pipeline too.
Please download - https://github.com/slothflowlabs/duckle/releases/tag/v0.5.0 and let me know!