Ploomber
The fastest way to build data pipelines
...It allows developers to transform exploratory data analysis workflows into production-ready pipelines without rewriting large portions of code. The system integrates with common development environments such as Jupyter Notebook, VS Code, and PyCharm, enabling data scientists to continue working with familiar tools while building scalable workflows. Ploomber automatically manages task dependencies and execution order, allowing complex pipelines with multiple stages to run reliably. The framework can deploy pipelines across different computing environments including Kubernetes, Airflow, AWS Batch, and high-performance computing clusters. ...