Ploomber
The fastest way to build data pipelines
...The system integrates with common development environments such as Jupyter Notebook, VS Code, and PyCharm, enabling data scientists to continue working with familiar tools while building scalable workflows. Ploomber automatically manages task dependencies and execution order, allowing complex pipelines with multiple stages to run reliably. The framework can deploy pipelines across different computing environments including Kubernetes, Airflow, AWS Batch, and high-performance computing clusters. It also helps teams maintain reproducibility by tracking changes in code and rerunning only outdated pipeline tasks.