Datapipe

I'm not sure, but Datapipe may actually be unique. It's flexibility in ETL is just amazing. Basically it's just python functions describing what happens to data between input and output tables. Obviously it can do multiple inputs/outputs, obviously it has connectors to databases, filesystems, aws, google cloud, etc... It's smart enough to build correct orders of execution for complicated branching pipelines. It's incremental by design, so tricky things like redis caches with guaranteed consistency kinda just work out of the box. If datapipe becomes a bit user-friendly - it will become an ETL standard.

Datapipe is a great tool for creating complex and large data processing pipelines. The killer feature of this tool is, of course, incremental calculation. That is, there will be no need to run operations on the data for which everything has already been calculated.

Datapipe is a python library designed to help with organizing how we handle data in our projects. It's all about making sure that whenever we work with a lot of information, the system knows exactly which pieces of data are new or have changed. This way, we don't waste time or resources re-doing calculations on data that hasn't changed at all. The main idea is pretty straightforward: Datapipe keeps track of all the data and any updates to it. So, if something in the data changes or if we add something new, Datapipe makes sure that only these new or updated parts are processed. This makes our work more efficient because we're not going over the same data again and again. It's an approach to solving a common problem many of us face when dealing with big sets of data. By focusing on just the updates, Datapipe helps us keep our projects effective in terms of data processing, ensuring we're only working on what really needs attention.