Search Results for "flat file databases"

...You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else. You can build pretty much any task you want, but Luigi also comes with a toolbox of several common task templates that you use. It includes support for running Python mapreduce jobs in Hadoop, as well as Hive, and Pig, jobs. It also comes with file system abstractions for HDFS, and local files that ensures all file system operations are atomic.

Dagger evaluates file dependencies in a directed-acyclic-graph (DAG) like GNU make, but timestamps or hashes can be used on a per-file basis when evaluating which file nodes are up to date or stale. Use fast timestamp comparisons on large files when hashing is too slow, and hashing on small files. When hashing is used, it's stored in either a text file or sqlite database (in-memory at runtime is optional). Files can be forced as stale or uptodate. Dagger can be used as a building block for a...

Set of tools, web based and command line python based, to manage the status of builds on a linux based system. Ala Tinderbox style, task status (test, build, etc) are send via email to a rails engine that monitors build status and displays it on a web.