#1 Worker 'Push' based task completion

[Kyle Ellrott]

PIPPER utilizes a Master 'pull' method for work assignment, ie a task is not done until the master tells the worker to do it. The master has a full copy of the pipeline graph and calculates what work is ready and then tells workers to do it. The can cause latency problems as workers will sit idle with ready-to-do work waiting for the master to get to them. In situation with communication latency or high task counts, all work becomes limited by the speed of the master. The workers have subsets of the pipeline graph, but they can use these to determine what work can be done given the subset of data the hold.

Fix: Workers would scan their pipeline graphs, determine what work they can do, automatically do it, and return results to the master without waiting for the commands to do so. The master would be switched to a 'sit and listen' style of command when not distributing tasks or merging results.

Complications: Have to make sure that this new model doesn't cause lockups and communication weirdness, because now the master wouldn't know exactly what a worker was doing and any given moment.