Pipeline execution

We like it or not, we must prepare to reuse a memory allocated network bacause we can't allocate them all in what relates to the weights in the context of an ensemble method.

What is the best way to do so?

Remembering our old computer architecture courses at school, we may notice that we have all the qualities for a "RISK" pipeline execution. In a risk architecture, the instruction set is and must be simple. For instructions taking multiple cycles, if there is enough similarity between the instructions, we can begin the first cycle of an instruction and then execute the second cycle of the instruction at the next physical layer while beginning the first cycle of the next instruction at the first physical layer. Then the first instruction may execute its third cycle on the third physical layer, the second instruction its second cycle on the second physical layer and a new third instruction begins its first cycle on the first physical layer. Etc. Hence multiple instructions are computed at the same time much like multiple cars get assembled at the same time in an assembly line. The overall throughput is increased.

In our case, a network may be regarded as an instruction and since, in our design, the layers are executed one after the other, a layer execution may be regarded as an instruction cycle.

Of course, we must ensure that results of layer i are not overwritten by the excution of network x+1 before the execution of layer i+1 of network x has completed.

Considering that the physical memory for weights and neurons had been allocated for a single network, we have the following tasks to execute:

• Initialise input neurons;
• Initialise layer 1 weights for network i;
• Compute layer 1 for network i;
• Initialise layer 2 weights for network i;
• Compute layer 2 for network i;
• Initialise layer 3 weights for network i;
• Etc for all the layers and all networks

If we represent a dependency link between these tasks across the layers and across the networks by an arrow like A -> B to mean that the execution of B depends on the completion of A, we get the following dependency graph: