Menu
▾
▴
#25 process files in order in input file list
It would be nice if the create_file_iterator_list (or whichever that reads the input file list) created an iterator with them in the same order.
I know for things in parallel that doesn't really mean anything, but for something that processes in serial it would be useful in some cases.
Discussion
Logged In: YES
user_id=1016136
Originator: NO
Battery dying ... some notes real quick and I'll finish thoughts later:
While the create_iterator_groups.pl script has a --randomize option to control whether the inputs are shuffled, it is hard-coded to 1 in $;DOCS_DIR$;/iterator_template.xml
Logged In: YES
user_id=1016136
Originator: NO
To expand on my last comment, we never 'shuffle' steps within a component. So if you have steps coded to be run serially, they will be executed in the exact order they are defined in the XML file. What can be shuffled are the elements within an input list if the component is designed to distributed across groups to be submitted to a grid. This was very much by design, though through the underlying scripts we could make it optional for some components.
The idea for the input shuffling (actually via a fisher-yates shuffle) is that our first use case were the processing of assemblies whose files were named like this:
aa1.assembly.1.fsa
aa1.assembly.2.fsa
aa1.assembly.3.fsa
aa1.assembly.4.fsa
... and so on ...
The first numbers were always the largest assemblies, so if we divided these up into groups without shuffling some groups would represent much larger units of work than others, which is an inefficient use of the grid resources. The randomization of input files into the submitted groups helped to 'even out' the workload across grid nodes.
Are there any particular components you're trying to use where this behavior is not desired?
Logged In: NO
Yes, obo2chado to load ontologies. However, some of the ontologies have "dependencies" stored in the relationship ontology(ies), so they need to be loaded first.
What if we made the iterator not do any re-ordering of files in a list, but instead we create a 'reorder_list' component? It could have options for 'random', 'timestamp', or for creating bins of equivalent total-size-of-all-files.
Logged In: YES
user_id=1016136
Originator: NO
If the order of input files matters it *never* makes sense to distribute them into groups and submit those groups as jobs on a grid because you'll never be able to guarantee the order in which those groups are processed, they are individual grid jobs subject to the grid's scheduling rules. The default file iterator template assumes this isn't the case, but for any component that has specific needs a custom iterator could be written.
For this one, is it fair to say you want a single job to be submitted to the grid (or run locally) and within that job is a list of obo files to be loaded in order? If so we can change the option for that component to force only a single group to be created and for the input shuffling to be not occur.
I don't like the idea of making the input shuffling a component itself, because it's by far the most common behavior and the step itself is more atomic than I want general Ergatis components to be. At best, there can be a checkbox on each component to control whether input is shuffled and it can have some default value.
Logged In: YES
user_id=1766179
Originator: YES
Good point about not being able to ensure ordering between groups out on the grid. The obo2chado run I want is already creating just one group, but it randomizes the order of the files within the group. I don't know if there are other components or not that could require ordered inputs within a group (or individually ordered multiple groups).
Also, would "efficient packing" of groups according to a user-selectable algorithm be useful? If so, then I would assume you'd want that as its own component rather than adding it as options to everything that might use it.
Under the existing structure would non-randomizing iterators have to be defined on a per component basis or is there a more-generic way of changing it?