Menu
▾
▴
Re: [SloppyCell-users] Error when trying to use mpi parallelism
|
From: Gutenkunst, R. N - (rgutenk) <rg...@em...> - 2015-04-24 19:49:46
|
Hi Ally, If the model/data comparison involves multiple Networks, then the integrations for those multiple networks are run in parallel. If you're problem uses only a single network, then you won't see any parallelism at that level. Best, Ryan On Apr 24, 2015, at 12:40 PM, Ally Hume <a....@ed...> wrote: > Fair point! I never noticed parallelism speed up the hessian calculation so I assumed there was no parallelism is that function. I'll investigate next week why I saw no speed up. > > Regards, > > Ally > > Quoting "Gutenkunst, Ryan N - (rgutenk)" <rg...@em...> on Fri, 24 Apr 2015 16:42:47 +0000: > >> Hi Ally, >> >> Yes, the hessian calculations should be easily parallelizable. The tricky part will be ensuring that if that level is parallelized, then parallelization for the lower level cost calculations might need to be disabled, because I'm not sure how it will behave with mpi calls within mpi calls. >> >> Best, >> Ryan >> >> On Apr 24, 2015, at 9:03 AM, Ally Hume <a....@ed...> wrote: >> >>> Ryan, >>> >>> Fix seems to have worked well. >>> >>> I'm now starting to think about possible other parallelisations and was considering looking at the hessian() and hessian_log_params() as these can take while and there is a nice loop in there that looks easily parallelisable at first glance. It's just a suggestion and I cannot guarantee getting round to doing it but I was just wondering if you had any comments about the idea? >>> >>> I would, of course, contribute back any code I do develop. >>> >>> Regards, >>> >>> Ally >>> >>> >>> Ally Hume >>> Software Architect >>> EPCC, The University of Edinburgh >>> Tel: 0131 651 3397 Skype: ally.hume >>> >>> >>> >>> >>> >>> >>> >>> >>> On 18 Apr 2015, at 00:42, "Gutenkunst, Ryan N - (rgutenk)" <rg...@em...> wrote: >>> >>>> Hi Ally, >>>> >>>> First, thank you for the very thorough bug report! It was actually two bugs. The first was a simple typo bug in using the N_dyn_vars name. The second more subtle bug had to do with how events are handled in the integration when running in parallel. Both should now be fixed in the git repository. >>>> >>>> The parallel code hasn't been thoroughly tested in a while. To help ensure there aren't other lingering bugs, please advise the researcher to compare results of a parallel and non-paralllel jacobian evaluation to ensure they're equal, before running extensive calculations. >>>> >>>> Best, >>>> Ryan >>>> >>>> On Apr 17, 2015, at 9:39 AM, Ally Hume <a....@ed...> wrote: >>>> >>>>> I have trying to use the MPI parallelism functionality of sloppy cell and the code is hanging and simply never terminates. >>>>> >>>>> I have debugged it a bit and now know a more about the cause but I would need to understand the code much better to go any further. >>>>> >>>>> The Dyamics.integrate_sensitivity function sends out the tasks to the workers, processes some data at the master and then waits for each worker to return. When worker 1 returns the result it is parsed by _parse_sens_result. This function is throwing a NameError exception which is not caught here so the code does not try to get a result from workers 2,3,4... and hence these workers never terminate and the whole job hangs. >>>>> >>>>> So why does _parse_sens_result throw an exception? Because N_dyn_vars is not defined. If I record the stack trace I get: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/exports/work/physics_epcc/ahume/sloppycell_anaconda/lib/python2.7/site-packages/SloppyCell-CVS-py2.7-linux-x86_64.egg/SloppyCell/ReactionNetworks/Dynamics.py", line 958, in integrate_sensitivity >>>>> _parse_sens_result(result, net, vars_assigned[worker], yout, youtdt, events_occurred) >>>>> File "/exports/work/physics_epcc/ahume/sloppycell_anaconda/lib/python2.7/site-packages/SloppyCell-CVS-py2.7-linux-x86_64.egg/SloppyCell/ReactionNetworks/Dynamics.py", line 869, in _parse_sens_result >>>>> e.ysens_fired = scipy.concatenate((e.ysens_fired, result[-1][eii].ysens_fired[N_dyn_vars:])) >>>>> NameError: global name 'N_dyn_vars' is not defined >>>>> >>>>> The line numbers do not match the code in git as I have added lots of debug lines to get this far. But you will see the line that is causing the problem is the first line in _parse_sens_result that uses the variable N_dyn_vars. >>>>> >>>>> Could we be doing something wrong or is this a bug? This all comes from a call to: Optimization.fmin_lm_log_params >>>>> >>>>> If need be I can ask the researcher for more details of the model. My role here is to support him by getting this running on a cluster. >>>>> >>>>> Any advise? >>>>> >>>>> Regards, >>>>> >>>>> Ally >>>>> >>>>> Ally Hume >>>>> Software Architect >>>>> EPCC, The University of Edinburgh >>>>> Tel: 0131 651 3397 Skype: ally.hume >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> The University of Edinburgh is a charitable body, registered in >>>>> Scotland, with registration number SC005336. >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT >>>>> Develop your own process in accordance with the BPMN 2 standard >>>>> Learn Process modeling best practices with Bonita BPM through live exercises >>>>> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ >>>>> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF >>>>> _______________________________________________ >>>>> SloppyCell-users mailing list >>>>> Slo...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/sloppycell-users >>>> >>>> -- >>>> Ryan Gutenkunst >>>> Assistant Professor >>>> Molecular and Cellular Biology >>>> University of Arizona >>>> phone: (520) 626-0569, office LSS 325 >>>> http://gutengroup.mcb.arizona.edu >>>> >>>> >>> >>> >>> -- >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >> >> -- >> Ryan Gutenkunst >> Assistant Professor >> Molecular and Cellular Biology >> University of Arizona >> phone: (520) 626-0569, office LSS 325 >> http://gutengroup.mcb.arizona.edu >> >> >> > > > > ---------------------------------------------------------- > Ally Hume > Software Architect > EPCC, The University of Edinburgh > Tel: +44 131 651 3397 > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > -- Ryan Gutenkunst Assistant Professor Molecular and Cellular Biology University of Arizona phone: (520) 626-0569, office LSS 325 http://gutengroup.mcb.arizona.edu |
