A theory...

If I remember correctly, the nosttests was set up to execute in parallel using the default Multiprocessing settings, which is to have a process worker for each available CPU core. Perhaps this might be the crux of the issue with so many simultaneous tests running that the amount of memory used at the same time becomes too large. Or, am I thinking of the doc build system?

Ben Root


On Wed, Jun 4, 2014 at 12:20 PM, Nelle Varoquaux <nelle.varoquaux@gmail.com> wrote:
> Our standard test has gotten out of control.  The most serious problem is
> that running a full test suite now fails on a linux VM with 4 GB--it's out
> of memory. Half-way through the set, it is already using more than 2 GB.
> That's ridiculous.  Running nosetests separately on each test module keeps
> the max reported by top to 1.6 GB, and the max by report_memory to 0.5 GB;
> still quite a bit, but tolerable.  (I don't know why there is this factor of
> 3 between top and report_memory.)  This scheme of running test modules one
> at a time also speeds it up by a factor of 2; I don't understand why.
>
> The script I used for the module-at-a-time test is attached.  It is a
> modification of matplotlib.tests().
>
> Are there any nosetest experts out there with ideas about how to streamline
> the standard test routine?

This issue is probably worth mentionning on other mailing list of
people using nosetest, and nosetests.
I'm thinking of scikit-learn in particular, which also uses nosetest
heavily. The scipy-users list might be a good place to exchange
experience.

N

>
> Eric
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel