really odd test bug?

Help
Anonymous
2013-01-16
2013-11-04

  • Anonymous
    2013-01-16

    So was trying to run the various tests, and it failed on pest_test3.py.  After a bit of poking around, I narrowed it down to the following bizarre behavior:

    When I run it through os.system (as it is done in the run_all_tests script), like:

    glop-/Users/bblais/python/lib/PyDSTool/tests-> python
    Enthought Python Distribution - www.enthought.com
    Version: 7.3-2 (32-bit)

    Python 2.7.3 |EPD 7.3-2 (32-bit)| (default, Apr 12 2012, 11:28:34)
    on darwin
    Type "credits", "demo" or "enthought" for more information.
    >>> import os
    >>> os.system('python pest_test3.py')

    # ends with…

    *** CHOSE pars with residual 7.64718218
    Feature evaluation on solution set-up:  False
    geom feat residual:  76.4585619437

    Preparing plots
    Valid coordinate names:
    Traceback (most recent call last):
      File "pest_test3.py", line 227, in <module>
        plotData_par = sol_traj.sample()
      File "/Users/bblais/python/lib/PyDSTool/Trajectory.py", line 554, in sample
        res = self.__call__(, coords_sorted)
      File "/Users/bblais/python/lib/PyDSTool/Trajectory.py", line 351, in __call__
        raise ValueError("Invalid coordinate names passed")
    ValueError: Invalid coordinate names passed
    256

    Now, if I do from the command line (which should be the same thing):

    /Users/bblais/python/lib/PyDSTool/tests-> python pest_test3.py

    the test passes! (the plot window comes up, there are a few matplotlib user warnings, but everything is fine).

    I've never seen behavior like this.  Is there a path problem?  I checked that the path is the same, and /Users/bblais/python/lib/PyDSTool/ is in the python path.  Not sure what else to check, since os.system should be the same as running it from the command line!

    thanks!

     
  • Rob Clewley
    Rob Clewley
    2013-01-18

    Hi,

    I think (but I'm not certain) that this is related to a path problem, because a different version of python and/or some of its dependencies are behaving slightly differently. It's not clear how that is happening and I'm not familiar with how EPD configures paths etc. or what your shell aliases or other .profile settings are.

    I have noticed this weird error come up for some people recently, and I don't understand why some standard semantics would have changed in recent versions. However, there have been more issues arising installing on Macs in the past year. I will be posting a fully updated set of install instructions for recent Mac OS versions soon, so look on the GettingStarted page for the update and let me know if that doesn't resolve the problem.

     
  • Hi,

    FYI, in my environment 'pest_test3.py' demonstrates curious non-deterministic
    behaviour: sometimes it fails, sometimes does not. Probability of failure is
    about 0.25.

    I've made several run series with following results:
    First run:
        Failure = 29 / 100 (29.00%)
        Failed runs: 0,7,8,12,15,18,23,32,33,35,37,41,43,45,46,57,59,68,69,71,73,80,81,82,87,89,90,95,97

    Second run:
        Failure = 25 / 100 (25.00%)
        Failed runs: 1,17,27,28,29,32,33,43,44,48,49,52,53,54,55,57,58,63,66,70,74,75,83,87,93

    Third run:
        Failure = 28 / 100 (28.00%)
        Failed runs: 7,9,10,14,15,18,26,30,35,36,37,39,41,45,46,56,58,59,61,62,65,72,74,77,78,82,84,86

    Fourth (huge) run:
        Failure = 263 / 1000 (26.30%)
        Failed runs: (omitted)

    All successful output files are identical. Same for failed runs output.
    Difference between successful and failed output (with matplotlib warnings
    wiped out):

    --- success.dat 2013-05-22 11:14:29.978708613 +0400
    +++ failure.dat 2013-05-22 11:14:16.694594265 +0400
    @@ -554,3 +554,12 @@
    geom feat residual:  76.4584960857
    Preparing plots
    +Valid coordinate names: ['sptime', 'spval']
    +Traceback (most recent call last):
    +  File "pest_test3.py", line 232, in <module>
    +    plotData_par = sol_traj.sample(['v'])
    +  File "/home/vovka/.envs/science/local/lib/python2.7/site-packages/PyDSTool/Trajectory.py", line 593, in sample
    +    res = self.__call__([self.indepdomain.get()], coords_sorted)
    +  File "/home/vovka/.envs/science/local/lib/python2.7/site-packages/PyDSTool/Trajectory.py", line 390, in __call__
    +    raise ValueError("Invalid coordinate names passed")
    +ValueError: Invalid coordinate names passed
    

    Script I've used:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    #!/usr/bin/env bash
    COUNTER=0
    if [ $# -lt 1 ]; then
        NRUNS=100
    else
        NRUNS=$1
    fi
    rm -f pest*.dat
    while [ $COUNTER -lt $NRUNS ]; do
        python pest_test3.py > pest$COUNTER.dat 2>&1
        let COUNTER+=1
    done
    FAILURE=`grep -l '^Traceback' pest*.dat | wc -l`
    echo "Failure = $FAILURE / $NRUNS (`echo -e \"scale=2\n$FAILURE * 100 / $NRUNS\" | bc -l`%)"
    echo "Failed runs:" `grep -l '^Traceback' pest*.dat | sed -e 's/pest\(.*\).dat/\1/g' | sort -n | tr "\\n" ", "`
    

    My environment:
    Linux 3.8.0-22-generic i686 (Ubuntu Linux 13.04)
    Python 2.7.4
    numpy-1.7.1
    scipy-0.12.0

    PS.  Memory corruption bug somewhere in PyDSTool-scipy/numpy-Python stack?
    I'll try to investigate further. Any advice what to try?

     
  • Rob Clewley
    Rob Clewley
    2013-05-22

    I've seen this and been baffled by it too. I don't have any insights right now. Does it also happen for pest_test3_Cintegrator?

     
  • It seems, I've found the problem. It's in these comments and code from 'pest_test3.py':

    # solution trajectory involving voltage happens to be the second of the
    # two trajectories stored in each log (one for each model interface, and
    # stored in order of the names of the interfaces).
    sol_traj = pest2.log[log_ix].trajectories[-1]
    

    If model interfaces were ordered by name, solution trajectory involving
    voltage would be the _first_ of the two ('int_geom_iface' < 'int_spike_iface').

    Actually, interfaces are ordered by object identity ("address") of
    appropriate "type" in 'context' construction:

    pest_context = context([ (spike_interface, int_spike_iface),
                             (geom_interface, int_geom_iface) ])
    

    If, by any means, 'int_geom_iface' gets lower address than 'int_spike_iface',
    script 'pest_test3.py' fails.

    Tracing code:

    diff -r cc872dcafa84 MProject.py
    --- a/MProject.py
    +++ b/MProject.py
    @@ -1013,6 +1013,9 @@ class context(object):
         """
         def __init__(self, interface_pairs, debug_mode=False):
             self.interfaces = dict(interface_pairs)
    +        import sys
    +        for k,v in self.interfaces.iteritems():
    +            sys.stderr.write('Iface: {}, Id: {}\n'.format(v, id(v)))
             self.debug_mode = debug_mode
             # Determine which qt features have metrics to use to make a
             # residual function. Keep multiple views of this data for efficient
    diff -r cc872dcafa84 common.py
    --- a/common.py
    +++ b/common.py
    @@ -20,6 +20,7 @@ from numpy import Inf, NaN, atleast_1d, 
          complex_, complex64, complex128, argmin, argmax
     from numpy.linalg import norm
     from math import sqrt
    +import sys
    
     try:
         from numpy import float96
    @@ -1890,7 +1891,9 @@ def sortedDictLists(d, byvalue=True, onl
             onlykeys = d.keys()
         if byvalue:
             i = [(val, key) for (key, val) in d.items() if key in onlykeys]
    +        sys.stderr.write('before sorting: {}\n'.format([(v, id(v)) for v,_ in i]))
             i.sort()
    +        sys.stderr.write('after sorting: {}\n'.format([(v, id(v)) for v,_ in i]))
             if reverse:
                 i.reverse()
             rvals = [val for (val, key) in i]
    

    So all scripts that rely on certain order of model interfaces in 'context'
    value are vulnerable. But, due to the nature of the bug, it's possible
    never to be hit by it.

    I haven't dived deep in 'MProject.py', so I'm not sure how properly fix the bug to guarantee certain order of interfaces.
    Maybe replace 'dict' with 'OrderedDict' in context.__init__? Though 'OrderedDict' appears only in Python 2.7.

     
  • Rob Clewley
    Rob Clewley
    2013-05-23

    Ah I see, thanks. I'll put a solution in the next release!

     
  • In conclusion, serie of 2500 'pest_test3_Cintegrator.py' runs finished without failures on my Linux box. On my FreeBSD box (FreeBSD-current/amd64, python 2.7.5, numpy-1.7.0, scipy-0.11.0) even 'pest_test3.py' (serie of 1000 runs) runs without failures.

    PS. By the way, I've submitted patch which adds FreeBSD support to PyDSTool

    https://sourceforge.net/tracker/?func=detail&aid=3610464&group_id=140858&atid=747667