#445 general speed improvements

[Jeremy Faden]

No one is asking for this, but I'm sure there are performance enhancements to be made. For example, this past week I discovered that the huge scatter rendering method was calculating the cadence each time it was rendered, dramatically affecting performance. Fixing that is fairly inexpensive (easy to show that no new bugs are introduced), and it's worthwhile to go ahead low-hanging apples now and again.

I'm coming up with a set of benchmark scripts that can be used for testing now.

[Jeremy Faden]

I started adding logger.entering and logger.exiting messages in the middle of oft-used routines, such as extent.

[Jeremy Faden]

Another dirty secret of Autoplot is that fill is checked with fuzz, because the data is accessed as doubles. I should probably introduce the rule that the fill value must be a double.

[Jeremy Faden]

I experimented with doing a brute-force implementation of range, which avoids using the DataSetIterator. Comparing this to the simpleRange used in autorange, and Ops.extent:

 * Ops.extent: 53ms
 * simpleRange: 77ms
 * study445FastRange: 4ms


 * Ops.extent: 76ms 
 * simpleRange: 114ms 
 * study445FastRange: 12ms

It seems clear that the DataSetIterator performance penalty is bigger than I would have imagined, and I should start recoding hotspots to avoid its use.

Here's the study code:

# Does DataSetIterator introduce an undo burden?  It could easily be replaced for rank 1 cases.
# https://sourceforge.net/p/autoplot/feature-requests/445/

from java.lang import System
#ds= getDataSet('/home/jbf/ct/autoplot/data.backup/examples/cdf/huge/rbspa_l1_mscb1_20150203_v02.cdf?mscb1[0:67108863]&slice1=0')
ds= getDataSet('/home/jbf/ct/autoplot/data.backup/examples/cdf/huge/rbspa_l1_mscb1_20150203_v02.cdf?mscb1[0:10000000]&slice1=0')
#ds= getDataSet('/home/jbf/ct/autoplot/data.backup/examples/cdf/huge/rbspa_l1_mscb1_20150203_v02.cdf?mscb1[0:1000000]&slice1=0')
#ds= getDataSet('ftp://sidads.colorado.edu/pub/DATASETS/AMSRE/SAMPLES/AMSR_E_L3_5DaySnow_B07_20070126.hdf?Northern%20Hemisphere/Data%20Fields/SWE_NorthernPentad')

t0= System.currentTimeMillis();
dd1= extent(ds)
dt= System.currentTimeMillis()-t0
print dt, dd1.value(0), dd1.value(1)

from org.virbo.autoplot import AutoplotUtil
t0= System.currentTimeMillis();
dd1= AutoplotUtil.simpleRange(ds)
dt= System.currentTimeMillis()-t0
print dt, dd1[0], dd1[1]

from org.virbo.autoplot import AutoplotUtil
t0= System.currentTimeMillis();
dd1= AutoplotUtil.study445FastRange(ds)
dt= System.currentTimeMillis()-t0
print dt, dd1[0], dd1[1]

[Jeremy Faden]

I was looking at the LanlNNRebinner for low-hanging fruit, and I've made it avoid calls to locate the y pixel locations repeatedly when the y tags are rank 1. I was able to get about a factor of two speed up from this.

Also I played with caching the RebinDescriptor units converter, where I can get a factor of two improvement. This is simply fast, but for huge rank 1 datasets this effect may be visible. See RebinDescriptorBenchmark.

[Jeremy Faden]

I spent some time back in May 2019 looking for hot spots in the code which would benefit from optimization. The simpleRange code in AutoRangeUtil has the rank 1 branch where it zips through without using an DataSetIterator.

Masafumi just pointed out that this is now causing problems for his code because of a difference in how infinite data is handled, and I've tracked this down (to between 20190420a and 20190427a) with the testagator. See https://github.com/autoplot/dev/tree/master/bugs/2019/20190815 .

[Jeremy Faden]

See also https://sourceforge.net/p/autoplot/bugs/1801/