#875 fsm.getRepresentativeFile lists too much

[Jeremy Faden]

FileStorageModelNew.getRepresentativeFile lists all the parents completely, then picks one file from one of those. This is wasteful and I've noticed excessive hits on my server, and I just ran into the problem again when working with the POES datasets on virbo. These have one zip file per file, so listing all the parents means listing the whole set.

I changed the logic, so it now calls parent.getRepresentativeFile, and only works with that. This has the likely problem that there may be empty files in the parent that don't contain representatives for the child filesystem. We need to add a fallback so that we always find a representative file.

Something like this will show the bug:
mkdir 2010
mkdir 2011
mkdir 2012
touch 2011/2011.dat

I think 2012 will be the parent representative, but this will not contain 2012.dat.

[Jeremy Faden]

This is a perennial bug that is usually not a problem so it doesn't get fixed. I just found that when listing:

vap+cdf:https://emfisis-soc.physics.uiowa.edu/Flight/RBSP-A/L4/$Y/$m/$d/rbsp-a_density_emfisis_$Y$m$d_v$(v,sep).cdf?digi_type&timerange=2013-05-01

it fails in this mode, issuing the statement

fall back to old code that would list everything
get representative from wfs https://emfisis-soc.physics.uiowa.edu/Flight/RBSP-A/L4/ (..) range: June 2015
get representative from wfs https://emfisis-soc.physics.uiowa.edu/Flight/RBSP-A/L4/ (..) range: June 2015

many times, getting stuck and never succeeding. This is when loading test140's file:///home/jbf/project/autoplot/tests/pw/ordinalCdf/ordinalInCDF.vap on the tests at https://ci-pw.physics.uiowa.edu/job/autoplot-test140/540/console .

[Jeremy Faden]

I need to identify the conditions that create this bug (probably that the last $Y doesn't contain a file).

[Jeremy Faden]

1. paste in the data set selector, "vap+cdf:https://emfisis-soc.physics.uiowa.edu/Flight/RBSP-A/L4/$Y/$m/$d/rbsp-a_density_emfisis_$Y$m$d_v$(v,sep).cdf?digi_type&timerange=2013-05-01"
2. push the "inspect" button to enter the editor. Autoplot hangs.

[Jeremy Faden]

This script demos the bug (on 2019-04-07T07:48-0500):

# demo bug 0875
from org.das2.fsm import FileStorageModel
from org.das2.util.filesystem import FileSystem
fs= FileSystem.create('https://emfisis-soc.physics.uiowa.edu/Flight/RBSP-A/L4')
fsm= FileStorageModel.create( fs,'$Y/$m/$d/rbsp-a_density_emfisis_$Y$m$d_v$(v,sep).cdf' )
print fsm.getRepresentativeFile(monitor)

[Jeremy Faden]

The problem in this case is we got rid of all the old files which did not have L4 in the title, and it is exhaustively searching for one. This result is incredibly opaque. It would be nice if near misses could be reported, or some other procedure for identifying this common use case identified. Script looked for rbsp-a_density_emfisis_$Y$m$d_v$(v,sep).cdf, the folders now only contain rbsp-a_density_emfisis-L4_$Y$m$d_v$(v,sep).cdf . Perhaps there should be a rule that if the first year and last year contain no files, then it gives up looking.

[Jeremy Faden]

I cleaned this up earlier this year. I'm closing this old ticket.