#102 DigitiserCounts problem with .fil input

[Paul Demorest]

Trying to fold filterbank (aka sigproc) input files into PSRFITS output files segfaults at the step of writing the DigitiserCounts extension. I'm using dspsr 2023.01.25 and psrchive 2023.01.25, have not tried more recent versions but I don't see anything in the commits that would have addressed this. Adding -j "ext remove DigitserCounts" to the command line works around the failure, but here is some more info in case we want a better fix.

Here is a snippet of the verbose output:

Pulsar::ProfileColumn::unload writing data
FITSArchive::unload_integration DATA written
FITSArchive::unload_integration finished
FITSArchive::unload_integrations exit
Pulsar::Archive::get<Ext> name=FITSHdrExtension
Pulsar::Archive::get<Ext> name=Backend
Pulsar::Archive::get<Ext> name=dspReduction
Pulsar::Archive::get<Ext> name=DigitiserCounts
Pulsar::Archive::get<Ext> name=TwoBitStats
Pulsar::Archive::get<Ext> name=Passband
Pulsar::Archive::get<Ext> name=Telescope
Pulsar::Archive::get<Ext> name=Receiver
Pulsar::Archive::get<Ext> name=ProcHistory
Pulsar::Archive::get<Ext> failed to find extension type=PKN6Pulsar19DigitiserStatisticsE
FITSArchive::delete_hdu DIG_STAT HDU deleted
Pulsar::Archive::get<Ext> name=FITSHdrExtension
Pulsar::Archive::get<Ext> name=Backend
Pulsar::Archive::get<Ext> name=dspReduction
Pulsar::Archive::get<Ext> name=DigitiserCounts
saving dyn_levt0
saving ndigr 4096
saving diglev FIX
saving dig_mode DSPSR
saving npthist 0
saving nlev 0
digitiser counts keys unloaded
Segmentation fault

I assume the problem arises because SigProcUnpacker has not filled in the necessary histogram info (ie, nlev and npthist both zero). I'm not sure whether this histogram even makes sense for already-detected input, but fixes could include one or more of:

1. Make SigProcUnpacker a Unpacker rather than a HistUnpacker (this would be consistent with FITSUnpacker).
2. or, have SigProcUnpacker fill in the histogram
3. Fix the unloader in PSRCHIVE so it doesn't segfault when given zeros

Thoughts on the best approach?

[Willem van Straten]

Hi Paul,

Thanks for the report and analysis! I suggest 3. then 1.

Cheers,
willem

[Paul Demorest]

OK thanks Willem! If I implement this I would probably do 1 first then 3 but I agree they are both worth doing.

[Willem van Straten]

I've made an attempt at 3 and added some additional checks to the Pulsar::FITSArchive::unload DigitiserCounts function.

b65ae1e5f..43de8f6c3 master -> master

[Paul Demorest]

That didn't actually fix the problem though they are probably good checks to keep. Turns out the segfault was in templates.h/minmax() which was not checking for an empty range before dereferencing. I fixed this in psrchive commit 3cbc71b, take a look for any issues.

This did fix the segfault, and dspsr now produces valid PSRFITS output. It's a bit noisy as it prints an error message (see below) on every output file. But I guess it's all handled correctly since the files appear valid and do not contain a DIG_CNTS HDU. So maybe a bit more cleanup in PSRCHIVE would be good, though I'm not planning to do this. I will still try item #1 from this list.

Error::stack
    FITSArchive::unload(DigitiserCounts)
    psrfits_write_col(vector<T>)
Error::FailedCall
Error::message
    fits_write_col (type=int col=3 row=1 size=0): bad first element number (308)

[Paul Demorest]

Turned out to be pretty easy, #1 is done in commit 02c0fa1. Seems to work and removes that error message. I'll close this unless you have any other ideas.

[Willem van Straten]

oops. I wasn't checking npthist (as noted in your original post!)

I've checked in another set of checks that should stop the unloader from getting to the point of the fits_write_col error.

c3785a206..469f61224 master -> master

[Willem van Straten]

excellent! I'm glad that #1 was also easy. I think the bug is solved on both sides of the fence :-)

[Paul Demorest]

Thanks Willem, all looks good to me!