DSPSR: Digital Pulsar Signal Processing / Bugs / #114 GUPPI File read error

Willem van Straten - 2024-05-24

Hi Mugundhan, could you please re-run digihdr with an upper-case -V flag? This might print a bit more information about why the GUPPIFile class is not recognizing your data file.

Cheers,
Willem

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mugundhan - 2024-05-25

Hi Willem,

Thanks for your quick response! :)

I got this output when i did digihdr with -V.

opening data file /mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
dsp::File::create filename='/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
dsp::File::create with 10 registered sub-classes
dsp::File::create testing Dummy
dsp::DummyFile::is_valid first line != DUMMY
dsp::File::create testing DADA
dsp::DADAFile::is_valid file has no header and no matching header file found
dsp::File::create testing FITSFile
FITSFile::is_valid fits_open_file: 1st key not SIMPLE or XTENSION
dsp::File::create testing GMRT
dsp::GMRTFile invalid band_dir=538976288
dsp::File::create testing GMRT
dsp::GMRTBinaryFile::get_header could not open '/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw.hdr'
dsp::GMRTBinaryFile::is_valid empty header
dsp::File::create testing GMRTFilterbank
dsp::GMRTFilterbankFile::is_valid /mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw does not end int 'dat'
dsp::File::create testing GUPPI
dsp::File::create GUPPI::is_valid() returned true
dsp::Input ctor this=0x55af6864ab60
MJD (double mjd=0)
MJD this=0
MJD (double mjd=0)
MJD this=0
MJD (double mjd=0)
MJD this=0
dsp::File::open filename=/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
MJD (int intday=60353, double fracday=0.370398)
MJD this=60353.4
dsp::GUPPIBlockFile::parse_header ra_str=19:21:44.815008 dec_str=+21:53:02.24988
dsp::File::create failed while testing GUPPI
Error parsing block1 header.
dsp::File::create testing SigProc
MJD (double mjd=0)
MJD this=0
sigproc::read_header - no HEADER_START
dsp::File::create failed while testing SigProc
read_header failed
dsp::File::create testing MultiFile
dsp::MultiFile::is_valid meta filename=/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
dsp::MultiFile::is_valid 1 filenames read
dsp::MultiFile::is_valid 'SRC_NAME=' not found
dsp::File::create testing Multiplex
dsp::Multiplex::is_valid meta filename=/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
dsp::Multiplex::is_valid 1 filenames read
dsp::Multiplex::is_valid 'SRC_NAME=' invalid format
Error thrown:
Error::stack
dsp::File::create
Error::InvalidParam
Error::message
/mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw not a recognized file format
10 registered Formats: Dummy DADA FITSFile GMRT GMRT GMRTFilterbank GUPPI SigProc MultiFile Multiplex

I have used GUPPI files created same way with another container, where I had dspsr running, and digihdr used to work. I was trying to build the container where I get this error using instructions from here: https://github.com/louisbondonneau/Docker_receipts/blob/main/psrchive_py3/Singularity

But I did not use the same tags for the git repos said in this singularity file.

Regards,

Mugundhan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Willem van Straten - 2024-05-26

Hi Mugundhan, Thanks for sending the verbose output. The following error message is specific to the GUPPI file loader:

Error parsing block1 header.

and is output when another function named get_header returns 0; this function is defined in Kernel/Formats/guppi/GUPPIFile.C and is preced by the following note

// TODO could rewrite this using the fitshead.h stuff now, but // this already exists..

Perhaps it is time to rewrite this code using fitshead.h stuff, or perhaps there is a different bug.

If you could send a link to a data file, I could try to reproduce the error on my end.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mugundhan - 2024-05-27

Hi Willem, Please find the link to the data file attached.

https://chalmers-my.sharepoint.com/:i:/g/personal/mugvij_chalmers_se/EYM1sOmoOZVPtxypM7-pDtkBVHWPkeFKtkNIqfx_GeM59g?e=HjD8ng

It contains 16.384 seconds of data over ~ 24 MHz of bandwidth. The data has 122 channels (complex data), 2 polarizations.

Regards,

Mugundhan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Willem van Straten - 2024-05-27

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Mugundhan - 2024-05-27
  
  Hi Willem,
  
  This is because there is one data block in the file containing 3200000 time samples, where each sample is 2 bytes (1 byte real + 1 byte imaginary), and 2 polarizations, of 122 channels. This comes up to a block-size of 1561600000 (3200000x2bytesx2polsx122channels).
  
  Have I understood this correctly?
  
  Regards,
  Mugundhan
  
  From: bugs@dspsr.p.re.sourceforge.net bugs@dspsr.p.re.sourceforge.net on behalf of Willem van Straten straten@users.sourceforge.net
  Sent: Monday, May 27, 2024 12:22 PM
  To: [dspsr:bugs] 114@bugs.dspsr.p.re.sourceforge.net
  Subject: [dspsr:bugs] #114 GUPPI File read error
  
  Comment:
  
  Hi Mugundhan, Thanks for sending a link to the data file.
  
  The header of this file contains the string
  
  BLOCSIZE= 1561600000
  
  but this number is close to the size of the file and it seems too large. Should it be something more like the PKTSIZE = 7808 ?
  
  [bugs:#114]https://sourceforge.net/p/dspsr/bugs/114/ GUPPI File read error
  
  Status: open
  Group: version_1.0
  Created: Fri May 24, 2024 11:31 AM UTC by Mugundhan
  Last Updated: Mon May 27, 2024 07:38 AM UTC
  Owner: nobody
  
  Hi,
  I have installed dspsr in a container, and am trying to read some GUPPI files I created through digihdr. The command goes like this:
  digihdr -v /mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
  
  This throws this error:
  
  opening data file /mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw
  Error thrown:
  Error::stack
  dsp::File::create
  Error::InvalidParam
  Error::message
  /mnt/mugundhan/1d46be89-271a-4dcc-aca0-a12aafaf2b1d/data/proj51/sess_sid20240213T085300_SE607/raw/B1919+21_60353_32002_16070.raw not a recognized file format
  10 registered Formats: Dummy DADA FITSFile GMRT GMRT GMRTFilterbank GUPPI SigProc MultiFile Multiplex
  
  I'm able to open the guppi file, and read its header using the baseband python package, and the header has all the basic information needed. Any advice would be very helpful!
  
  Thanks,
  
  Mugundhan
  
  Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dspsr/bugs/114/
  
  To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
  
  Related
  
  Bugs: #114
  
  alternate
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Willem van Straten - 2024-05-27
    
    I see. Thanks for the clarification. My assumption that the block size should be small was incorrect. e.g.
    
    https://www.cv.nrao.edu/~pdemores/GUPPI_Raw_Data_Format/
    
    also describes a gigabyte block sizes.
    
    I think that there is a built-in assumption that a GUPPI file must contain more than one block. At least, the algorithm appears to be searching for the header of the second block (at the end of the file) ...
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Willem van Straten - 2024-05-27

Hi Mugundhan, Thanks for sending a link to the data file.

The header of this file contains the string

BLOCSIZE= 1561600000

but this number is close to the size of the file and it seems too large. Should it be something more like the PKTSIZE = 7808 ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Willem van Straten - 2024-05-27

Hi Mugundhan, I've checked in a potential fix.

Could you please try it out on your end?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mugundhan - 2024-05-28

Hi Willem,

Thank you for the fix, digihdr seems to work now, and prints the header without any errors. I'll try and see if other things work as well.

Regards,

Mugundhan :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Paul Demorest - 2024-05-29

Thanks Willem. You are right that there was an implicit assumption that a file would contain more than one data block. I don't remember details at this point but it looks like there must have been cases when the first data block was bad and needed to be ignored. I quickly skimmed your changes and they all seem reasonable to me!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

GUPPI File read error

Group

Searches

Help

#114 GUPPI File read error

Related

Discussion

Related