#78 problems with important digifits header and data table fields when processing GUPPI RAW
Description: I am trying to use digifits to produce psrfits
search-mode data from data in the GUPPI RAW format. The output cannot
be processed with presto tools or psrfits_utils tools because
important header and data table fields are not being recognized as valid.
Command: digifits -D 11.217356 -do_dedisp -t 40.96e-6 -p 4 -b 8 -K ../guppi_58626_J1125+7819_0013.0000.raw
Output from digifits:
digifits: requested tsamp=4.096e-05 rate=781250
actual tsamp=4.096e-05 (tscrunch=32)
digifits: input data not detected
digifits: performing coherent dedispersion at 11.2174
digifits: creating convolution operation
digifits: processing 256 channels
digifits: dedispersion filter length=128 (minimum=32) complex samples
digifits: convolution requires at least 128 samples
digifits: blocksize=128 samples or 2.625 MB
Finished 52.7 s (100%)
Example PRESTO command (rfifind): rfifind -o guppi_58626_J1125+7819_0013 -time 2 2019-05-23-11:27:53.sf
Example rfifind output:
Pulsar Data RFI Finder by Scott M. RansomAssuming the data are PSRFITS format...
Reading PSRFITS data from 1 file:
'2019-05-23-11:27:53.sf'Error 409 reading key TBIN
Error 409 reading key NCHAN
Error 409 reading key NPOL
Error 409 reading key POL_TYPE
Error 409 reading key NSBLK
Error 409 reading key NBITS
Warning!: All OFFS_SUB are 0.0. Assuming no missing rows.
Number of files = 1
Num of polns = 1
Center freq (MHz) = 1500.3906
Num of channels = 0
Sample time (s) = 0
Spectra/subint = 0
Total points (N) = 0
Total time (s) = 0
Clipping sigma = 6.000
Invert the band? = True
Byteswap? = False
Remove zeroDM? = False
Apply scaling? = False
Apply offsets? = False
Apply weights? = FalseFile Start Spec Samples Padding Start MJD
1 -9223372036854775808 0 0 0.00000000000000
Example fold_psrfits command: fold_psrfits -F 25 2019-05-23-11:27:53.sf
Example fold_psrfits output:
Found 1 valid PSRFITS files for input.
Opened file '2019-05-23-11:27:53.sf'FITSIO status = 409: bad string to double convert
Error in ffc2dd converting string to double: *
Error in ffc2d evaluating string as a double: '* 'Here is a dump of the primary and data headers using astropy:
In [1]: from astropy.io import fits
In [2]: f = fits.open("2019-05-23-11:27:53.sf")
In [3]: f[0].header
Out[3]:
SIMPLE = T / file does conform to FITS standard
BITPIX = 8 / number of bits per data pixel
NAXIS = 0 / number of data axes
EXTEND = T / FITS dataset may contain extensions
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
COMMENT FITS (Flexible Image Transport System) format defined in Astronomy and
COMMENT Astrophysics Supplement Series v44/p363, v44/p371, v73/p359, v73/p365.
COMMENT Contact the NASA Science Office of Standards and Technology for the
COMMENT FITS Definition document #100 and other FITS information.
HDRVER = '6.1 ' / Header version
FITSTYPE= 'PSRFITS ' / FITS definition for pulsar data files
DATE = '2019-11-13T15:43:59' / File creation date (YYYY-MM-DDThh:mm:ss UTC)
OBSERVER= ' ' / Observer name(s)
PROJID = ' ' / Project name
TELESCOP= 'GBT ' / Telescope name
ANT_X = '* ' / [m] Antenna ITRF X-coordinate (D)
ANT_Y = '* ' / [m] Antenna ITRF Y-coordinate (D)
ANT_Z = '* ' / [m] Antenna ITRF Z-coordinate (D)
FRONTEND= 'Rcvr1_2 ' / Receiver ID
IBEAM = ' ' / Beam ID for multibeam systems
NRCVR = '* ' / Number of receiver polarisation channels
FD_POLN = 'LIN ' / LIN or CIRC
FD_HAND = 1 / +/- 1. +1 is LIN:A=X,B=Y, CIRC:A=L,B=R (I)
FD_SANG = 45. / [deg] FA of E vect for equal sig in A&B (E)
FD_XYPH = 0. / [deg] Phase of A^* B for injected cal (E)
BACKEND = 'GUPPI ' / Backend ID
BECONFIG= ' ' / Backend configuration file name
BE_PHASE= 1 / 0/+1/-1 BE cross-phase:0 unknown,+/-1 std/rev
BE_DCC = 0 / 0/1 BE downconversion conjugation corrected
BE_DELAY= 0. / [s] Backend propn delay from digitiser input
TCYCLE = '* ' / [s] On-line cycle time (D)
OBS_MODE= 'SEARCH ' / (PSR, CAL, SEARCH)
DATE-OBS= 'UNSETTUNSET' / Date of observation (YYYY-MM-DDThh:mm:ss UTC)
OBSFREQ = 1500.390625 / [MHz] Centre frequency for observation
OBSBW = 200. / [MHz] Bandwidth for observation
OBSNCHAN= 256 / Number of frequency channels (original)
CHAN_DM = '* ' / [cm-3 pc] DM used for on-line dedispersion
PNT_ID = ' ' / Name or ID for pointing ctr (multibeam feeds)
SRC_NAME= 'J1125+7819' / Source or scan ID
COORD_MD= 'J2000 ' / Coordinate mode (J2000, GALACTIC, ECLIPTIC)
EQUINOX = 2000. / Equinox of coords (e.g. 2000.0)
RA = '11:25:00.192' / Right ascension (hh:mm:ss.ssss)
DEC = '+78:18:59.760' / Declination (-dd:mm:ss.sss)
BMAJ = 0. / [deg] Beam major axis length
BMIN = 0. / [deg] Beam minor axis length
BPA = 0. / [deg] Beam position angle
STT_CRD1= 'UNSET ' / Start coord 1 (hh:mm:ss.sss or ddd.ddd)
STT_CRD2= 'UNSET ' / Start coord 2 (-dd:mm:ss.sss or -dd.ddd)
TRK_MODE= 'UNSET ' / Track mode (TRACK, SCANGC, SCANLAT)
STP_CRD1= 'UNSET ' / Stop coord 1 (hh:mm:ss.sss or ddd.ddd)
STP_CRD2= 'UNSET ' / Stop coord 2 (-dd:mm:ss.sss or -dd.ddd)
SCANLEN = '* ' / [s] Requested scan length (E)
FD_MODE = 'FA ' / Feed track mode - FA, CPA, SPA, TPA
FA_REQ = 0. / [deg] Feed/Posn angle requested (E)
CAL_MODE= ' ' / Cal mode (OFF, SYNC, EXT1, EXT2)
CAL_FREQ= '* ' / [Hz] Cal modulation frequency (E)
CAL_DCYC= '* ' / Cal duty cycle (E)
CAL_PHS = '* ' / Cal phase (wrt start time) (E)
CAL_NPHS= '* ' / Number of states in cal pulse (I)
STT_IMJD= 58626 / Start MJD (UTC days) (J - long integer)
STT_SMJD= 41273 / [s] Start time (sec past UTC 00h) (J)
STT_OFFS= 0.00251008 / [s] Start time offset (D)
STT_LST = 0. / [s] Start LST (D)
In [4]: f[-1].header
Out[4]:
XTENSION= 'BINTABLE' / ***** Subintegration data *****
BITPIX = 8 / N/A
NAXIS = 2 / 2-dimensional binary table
NAXIS1 = 2108456 / width of table in bytes
NAXIS2 = 629 / Number of rows in table (NSUBINT)
PCOUNT = 0 / size of special data area
GCOUNT = 1 / one data group (required keyword)
TFIELDS = 10 / Number of fields per row
TTYPE1 = 'INDEXVAL' / Optionally used if INT_TYPE != TIME
TFORM1 = '1D ' / Double
TTYPE2 = 'TSUBINT ' / Length of subintegration
TFORM2 = '1D ' / Double
TTYPE3 = 'OFFS_SUB' / Offset from Start of subint centre
TFORM3 = '1D ' / Double
TTYPE4 = 'AUX_DM ' / additional DM (ionosphere, corona, etc.)
TFORM4 = '1D ' / Double
TTYPE5 = 'AUX_RM ' / additional RM (ionosphere, corona, etc.)
TFORM5 = '1D ' / Double
TTYPE6 = 'DAT_FREQ' / [MHz] Centre frequency for each channel
TFORM6 = '256D ' / NCHAN doubles
TTYPE7 = 'DAT_WTS ' / Weights for each channel
TFORM7 = '256E ' / NCHAN floats
TTYPE8 = 'DAT_OFFS' / Data offset for each channel
TFORM8 = '1024E ' / NCHAN*NPOL floats
TTYPE9 = 'DAT_SCL ' / Data scale factor (outval=dataval*scl + offs)
TFORM9 = '1024E ' / NCHAN*NPOL floats
EPOCHS = 'VALID ' / Epoch convention (VALID, MIDTIME, STT_MJD)
INT_TYPE= 'TIME ' / Time axis (TIME, BINPHSPERI, BINLNGASC, etc)
INT_UNIT= 'SEC ' / Unit of time axis (SEC, PHS (0-1), DEG)
SCALE = 'FluxDen ' / Intensity units (FluxDen/RefFlux/Jansky)
POL_TYPE= 'AABBCRCI' / Polarisation identifier (e.g., AABBCRCI, AA+BB)
NPOL = 4 / Nr of polarisations
TBIN = 4.096E-05 / [s] Time per bin or sample
NBIN = 1 / Nr of bins (PSR/CAL mode; else 1)
NBIN_PRD= '* ' / Nr of bins/pulse period (for gated data)
PHS_OFFS= '* ' / Phase offset of bin 0 for gated data
NBITS = 8 / Nr of bits/datum (SEARCH mode data, else 1)
ZERO_OFF= 127.5 / Zero offset for SEARCH-mode data
SIGNINT = 0 / 1 for signed ints in SEARCH-mode data, else 0
NSUBOFFS= 0 / Subint offset (Contiguous SEARCH-mode files)
NCHAN = 256 / Number of channels/sub-bands in this file
CHAN_BW = -0.78125 / [MHz] Channel/sub-band width
REFFREQ = 1500.390625 / [MHz] Reference frequency
DM = 11.217356 / [cm-3 pc] DM for post-detection dedisperion
RM = 0. / [rad m-2] RM for post-detection deFaraday
NCHNOFFS= '* ' / Channel/sub-band offset for split files
NSBLK = 2048 / Samples/row (SEARCH mode, else 1)
NSTOT = 1288192 / Total number of samples (SEARCH mode, else 1)
EXTNAME = 'SUBINT ' / name of this binary table extension
TUNIT2 = 's ' / Units of field
TUNIT3 = 's ' / Units of field
TUNIT4 = 'CM-3 '
TUNIT5 = 'RAD '
TUNIT6 = 'MHz ' / Units of field
EXTVER = 1 / auto assigned by template parser
TTYPE10 = 'DATA ' / label for field
TFORM10 = '2097152B' / format of field
TDIM10 = '(256,4,2048)'
In [5]: f.close()
As you can see, the header keywords seem to be there but they are not being read correctly.
