#283 PDL::IO::FITS::wfits corrupting FITS image

closed-fixed
nobody
other (94)
5
2014-08-24
2011-08-19
Tim
No

When running the code example given below under the pdl command line, the output is fine. When running the exact same code example in a script using perl (5.10), the output is mangled: the FITS headers are fine, but the data is corrupted. Always reproducible on other FITS images. Also reproducible if the output FITS file is the input FITS file.

The example FITS image is available here: http://sourceforge.net/projects/pdl/files/PDL/

<code>
use strict;
use warnings;
use PDL::IO::FITS;
my $pdl = PDL::IO::FITS::rfits('m51_raw.fits');
PDL::IO::FITS::wfits($pdl,'TEMP_m51_raw.fits');
</code>

Output of perldl -V:

perlDL shell v1.354
PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
'COPYING' in the PDL distribution. This is free software and you
are welcome to redistribute it under certain conditions, see
the same file for details.

Summary of my PDL configuration

VERSION: PDL v2.4.9 (supports bad values)

$%PDL::Config = {
'BADVAL_PER_PDL' => '0',
'WITH_PROJ' => '0',
'FFTW_TYPE' => 'double',
'FFTW_LIBS' => [
'/lib',
'/usr/lib',
'/usr/local/lib'
],
'WITH_FFTW' => '0',
'GSL_LIBS' => undef,
'WITH_IO_BROWSER' => '0',
'PROJ_INC' => undef,
'WHERE_PLPLOT_INCLUDE' => undef,
'HTML_DOCS' => '1',
'SKIP_KNOWN_PROBLEMS' => '0',
'WHERE_PLPLOT_LIBS' => undef,
'WITH_3D' => '0',
'FFTW_INC' => [
'/usr/include/',
'/usr/local/include'
],
'WITH_POSIX_THREADS' => '1',
'POGL_VERSION' => '0.63',
'HIDE_TRYLINK' => '1',
'WITH_HDF' => '0',
'HDF_INC' => undef,
'POGL_WINDOW_TYPE' => 'glut',
'WITH_BADVAL' => '1',
'WITH_GD' => '0',
'FITS_LEGACY' => '1',
'WITH_SLATEC' => '1',
'BADVAL_USENAN' => '0',
'WITH_DEVEL_REPL' => '1',
'TEMPDIR' => '/tmp',
'PROJ_LIBS' => undef,
'USE_POGL' => '0',
'GD_LIBS' => undef,
'GSL_INC' => undef,
'GD_INC' => undef,
'WITH_GSL' => '0',
'OPTIMIZE' => undef,
'HDF_LIBS' => undef,
'MALLOCDBG' => {},
'WITH_MINUIT' => '1',
'WITH_PLPLOT' => '0',
'MINUIT_LIB' => undef
};
Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

Platform:
osname=linux, osvers=2.6.24-29-server, archname=x86_64-linux-gnu-thread-multi
uname='linux crested 2.6.24-29-server #1 smp wed mar 16 19:04:28 utc 2011 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.1 -Dsitearch=/usr/local/lib/perl/5.10.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Dplibpth=/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu -Duseshrplib -Dlibperl=libperl.so.5.10.1 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.5.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu /lib /usr/lib /lib64 /usr/lib64
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=, so=so, useshrplib=true, libperl=libperl.so.5.10.1
gnulibc_version='2.13'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Output of perl -V:

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

Platform:
osname=linux, osvers=2.6.24-29-server, archname=x86_64-linux-gnu-thread-multi
uname='linux crested 2.6.24-29-server #1 smp wed mar 16 19:04:28 utc 2011 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.1 -Dsitearch=/usr/local/lib/perl/5.10.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Dplibpth=/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu -Duseshrplib -Dlibperl=libperl.so.5.10.1 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.5.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu /lib /usr/lib /lib64 /usr/lib64
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=, so=so, useshrplib=true, libperl=libperl.so.5.10.1
gnulibc_version='2.13'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API
Locally applied patches:
DEBPKG:debian/arm_thread_stress_timeout - http://bugs.debian.org/501970 Raise the timeout of ext/threads/shared/t/stress.t to accommodate slower build hosts
DEBPKG:debian/cpan_config_path - Set location of CPAN::Config to /etc/perl as /usr may not be writable.
DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
DEBPKG:debian/db_file_ver - http://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
DEBPKG:debian/enc2xs_inc - http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
DEBPKG:debian/errno_ver - http://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
DEBPKG:debian/extutils_hacks - Various debian-specific ExtUtils changes
DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
DEBPKG:debian/m68k_thread_stress - http://bugs.debian.org/495826 Disable some threads tests on m68k for now due to missing TLS.
DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
DEBPKG:debian/module_build_man_extensions - http://bugs.debian.org/479460 Adjust Module::Build manual page extensions for the Debian Perl policy
DEBPKG:debian/perl_synopsis - http://bugs.debian.org/278323 Rearrange perl.pod
DEBPKG:debian/prune_libs - http://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
DEBPKG:debian/use_gdbm - Explicitly link against -lgdbm_compat in ODBM_File/NDBM_File.
DEBPKG:fixes/assorted_docs - http://bugs.debian.org/443733 [384f06a] Math::BigInt::CalcEmu documentation grammar fix
DEBPKG:fixes/net_smtp_docs - http://bugs.debian.org/100195 [rt.cpan.org #36038] Document the Net::SMTP 'Port' option
DEBPKG:fixes/processPL - http://bugs.debian.org/357264 [rt.cpan.org #17224] Always use PERLRUNINST when building perl modules.
DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
DEBPKG:fixes/pod2man-index-backslash - http://bugs.debian.org/521256 Escape backslashes in .IX entries
DEBPKG:debian/disable-zlib-bundling - Disable zlib bundling in Compress::Raw::Zlib
DEBPKG:fixes/kfreebsd_cppsymbols - http://bugs.debian.org/533098 [3b910a0] Add gcc predefined macros to $Config{cppsymbols} on GNU/kFreeBSD.
DEBPKG:debian/cpanplus_definstalldirs - http://bugs.debian.org/533707 Configure CPANPLUS to use the site directories by default.
DEBPKG:debian/cpanplus_config_path - Save local versions of CPANPLUS::Config::System into /etc/perl.
DEBPKG:fixes/kfreebsd-filecopy-pipes - http://bugs.debian.org/537555 [16f708c] Fix File::Copy::copy with pipes on GNU/kFreeBSD
DEBPKG:fixes/anon-tmpfile-dir - http://bugs.debian.org/528544 [perl #66452] Honor TMPDIR when open()ing an anonymous temporary file
DEBPKG:fixes/abstract-sockets - http://bugs.debian.org/329291 [89904c0] Add support for Abstract namespace sockets.
DEBPKG:fixes/hurd_cppsymbols - http://bugs.debian.org/544307 [eeb92b7] Add gcc predefined macros to $Config{cppsymbols} on GNU/Hurd.
DEBPKG:fixes/autodie-flock - http://bugs.debian.org/543731 Allow for flock returning EAGAIN instead of EWOULDBLOCK on linux/parisc
DEBPKG:fixes/archive-tar-instance-error - http://bugs.debian.org/539355 [rt.cpan.org #48879] Separate Archive::Tar instance error strings from each other
DEBPKG:fixes/positive-gpos - http://bugs.debian.org/545234 [perl #69056] [c584a96] Fix \G crash on first match
DEBPKG:debian/devel-ppport-ia64-optim - http://bugs.debian.org/548943 Work around an ICE on ia64
DEBPKG:fixes/trie-logic-match - http://bugs.debian.org/552291 [perl #69973] [0abd0d7] Fix a DoS in Unicode processing [CVE-2009-3626]
DEBPKG:fixes/hppa-thread-eagain - http://bugs.debian.org/554218 make the threads-shared test suite more robust, fixing failures on hppa
DEBPKG:fixes/crash-on-undefined-destroy - http://bugs.debian.org/564074 [perl #71952] [1f15e67] Fix a NULL pointer dereference when looking for a DESTROY method
DEBPKG:fixes/tainted-errno - http://bugs.debian.org/574129 [perl #61976] [be1cf43] fix an errno stringification bug in taint mode
DEBPKG:fixes/safe-upgrade - http://bugs.debian.org/582978 Upgrade Safe.pm to 2.25, fixing CVE-2010-1974
DEBPKG:fixes/tell-crash - http://bugs.debian.org/578577 [f4817f3] Fix a tell() crash on bad arguments.
DEBPKG:fixes/format-write-crash - http://bugs.debian.org/579537 [perl #22977] [421f30e] Fix a crash in format/write
DEBPKG:fixes/arm-alignment - http://bugs.debian.org/289884 [f1c7503] Prevent gcc from optimizing the alignment test away on armel
DEBPKG:fixes/fcgi-test - Fix a failure in CGI/t/fast.t when FCGI is installed
DEBPKG:fixes/hurd-ccflags - http://bugs.debian.org/587901 Make hints/gnu.sh append to $ccflags rather than overriding them
DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
DEBPKG:fixes/lc-numeric-docs - http://bugs.debian.org/379329 [perl #78452] [903eb63] LC_NUMERIC documentation fixes
DEBPKG:fixes/lc-numeric-sprintf - http://bugs.debian.org/601549 [perl #78632] [b3fd614] Fix sprintf not to ignore LC_NUMERIC with constants
DEBPKG:fixes/concat-stack-corruption - http://bugs.debian.org/596105 [perl #78674] [e3393f5] Fix stack pointer corruption in pp_concat() with 'use encoding'
DEBPKG:fixes/cgi-multiline-header - http://bugs.debian.org/606995 [CVE-2010-2761 CVE-2010-4410 CVE-2010-4411] CGI.pm MIME boundary and multiline header vulnerabilities
DEBPKG:fixes/h2ph-gcc-4.5 - http://bugs.debian.org/599933 [8d66b3f] Fix h2ph and test
DEBPKG:fixes/threads-tmps-crash - [perl #70411] [24855df] Conditionally compile tmps stack cleanup code
DEBPKG:patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.10.1-17ubuntu1 in patchlevel.h
Built under linux
Compiled at Apr 26 2011 15:56:28
@INC:
/etc/perl
/usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.10
/usr/share/perl/5.10
/usr/local/lib/site_perl
.

Discussion

  • Chris Marshall
    Chris Marshall
    2011-08-19

    You'll need to be more specific in the bug
    report with respect to what you mean by
    "corrupted". I read in the original and the
    TEMP versions of the file and the resulting
    data was identical to the bit.

    Generally, a bug report needs a small,
    easily reproduced piece of code together
    with what was expected and what you
    actually got.

    A lot of times it makes sense to query
    the perldl mailing list before hand to
    better formulate the report.

     
  • Chris Marshall
    Chris Marshall
    2011-08-19

    • milestone: --> 100444
     
  • Chris Marshall
    Chris Marshall
    2011-08-19

    To clarify, the result of reading the two FITS files with rfits()
    resulted in two identical pdls. I don't know enough about
    the header stuff to determine if there is some hidden difference.

    pdl> lf '*.fits'
    TEMP_m51_raw.fits m51.fits m51_raw.fits

    pdl> $p1 = rfits('m51_raw.fits')
    Reading IMAGE data...
    BITPIX = 32 size = 147456 pixels
    Reading 589824 bytes
    BSCALE = && BZERO =

    pdl> $pnew = rfits('TEMP_m51_raw.fits')
    Reading IMAGE data...
    BITPIX = 32 size = 147456 pixels
    Reading 589824 bytes
    BSCALE = && BZERO =

    pdl> p $p1->stats
    338.535430908203 108.524703979492 325 40 761 9.19002628326416 108.524337768555

    pdl> p $pnew->stats
    338.535430908203 108.524703979492 325 40 761 9.19002628326416 108.524337768555

    pdl> p sum($p1 - $pnew)
    0

     
  • Tim
    Tim
    2011-08-19

    marshallch, thank you much for taking a look at this. Sorry, I did not provide enough detail about the output corruption. I was using ds9 (SAOImage) to view the two FITS files. The TEMP file was consistently giving me a garbage image. I added plotting of the images, so it is easier to visualize the output. I performed the stats measures against the original and TEMP versions of the files as you had done, but I get wildly different answers (see below) depending upon the method of execution. Unfortunately, the only code that is needed to replicate the problem is what I gave in the original report. I merely read in the FITS file, then wrote it out with a different file name. When you performed this test, were you using the command line version of pdl (perldl, pdl2) or were you running it from a perl script? When I run the code example from the pdl command line (using pdl2), I get perfect results just as you did. However, when I run it from a perl script, I get the garbage FITS image.

    I have manually re-installed PDL v2.4.9 using CPAN, but that did not fix the issue. Given that you are not having the same issues, it would seem that the problem most likely lies in my kit somewhere.

    Thanks again for your help.

    <code>
    use PDL::Graphics::PGPLOT;
    my $pdl = rfits('m51_raw.fits');
    $pdl->wfits('TEMP_m51_raw.fits');
    my $pdl2 = rfits('TEMP_m51_raw.fits');
    print $pdl->stats;
    print $pdl2->stats;
    imag $pdl, 0,300;
    <STDIN>;
    imag $pdl2, 0,300;
    <STDIN>;
    </code>

    <output when executed in perl script>
    original: 67.9894256591797 154.611404418945 43.6842994689941 -55.3413391113281 30763.66796875 7.33825778961182 154.611099243164

    temp: inf inf 2.28159415961367e-41 -3.39300211518291e+38 3.39304065176116e+38 inf inf
    </output>

    <output when executed in pdl2>
    original: 67.9894256591797 154.611404418945 43.6842994689941 -55.3413391113281 30763.66796875 7.33825778961182 154.611099243164

    temp: 67.9894256591797 154.611404418945 43.6842994689941 -55.3413391113281 30763.66796875 7.33825778961182 154.611099243164
    </output>

     
  • Chris Marshall
    Chris Marshall
    2011-08-19

    I created the file by your program and then checked the
    data from the pdl2 shell. I would try a find on your system
    to find any/all versions of PDL. In my experience, version
    skew between multiple PDL installs can cause problems.

    Also, I think you can get PDL from the debian packages
    which may give you a way to try a clean install. Good luck.
    Sorry I couldn't be of more help.

    --Chris

     
  • Tim
    Tim
    2011-08-21

    Chris,

    I think I may have discovered the issue. Try executing the following in a perl script.

    <code>
    use strict;
    use warnings;
    use Cwd qw(cwd);
    use Carp qw(croak);
    use PDL;
    use PDL::IO::FITS qw(rfits wfits);
    use PDL::Graphics::PGPLOT;
    $\ = "\n";
    $, = ' ';
    $" = ' ';
    $PDL::verbose = 1;
    my $pdl = rfits('m51_raw.fits');
    $pdl->wfits('TEMP_m51_raw.fits');
    my $pdl2 = rfits('TEMP_m51_raw.fits');
    print 'OLD:', $pdl->stats;
    print "TEMP:", $pdl2->stats;
    imag $pdl, 0,300; <STDIN>;
    imag $pdl2, 0,300; <STDIN>;
    </code>

    If the second image is garbage and the stats don't match, turn off the end-of-line terminator (i.e., $\ = undef). When I did this, the errors ceased to occur. I edited PDL/IO/FITS.pm and added "local $\ = undef;" to line 1816 (just after the parameter check). This resolved my issue completely. When the $fh->print routines were getting called, $\ was adding an extra character that wasn't accounted for in the buffer variables (e.g., $nbytes). This was causing the binary data to be written in locations that the FITS tables were not pointing to. Please let me know if you can replicate this issue now. I will be more than willing to submit a formal patch if this resolves the issue.

    Many thanks for your help.

    - Tim

     
  • Chris Marshall
    Chris Marshall
    2011-08-21

    The CHM/PDL-2.4.9_007.tar.gz CPAN developers release has this
    proposed fix in it. Please verify.

     
  • Chris Marshall
    Chris Marshall
    2011-08-21

    • milestone: 100444 -->
    • status: open --> open-fixed
     
  • Chris Marshall
    Chris Marshall
    2011-09-04

    Fix verified by kxtor. Thanks.

     
  • Chris Marshall
    Chris Marshall
    2011-09-04

    • status: open-fixed --> closed-fixed