[dbfpy-devel] [ dbfpy-Bugs-3560359 ] dbfpy does not handle NaN or Infinity value in dbf files

[SourceForge.net <no...@so...>]

Bugs item #3560359, was opened at 2012-08-21 07:03
Message generated for change (Comment added) made by a1s
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=746843&aid=3560359&group_id=140566

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: Reuter (gisxperts)
Assigned to: Nobody/Anonymous (nobody)
Summary: dbfpy  does not handle NaN or Infinity value in dbf files

Initial Comment:
db = dbf.Dbf(infile, new=False)
[Dbg]>>> rec=db[0]
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python26\ArcGIS10.0\lib\site-packages\dbfpy\dbf.py", line 242, in __getitem__
    return self.RecordClass.fromStream(self, self._fixIndex(index))
  File "C:\Python26\ArcGIS10.0\lib\site-packages\dbfpy\record.py", line 121, in fromStream
    return cls.fromString(dbf, cls.rawFromStream(dbf, index), index)
  File "C:\Python26\ArcGIS10.0\lib\site-packages\dbfpy\record.py", line 140, in fromString
    [_fd.decodeFromRecord(string) for _fd in dbf.header.fields])
  File "C:\Python26\ArcGIS10.0\lib\site-packages\dbfpy\fields.py", line 173, in decodeFromRecord
    return self.decodeValue(self.rawFromRecord(record))
  File "C:\Python26\ArcGIS10.0\lib\site-packages\dbfpy\fields.py", line 241, in decodeValue
    return float(value)
ValueError: invalid literal for float(): -1.#IND00000000000

----------------------------------------------------------------------

>Comment By: alexander smishlajev (a1s)
Date: 2012-08-29 01:02

Message:
So, these files are not produced by xBase software - dBase, FoxPro, Clipper
or some such.

Futhermore, formatting is not what one would expect (see f.e.
http://en.wikipedia.org/wiki/Printf_format_string): 'nan' for IEEE 754 NaN
and 'inf' for infinity.

GeoTIFF stores numbers in binary format, so the formatting behaviour is set
up in the extraction software.  Perhaps you could talk to authors of that
software.

As for dbfpy, I am resistant to amend base functionality for non-standard
uses.  Instead, I recommend subclassing DbfNumericFieldDef in your
application.  (Call registerField() to install customized field class.)

Closing this issue for now; please reopen if you see reasons to reconsider.

----------------------------------------------------------------------

Comment By: Reuter (gisxperts)
Date: 2012-08-28 15:52

Message:
python libraries linking GDAL extracted values from TIF files was able to
write these values into dbf. no real documentation, but observed values. 


----------------------------------------------------------------------

Comment By: alexander smishlajev (a1s)
Date: 2012-08-21 09:14

Message:
Thank you for reporting this.

Could you point to documentation describing storage format for these
values?

----------------------------------------------------------------------

Comment By: Reuter (gisxperts)
Date: 2012-08-21 07:59

Message:
Patch for the problem extend the lines in field.py line 228ff. however a
cleaner solution would be better. 

     def decodeValue(self, value):
        """Return a number decoded from ``value``.

        If decimals is zero, value will be decoded as an integer;
        or as a float otherwise.

        Return:
            Return value is a int (long) or float instance.

        """
        value = value.strip(" \0")
        if "." in value:
            # a float (has decimal separator)
            if '#IND' in value:
                return(-9999.9999)
            elif '#QNAN' in value:
                return(-9999.9999)
            else:
                return float(value)
        elif '*' in value:
                return(-9999.9999)
        elif value:
            # must be an integer
            return int(value)
        else:
            return 0

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=746843&aid=3560359&group_id=140566