#388 When using Python wrapper, GetBuffer returns too many bytes

[Thomas Robitaille]

I am trying to read one of the example CETAUTOMATIX datasets from here:

http://www.osirix-viewer.com/datasets/

The minimal example below causes the buffer returned at the end to have 4737024 bytes, even though the image should only be 192 * 256 * 2 = 49152 bytes (the data is uint16)

import gdcm
import numpy as np

filename = 'CETAUTOMATIX/CETAUTOMATIX/Cardiovascular Heart-Cardiac Function/cine_retro_normal_hla/IM-0001-0007.dcm'

gdcm_image_reader = gdcm.ImageReader()
gdcm_image_reader.SetFileName(filename)
gdcm_image_reader.Read()
gdcm_image = gdcm_image_reader.GetImage()

gdcm_numpy_typemap = {
    gdcm.PixelFormat.INT8:     np.int8,
    gdcm.PixelFormat.UINT8:    np.uint8,
    gdcm.PixelFormat.UINT16:   np.uint16,
    gdcm.PixelFormat.INT16:    np.int16,
    gdcm.PixelFormat.UINT32:   np.uint32,
    gdcm.PixelFormat.INT32:    np.int32,
    gdcm.PixelFormat.FLOAT32:  np.float32,
    gdcm.PixelFormat.FLOAT64:  np.float64
}

gdcm_pixel_format = gdcm_image.GetPixelFormat().GetScalarType()
numpy_dtype = gdcm_numpy_typemap[gdcm_pixel_format]

pixel_bytearray = gdcm_image.GetBuffer()

print(len(pixel_bytearray))
# returns 4737024 instead of 49152

Furthermore, after the 49152-th byte, all the remaining bytes are zero. So this is a case of GetBuffer including unecessary padding at the end.

[Mathieu Malaterre]

I think you are using an ancient python version. Tested with 2.6.1 does not exhibit the issue you show.

[Thomas Robitaille]

I'm actually using Python 2.7.11!

$ python --version
Python 2.7.11 :: Continuum Analytics, Inc.

So I think this is a real issue.

[Mathieu Malaterre]

Meant to say python-gdcm version sorry. Try GDCM > 2.6.1

[Mathieu Malaterre]

Hum...using OpenJPEG 2.x I can get trigger an assert(). My guess is that there is an unitilized read happening in your case. Could you confirm that you are using a custom OpenJPEG 2.x library instead of the default ship-in openjpeg lib.

[Thomas Robitaille]

I just installed the latest version using

 * git clone --branch release git://git.code.sf.net/p/gdcm/gdcm
 * mkdir gdcmbin
 * cd gdcmbin
 * ccmake ../gdcm
 * make
 * make install

The latest commit is:

commit 923ec07fd011a30c5c80abae643e68ec011ec2aa
Author: Mathieu Malaterre <mathieu.malaterre@gmail.com>
Date:   Wed Jun 29 10:42:00 2016 +0200

and to be extra sure:

In [4]: import gdcm

In [5]: gdcm.GDCM_VERSION
Out[5]: '2.6.4'

I do think this is a real bug in the latest version.

[Mathieu Malaterre]

That's an impossible combination. You cannot use the default static lib with python binding. Please describe exactly the option(s) you used in cmake. Ideally post the actual

$ cmake -DBLA:BOOL=ON /path/to/gdcm/source

[Thomas Robitaille]

What is the easiest way to find out which OpenJPEG is being used? I do have library files called libgdcmopenjpeg.a for instance.

[Thomas Robitaille]

Just in case it helps, here is the install log:

https://gist.github.com/astrofrog/96b769397a3c0aff86f0c24efc805e2c

and I also see this in IPython:

In [10]: gdcm.OPENJPEG_MAJOR_VERSION
Out[10]: 1

[Thomas Robitaille]

Here's what I get for the cmake command you asked for:

https://gist.github.com/astrofrog/bcb21b1fdbe878d3eadd3d593a33c9cc

This was in a clean build directory. Does this help?

[Mathieu Malaterre]

No this is impossible. On one hand you have _gdcmswig.so building, but on the other hand you are specifing the python binding during cmake configuration.

[Thomas Robitaille]

And just to be clear, when I originally ran cmake, I ran the exact commands in INSTALL - i.e. cmake ../gdcm - without any flags.

[Mathieu Malaterre]

I think I understand what you did:

1. Install some kind of python-gdcm package (which provides _gdcmswig.so, use python -v to see which one is used)
2. rebuild gdcm, trying to override the C++ GDCM library (using static lib !) but omitting the python binding...which would explain you manage to use python binding and static lib without advanced gcc flags on Linux.

I am going to close this bug. If you want to re-open this bug report, delete all your system _gdcmswig.so files and then only then report a bug report. gdcm build system is somewhat flexible and should be able to run your python script within the gdcmbin/bin directory directly. If you do so, I'll only accept if you do provide the python -v output.

[Thomas Robitaille]

Ah sorry when I say I didn't add any flags, in the cmake config menu I did select to install the Python bindings - so I'm pretty sure the _gdcmswig.so file WAS installed at the same time.

[Mathieu Malaterre]

Hum...in this case you are not using x86_64, are you ? Please be a little more verbose on your setup:

• System (linux/mac/windows)
• Linux dist
• arch (x86/arm/powerpc)
• swig version

[Thomas Robitaille]

I'm using MacOS X 10.11, and my Python installation is 64-bit, installed from Anaconda.

So I wiped all GDCM files from my system and started fresh. This time, I did

ccmake -DGDCM_WRAP_PYTHON=ON ../gdcm
make

then I went inside the bin directory, and copied my debugging script and data there. I then ran the debug script and got the same answer again. The output from python -v is:

>>> import gdcm
# gdcm.pyc matches gdcm.py
import gdcm # precompiled from gdcm.pyc
# gdcmswig.pyc matches gdcmswig.py
import gdcmswig # precompiled from gdcmswig.pyc
import imp # builtin
dlopen("./_gdcmswig.so", 2);
import _gdcmswig # dynamically loaded from _gdcmswig.so
# /Users/tom/miniconda3/envs/dev27/lib/python2.7/weakref.pyc matches /Users/tom/miniconda3/envs/dev27/lib/python2.7/weakref.py
import weakref # precompiled from /Users/tom/miniconda3/envs/dev27/lib/python2.7/weakref.pyc

Let me know if there is any other information I can provide!

[Thomas Robitaille]

Ah and the SWIG version is:

$ swig -version

SWIG Version 3.0.8

Compiled with g++ [x86_64-apple-darwin11.4.2]

Configured options: +pcre

[Thomas Robitaille]

I think that the issue is somehow caused by a conflict with MacPorts. I haven't managed to narrow down exactly why, but by moving MacPorts out of the way, I managed to avoid this issue. It would still be interesting to understand why the issue occurred, but I think we're all short on time :)

[Mathieu Malaterre]

wow ! Thanks for the update, I feared it was a MacOSX + clang specific issue when compiling OpenJPEG code ...

As mentionned I do not have neither the time nor the hardware (mac) to reproduce the issue. For a start I would still reduce the bug to the C++ layer to check if we can remove python and swig from the depdencies:

gdcm::ImageReader gdcm_image_reader = gdcm::ImageReader();
gdcm_image_reader.SetFileName(filename);
gdcm_image_reader.Read();
gdcm::Image & gdcm_image = gdcm_image_reader.GetImage();

std::cout << gdcm_image.GetBufferLength() << std::endl;

See also: http://gdcm.sourceforge.net/html/PatchFile_8cxx-example.html#a39

[Thomas Robitaille]

This might be a silly question, but I'm trying to compile the example you provided (for now, just the PatchFile one) and am trying to compile with:

$ clang++ test.cpp -I/usr/local/include/gdcm-2.6/ -L/usr/local/lib -lgdcmCommon -lgdcmDICT -lgdcmDSED -lgdcmIOD -lgdcmMEXD -lgdcmMSFF -lgdcmcharls -lgdcmexpat -lgdcmjpeg12 -lgdcmjpeg16 -lgdcmjpeg8 -lgdcmopenjpeg -lgdcmuuid -lgdcmzlib -lsocketxx

but I get the following error:

Undefined symbols for architecture x86_64:
  "_CFBundleCopyExecutableURL", referenced from:
      gdcm::System::GetCurrentProcessFileName() in libgdcmCommon.a(gdcmSystem.cxx.o)
  "_CFBundleCopyResourcesDirectoryURL", referenced from:
      gdcm::System::GetCurrentResourcesDirectory() in libgdcmCommon.a(gdcmSystem.cxx.o)
  "_CFBundleGetMainBundle", referenced from:
      gdcm::System::GetCurrentProcessFileName() in libgdcmCommon.a(gdcmSystem.cxx.o)
      gdcm::System::GetCurrentResourcesDirectory() in libgdcmCommon.a(gdcmSystem.cxx.o)
  "_CFRelease", referenced from:
      gdcm::System::GetCurrentProcessFileName() in libgdcmCommon.a(gdcmSystem.cxx.o)
      gdcm::System::GetCurrentResourcesDirectory() in libgdcmCommon.a(gdcmSystem.cxx.o)
  "_CFURLGetFileSystemRepresentation", referenced from:
      gdcm::System::GetCurrentProcessFileName() in libgdcmCommon.a(gdcmSystem.cxx.o)
      gdcm::System::GetCurrentResourcesDirectory() in libgdcmCommon.a(gdcmSystem.cxx.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Do you know what this is due to?

[Mathieu Malaterre]

You are missing a MacOSX specific library. Either use cmake:

$ cat CMakeLists.txt
find_package(GDCM REQUIRED)
add_executable(test test.cpp)
target_link_librarires(test gdcmMSFF)
$ cmake . && make

Or, simply do a touch on any *.cxx file within Source/Common and then type:

make gdcmCommon VERBOSE=1

You should see what is being added by cmake (link line is at the end).

From my linux system all I can see is that it is referenced by:

target_link_libraries(gdcmCommon ${COREFOUNDATION_LIBRARY})

but I do not know the value of this thingy...