[Teem-users] teem 1.8.0 out

[Gordon K. <gk...@bw...>]

hello,

I've released Teem version 1.8.0.  The big changes here are in nrrd:
- ability to represent location and orientation
- ability for headers to refer to multiple data files (copied the 
MetaImage syntax)
- ability to represent sample thickness (as with MRI slice thickness)

A more detailed list of changes is included below, but this has been 
created manually, and may miss some of the smaller changes.

There are some things I wanted to get done for 1.8 but lost steam on:
make system: serious bug which means unu isn't recompiled after changes 
in unrrdu/*.c source
make system: create separate libraries for static and shared libraries
gage: reorganize so that it doesn't allocate a padded volume
gage: reorganize to enable bricked data
nrrd: allow nrrd header I/O to and from strings
nrrd: enable ImageMagick style flags for file format information, e.g. 
"PNM:blah.ppm"
bane: still broken, after version 1.7.0's massive reorganization of gage

With time these things are going to get done, but I've started to think 
that what I really need is a test suite, so that I can quickly detect 
big bugs (like the one James found: PNM and VTK readers never allocated 
data).  I'm very open to input from you users on what form this test 
suite should take.  Using Kitware's Dart system is one possibility, and 
I'd love help figuring that out.

Gordon

----------------------------------------
air:

API GONE, API NEW:

airExists_d() and airExists_f() are gone, airExists() is
new. airExists() takes a double argument and can be used in
place of either airExists_d() or airExists_f().
----------------------------------------
ten:

API CHANGE:

int
tenEvecRGB(Nrrd *nout, Nrrd *nin, int which, int aniso,
            double cthresh, double gamma,
            double bgGray, double isoGray)

has a new (fourth) argument, cthresh, which is a threshold on the
confidence value.  This is used to determine whether the RGB color
from the eigenvector is used, or whether the bgGray value is used.
The old code did a lerp based on confidence, but this was slightly
too cute.
----------------------------------------
nrrd:

API NEW:  (actually these have been around for awhile, but were added
just after 1.7.0 was released)

nrrdTile2D, nrrdUntile2D: for doing (and undoing) the "tiling" operation
that normally is implemented with a painful combination of axsplit,
permute, axmerge, axmerge, whereby the slices along one axis are 
shuffled
into the axes of two other existing axes
----------------------------------------
nrrd:

BEHAVIOR CHANGE:

nrrdDefCenter is now (compile-time) initialized to nrrdCenterCell,
instead of nrrdCenterNode.  So in this sense, the default centering
of nrrd has changed from node to cell centered.  This potentially
effects anything that used nrrdSpatialResample without explicitly
setting the centering on the input axes.
----------------------------------------
nrrd:

API NEW:

the NrrdResampleInfo struct now has a new field, "cheap", which, when
non-zero, causes nrrdSpatialResample to *not* scale up the filters
when doing downsampling (reducing number of of samples).  That is,
previously all downsampling was filtered downsampling: input samples
were "blurred" prior to subsampling.  This way, the downsampling is
just subsampling- for example picking off every other sample (the
same as what xv does).  You still have to set a kernel for this
behavior to be generated, but you can pick any interpolating kernel,
such as nrrdKernelBox.
----------------------------------------
unu:

BEHAVIOR CHANGE:

now sets nrrdStateKindNoop to AIR_FALSE (unless the user has used the
NRRD_STATE_KIND_NOOP environment variable to set it otherwise), since
it was annoying to have "unu shuffle" turn a nrrdKindList into a
nrrdKindUnknown.
----------------------------------------
biff:

API NEW:

Previously, the only way to get error message from biff was to use
either biffGet() or biffGetDone().  The possible drawback to this is
that the caller was responsible for freeing the resulting string,
which was actually allocated by biff, which is rather asymmetric.

Now, there is a new function which allows you to learn the required
length to allocate an error message buffer: biffGetStrlen(), which is
indexed by the usual key string.  Then, to put the error message in
the buffer: biffSetStr() or biffSetStrDone().

----------------------------------------
nrrd:

FILE FORMAT CHANGE:

allowing multiple separata datafiles.  See
http://teem.sourceforge.net/nrrd/format.html
for a description of how the file format does this (the syntax from
MetaIO was basically copied wholesale).  There are no external API
changes in association with this.

There is still a need for something like a "nrrdMake" function that
can go from a set of filenames to a nrrd, but now it will be simpler
to implement (since reading multiple files is done inside nrrd instead
of outside).  For example, teem/src/unrrdu/make.c became much simpler
in this respect.

----------------------------------------
nrrd:

API NEW:

There is a new field in the Nrrd struct, a string called
"sampleUnits".  The purpose of this is to store the units with which
the (scalar) values inside the nrrd array are measured.  For many
nrrdKinds, these units are legit even if the values in the nrrd are
logically not scalars.  3-vectors representing position offsets have
coefficients which have units of position, and this would be stored in
sampleUnits.  Less clear-cut is what to do with, for example,
nrrdKind3DMaskedSymMatrix, since in this case, the units of
diffusivity are clearly the right thing for most of the values, but
the mask value doesn't really obey this.  There are no restrictions
or semantics with respect to when the sampleUnits field may be set in
the Nrrd.

NOTE: when histogramming, the sampleUnits will be converted into
per-axis units, since that's the logical thing to do.  It "will be",
as in, it will be once I implement this.

----------------------------------------
nrrd:

API CHANGE, API RENAME:

some functions have been renamed and their arguments are different:
nrrdPeripheralInit, nrrdPeripheralCopy -> nrrdBasicInfoInit, 
nrrdBasicInfoCopy

The new functions take an "exclusion bitflag", exactly like
nrrdAxisInfoCopy.  The "basic info" bitflags are based on the values
in the *new* nrrdBasicInfo airEnum.  Using the bitflags allows much
finer-grained control over which values are initialized or copied at
each operation.  I've followed through with these changes in all the
Nrrd code that goes into NrrdIO (used in ITK), but have yet to finish
carefully propagating it all to the rest of nrrd.

Actually, for the time being, nrrdPeripheralInit and nrrdPeripheralCopy
are still there, but now they're declared in privateNrrd.h, not in
nrrd.h.

----------------------------------------
nrrd:

API NEW, FILE FORMAT NEW:

This is a big change: Nrrd can now represent general orientation
information, beyond the simple axis-aligned position stuff that is
handled by the per-axis min/max/spacing.

I've chosen to put all the orientation information into a set of
per-axis vectors, and a single per-array vector.  All the new fields
in Nrrd have to do with identifying the number of coefficients in
those vectors, or describing the basis vectors and units with which
those coefficients are measured.  The per-axis vector is called
"spaceDirection", it is now a member of the NrrdAxisInfo struct.  ***
Despite how the name might imply a unit-length vector, the
spaceDirection vector in fact represents both the sample spacing along
that axis (recovered as the vector norm), and the direction of the
axis ***.  The spaceDirection vector for axis i identifies the
difference in spatial location between samples whose indices differ by
1 along axis i, extending from the low index to the high index
positions.  For example, spaceDirection for axis 0 is the vector from
the location of sample[0,0,0] to sample[1,0,0].  So the spaceDirection
vectors define the edges of the voxel.

The per-array vector is called "spaceOrigin", it now a member of the
Nrrd struct itself.  I (with Dave's input) decided to go with the name
"origin" because it seemed more precise than the term used in the
DICOM format, "location", and because VTK currently uses "ORIGIN" for
the same point.  spaceOrigin is a new member of the Nrrd struct.  Like
DICOM, this identifies the spatial location of the center of the
"first sample transmitted"- the sample with the lowest memory
location.

The maximum dimension of the "space" around the array is set in
nrrdDefines.h to NRRD_SPACE_DIM_MAX (analagous to but distinct from
NRRD_DIM_MAX).

Here are the other new fields in the Nrrd struct.  The dimension of
the space that spaceDirection and spaceOrigin vectors live in (the
number of valid coefficients that they have) is called
"spaceDimension".  *** This is a separate variable from the intrinsic
array dimension ("dimension") because it logically is separate ***.
For example, a single time-step of an fMRI experiment is a 3D array
which lives on a slightly sheared slice in a 4D XYZT space, while a
DT-MRI scan is a 4D array which lives in a 3D XYZ space.  When known,
the canonical space containing the vectors is represented by "space",
which takes on integral values from the new "nrrdSpace" airEnum.
Finally, the units of measurement of the vector coefficients are
represented by an array of strings, "spaceUnits".  Here's how the
basic structs look now:

Associated with these new orientation variables are new fields in the
NRRD file format:

   space:
   space dimension:
   space units:
   space directions:
   space origin:

Using these fields requires using the new magic "NRRD0004".
nrrdLoad() and nrrdRead() can read these, and nrrdSave() and
nrrdWrite() and write them.  Here's how the new fields are used.

You can either use "space: " or "space dimension: " to describe the
space, but not both.  When you use "space: ", you're defining both
nrrd->space and nrrd->spaceDim.  Some samples of using this:

   space: RAS
   space: right-anterior-superior
   space: LAS
   space: left-anterior-superior
   space: LPS
   space: left-posterior-superior

Note that these all imply nrrd->spaceDim = 3.  The strings following
"space: " are parsed with the nrrdSpace airEnum, so different strings
can be used to represent the same setting.  If someone wants to use
nrrd to represent orientation in a space which is as yet unnamed, they
can identify the spaceDim itself:

   space dimension: 3

Either "space: " or "space dimension: " must be used before any of the
other new fields, because these identify the number of coefficients in
the spaceDirection and spaceOrigin vectors that will be parsed.  All
these vectors are represented in a form like "(x,y,z)".  For example:

   space origin: (-10,-10,-20)

Then, like all per-axis fields, you give as many spaceDirection
vectors as there are axes:

   space directions: (1,0,0) (0,1,0) (0,0,2)

Finally, the units for each coefficients of these vectors is given as
a list of quoted strings:

   space units: "mm" "mm" "mm"

Note that if you wanted to represent a single fMRI timestep, where
each Z slice lives it a slightly different time, you could use:

   dimension: 3
   space: RAST
   space units: "mm" "mm" "mm" "ms"
   space origin: (-10,-10,-20,0)
   space directions: (1,0,0,0) (0,1,0,0) (0,0,2,0.01)

Note that you still have one spaceDirection vector for each of the
three array axes, but the number of coefficients in those vectors is
determined by the "space" (right-anterior-superior-time).

    ----------- interaction with non-scalar types

When you use an array axis to represent non-scalar data, that axis
doesn't really have any spatial existence.  Just like the existing
per-axis min, max, and spacing, Nrrd uses NaN (not-a-number) to say
"this axis has no spaceDirection".  For example:

   dimension: 4
   kinds: nrrdKind3DMaskedSymMatrix nrrdKindDomain nrrdKindDomain 
nrrdKindDomain
   sizes: 7 256 256 90
   space dimension: 3
   space units: "mm" "mm" "mm"
   space origin: (-10,-10,-20)
   space directions: (nan,nan,nan) (1,0,0) (0,1,0) (0,0,2)

As a convenience, "none" can be used for an all-NaN vector:

   space directions: none (1,0,0) (0,1,0) (0,0,2)

The non-spatial vector is not included in the IJK-to-XYZ transform.
The AIR_EXISTS() macro is still used to determine if value is NaN
(just as its currently used for the per-axis min, max, and spacing).

   ------------ interaction with axis-aligned position

Nrrd already has a simple notion of sample "position", based on the
per-axis min, max, and spacing.  This is used, for example, to
represent a VTK STRUCTURED_POINTS grid which is axis-aligned, but has
different spacings along each of the three axes:

   dimension: 3
   sizes: 256 256 80
   spacings: 1 1 3
   units: "mm" "mm" "mm"
   mins: -10 -10 -20

In this case, the per-axis "min" plays the role of the VTK "ORIGIN".
When an axis represents a histogram, min and max are used together.
For a joint histogram of two scalars, for example, you might have:

   dimension: 2
   sizes: 500 500
   mins: -10 0.3
   maxs: 40 0.9

The sample position information implied by min, max, and spacing is
inherently one-dimensional, with the assumption that the array axis is
somehow aligned with the axis of the enclosing space.

To enforce some minimal semantic consistency between axis-aligned and
general orientation information, there must be mutual exclusion
between the use of min/max/spacing/units, and the spaceDirection
vector.  That is, for each NrrdAxisInfo *axis in the array, you can
either use axis->min, axis->max, axis->spacing, and axis->units (or
some subset of those), or you can use axis->spaceDirection.  But using
axis->spaceDirection disallows using the other fields.

This exclusion does not apply to axis->thickness, axis->center (for
representing cell versus node centering), axis->kind, and axis->label,
because these aspects of per-axis meta-information do not constrain
anything about sample positions.  I initially thought that
node-versus-cell centering did effect sample positioning, but it turns
out it doesn't.

For a crazy example, here's an array storing per-voxel histograms of
RGB color values.  The histogram axis index is 1.

   dimension: 5
   space: RAS
   sizes: 3 100 256 256 80
   mins: nan 0 nan nan nan
   maxs: nan 255 nan nan nan
   space units: "mm" "mm" "mm"
   space origin: (-10,-10,-20)
   space directions: none none (1,0,0) (0,1,0) (0,0,1)

There is no problem here because axis 0 uses neither min/max or
spaceDirection, axis 1 uses min/max but not spaceDirection, and the
other axes use spaceDirections but not min/max.

----------------------------------------
nrrd:

API RENAME:

char *unit --> char *units

Renaming from "unit" to "units" is just in recognition that idiomatic
usage of the word is overwhelmingly plural.

There was previously an effort to make a nrrdUnit* enum and select
values from that for the per-axis units, but it would never be diverse
enough to represent the units of all the scalar measurements that
might be put into a nrrd, all of which would become possible axis
units upon histogramming.

----------------------------------------
nrrd:

API CHANGE:

These nrrdKind values:

   nrrdKind2DSymMatrix,       /* 12: Mxx Mxy Myy */
   nrrdKind2DMaskedSymMatrix, /* 13: mask Mxx Mxy Myy */
   nrrdKind2DMatrix,          /* 14: Mxx Mxy Myx Myy */
   nrrdKind2DMaskedMatrix,    /* 15: mask Mxx Mxy Myx Myy */
   nrrdKind3DSymMatrix,       /* 16: Mxx Mxy Mxz Myy Myz Mzz */
   nrrdKind3DMaskedSymMatrix, /* 17: mask Mxx Mxy Mxz Myy Myz Mzz */
   nrrdKind3DMatrix,          /* 18: Mxx Mxy Mxz Myx Myy Myz Mzx Mzy Mzz 
*/
   nrrdKind3DMaskedMatrix,    /* 19: mask Mxx Mxy Mxz Myx Myy Myz Mzx 
Mzy Mzz */

have "Matrix" where they used to have "Tensor".  However, because I'm
a nice guy, the nrrdKind airEnum will still happily recognize
"3D-symmetric-tensor" and the like.  So any older NRRD files are
still perfectly usable, but code which uses these kinds will have to be
updated.

----------------------------------------
nrrd:

API NEW, FILE FORMAT NEW:

There is now a notion of sample "thickness" as well as spacing.  This
is a recognition of the fact that in acquired medical data (from CT
and MRI), the region of space measured to determine each sample value
is often not the same as the distance between the samples.  The
quantity is usually referred to as "slice thickness".

I was torn between making slice thickness a per-nrrd attribute and
making it a per-axis attribute.  All the people who care about slice
thickness will expect it to be per-dataset.  The DICOM format has a
tag for slice thickness, (0018,0050), so in that case its basically
per-slice.  On the other hand, MetaIO has an ElementSize field which
is per-axis.  I think the single most compelling reason to make slice
thickness be a per-axis attribute is that if it isn't, then its not
clear which axis has thickess, and which do not.  Also, if the axes
are permuted, per-axis thickness information should also be permuted.
Also, the units for spacing apply just as well to thickness, so if the
thickness was not per-axis then you'd need another field for thickness
units (DICOM dictates millimeters).

So, the changes are:

- a new field in the NrrdAxis struct: "thickness", which like spacing,
min, and max, is a double.

- new value "nrrdAxisInfoThickness" in nrrdAxisInfo* enum.

- new value "nrrdField_thicknesses" in nrrdField_* enum

- new per-axis field "thicknesses: " in the NRRD file format.  This is
one of the things that requires the use of the (new) NRRD0004 magic.

The only slightly complicated thing about thickness is how to update
it upon array operations.  For the most part, it is just propagated
through operations that don't do anything with the spatial lay-out of
the data (cropping, slicing, etc).  Unlike spacing, there is no
interaction between thickness and min, max, and centering- nrrd doesn't
ever *use* the thickness information for anything.

For operations like resampling, there is no meaningful or safe way to
update the thickness information, without getting extremely clever
with respect to what kernel is being used.  So, thickness is set to
NaN whenever the axis is resampled (but thickness is copied if the
resampling doesn't touch that axis).

----------------------------------------
nrrd:

FILE FORMAT CHANGE:

Now, you can use "none" as well as "???" to signify an unknown
nrrdCenter or nrrdKind