teem-users — Discussions of usage, coding, and proposed changes

Re: [Teem-users] IO problems on Windows XP

[Gordon L. K. <gl...@uc...>]

The problem, it turns out, had nothing to do with Windows XP, or issues with fseek on files opened as binary.

The problem was that I was naively defining multiple tests that ran in parallel, but wrote to the same file, and they unsurprisingly clobbered each other's results.  Thanks to David Cole of Kitware for explaining my mis-use of CTest.

The dashboard is green
http://my.cdash.org/index.php?project=Teem
though there are far fewer tests than there ought to be.  

So again: if your software project has benefitted from using Teem, one way that you can repay the cost of time and effort that has gone into creating Teem, is to help by writing tests that exercise the functionality your project uses.  Email me for SVN write access.

Gordon

On Oct 13, 2012, at 10:13 PM, Gordon L. Kindlmann wrote:

> Hello,
> 
> I'm trying to track down a very odd problem with Nrrd IO in Windows XP.   Your help is appreciated.
> 
> Some tests are failing on Windows (and only Windows):
> 
> http://my.cdash.org/index.php?project=Teem&date=20121013
> 
> The tests that are failing are related to ensuring that the "byte skip: N" and "byte skip: -M" fields of the NRRD file header work as expected (skipping positive N bytes from beginning of file, or M-1 bytes from end of the file).  These are tested by making writing a raw data segment with padding to disk, using a sequence of fputc (for positive padding), fwrite, and fputc again (for negative padding).  Then a  NRRD header is written to refer to the raw data with the appropriate skipping, which is then nrrdLoad()ed and passed to nrrdCRC32() to see if the data was read correctly.
> 
> For some reason, on Windows XP at least, these doesn't work.  Either not all the data is written (so the nrrdLoad fails outright), or something else fails (so the CRC check fails). What on earth is going on?
> 
> You can see the source in teem/Testing/nrrd/tskip.c.  Based on some things I saw online, I added a (non-standard) "c" flag to the fopen options (so its fopen("ts.raw", "wbc"), which I thought would make fflush() really do its job, but no luck.
> 
> Thanks for any tips,
> Gordon
> 
> 
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users
> 

[Teem-users] ITK NrrdIO and Teem NrrdIO have been synched

[Gordon L. K. <gl...@uc...>]

For those that were nervous that bits of Teem code strewn across the universe were gradually being pulled further and further apart by the incessant forces of impatience and version skew, rest assured.  There is at least one place where those forces have been thwarted:

http://review.source.kitware.com/#/c/8057/

The patch has been merged.

Gordon

Re: [Teem-users] IO problems on Windows XP

[David C. <dav...@ki...>]

Is this an endian problem? All the test failures I look at have a
checksum mismatch.


On Sat, Oct 13, 2012 at 11:13 PM, Gordon L. Kindlmann <gl...@uc...> wrote:
> Hello,
>
> I'm trying to track down a very odd problem with Nrrd IO in Windows XP.   Your help is appreciated.
>
> Some tests are failing on Windows (and only Windows):
>
> http://my.cdash.org/index.php?project=Teem&date=20121013
>
> The tests that are failing are related to ensuring that the "byte skip: N" and "byte skip: -M" fields of the NRRD file header work as expected (skipping positive N bytes from beginning of file, or M-1 bytes from end of the file).  These are tested by making writing a raw data segment with padding to disk, using a sequence of fputc (for positive padding), fwrite, and fputc again (for negative padding).  Then a  NRRD header is written to refer to the raw data with the appropriate skipping, which is then nrrdLoad()ed and passed to nrrdCRC32() to see if the data was read correctly.
>
> For some reason, on Windows XP at least, these doesn't work.  Either not all the data is written (so the nrrdLoad fails outright), or something else fails (so the CRC check fails). What on earth is going on?
>
> You can see the source in teem/Testing/nrrd/tskip.c.  Based on some things I saw online, I added a (non-standard) "c" flag to the fopen options (so its fopen("ts.raw", "wbc"), which I thought would make fflush() really do its job, but no luck.
>
> Thanks for any tips,
> Gordon
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users

Re: [Teem-users] updates and helping with tests

[David C. <dav...@ki...>]

On Mon, Oct 15, 2012 at 10:16 AM, Sean McBride <se...@ro...> wrote:
> On Fri, 12 Oct 2012 21:33:55 -0500, Gordon L. Kindlmann said:
>
>>Having more different mac builds contributing there would be tremendous :)
>
> I don't actually use teem, I only know of it through ITK, but I could probably contribute a few dashboards if you can help me do it quickly. :)  I thought I'd copy an existing Mac one, like this:
>
> <http://my.cdash.org/viewNotes.php?buildid=398990>
>
> but there's a huge amount of stuff there...
>

Don't just copy the scripts. Follow the directions on how those
scripts are used effectively:

  http://cmake.blogspot.com/2012/09/one-way-to-run-dashboard-on-windows.html

(I know it's "Windows" focused, but that's just for example's sake,
and because most open source devs are still allergic to Windows.)

The easy dashboard scripts work on Mac and Linux, too.

Teem is a great project because the build/test cycle is ridiculously fast.


Cheers,
David


>>BTW, gerrit patch above includes the removal of signed integer overflow
>>in the sanity checks.
>
> Great!
>
> Cheers,
>
> --
> ____________________________________________________________
> Sean McBride, B. Eng                 se...@ro...
> Rogue Research                        www.rogue-research.com
> Mac Software Developer              Montréal, Québec, Canada
>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users

Re: [Teem-users] updates and helping with tests

[Sean M. <se...@ro...>]

On Fri, 12 Oct 2012 21:33:55 -0500, Gordon L. Kindlmann said:

>Having more different mac builds contributing there would be tremendous :)

I don't actually use teem, I only know of it through ITK, but I could probably contribute a few dashboards if you can help me do it quickly. :)  I thought I'd copy an existing Mac one, like this:

<http://my.cdash.org/viewNotes.php?buildid=398990>

but there's a huge amount of stuff there...

>BTW, gerrit patch above includes the removal of signed integer overflow
>in the sanity checks.

Great!

Cheers,

-- 
____________________________________________________________
Sean McBride, B. Eng                 se...@ro...
Rogue Research                        www.rogue-research.com 
Mac Software Developer              Montréal, Québec, Canada

Re: [Teem-users] z project and merge nrrds

[Gordon L. K. <gl...@uc...>]

And I meant to mention on this:

For using different kinds of projections for volume data inspection, besides using different color channels for different signals, some other things that might prove useful:

* using percentile-based quantization ranges in "unu quantize".  Instead of having to specify explicit ranges of values the min and max (to map to 0 and 255, respectively), you can use "-min 0.5% -max 0.5%" and that will say "the lowest and highest 0.5% of the values should be mapped to 0 and 255.  This tends to make things much more reliable in the face of noise or outliers.  

* using some histogram equalization in "unu heq".  I tend to use something like "unu heq -b 3000 -s 2 -a 0.5", which computes the equalization with a 3000 bin histogram, with 2 "smart bins" (see "unu heq" usage info), and only apply the equalization half-way.  "unu gamma" may also be a useful way of remapping values in a more controlled values.

On Aug 27, 2012, at 11:23 PM, Dr Gregory Jefferis wrote:

> Dear Thomas,
> 
> On 16 Aug 2012, at 08:42, Thomas Schultz wrote:
> 
>> if red.png green.png and blue.png are monochrome pngs (with a single
>> color channel), then
>> 
>> unu join -a 0 -incr -i red.png green.png blue.png -o rgb.png
>> 
>> should produce an RGB png.
> 
> 
> Sorry I forgot to reply at the time. This worked perfectly! Many thanks,
> 
> Greg.
> 
> --
> Gregory Jefferis, PhD                  jef...@mr...
> Division of Neurobiology               LMB Lab:   +44 (0)1223 252943
> MRC Laboratory of Molecular Biology,   LMB Office:+44 (0)1223 402333 [NEW NUMBER]
> Hills Road,                            LMB Fax:   +44 (0)1223 402310
> Cambridge, CB2 0QH, UK.                    
> 
> http://www2.mrc-lmb.cam.ac.uk/group-leaders/h-to-m/g-jefferis
> http://www.neuroscience.cam.ac.uk/directory/profile.php?gsxej2
> http://flybrain.stanford.edu
> 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users
> 

[Teem-users] IO problems on Windows XP

[Gordon L. K. <gl...@uc...>]

Hello,

I'm trying to track down a very odd problem with Nrrd IO in Windows XP.   Your help is appreciated.

Some tests are failing on Windows (and only Windows):

http://my.cdash.org/index.php?project=Teem&date=20121013

The tests that are failing are related to ensuring that the "byte skip: N" and "byte skip: -M" fields of the NRRD file header work as expected (skipping positive N bytes from beginning of file, or M-1 bytes from end of the file).  These are tested by making writing a raw data segment with padding to disk, using a sequence of fputc (for positive padding), fwrite, and fputc again (for negative padding).  Then a  NRRD header is written to refer to the raw data with the appropriate skipping, which is then nrrdLoad()ed and passed to nrrdCRC32() to see if the data was read correctly.

For some reason, on Windows XP at least, these doesn't work.  Either not all the data is written (so the nrrdLoad fails outright), or something else fails (so the CRC check fails). What on earth is going on?

You can see the source in teem/Testing/nrrd/tskip.c.  Based on some things I saw online, I added a (non-standard) "c" flag to the fopen options (so its fopen("ts.raw", "wbc"), which I thought would make fflush() really do its job, but no luck.

Thanks for any tips,
Gordon

Re: [Teem-users] updates and helping with tests

[Steve P. <pi...@ib...>]

Hi Gordon - I'll pass that question over to C-F and Demian - I agree
that there should be some very solid quantitative tests in place
before we switch versions of teem.

Best,
Steve

On Sat, Oct 13, 2012 at 11:09 AM, Gordon L. Kindlmann <gl...@uc...> wrote:
> Thanks for the PNG patch and the kind words. I will try it when I'm back on my laptop.
>
> It would be good if there were Teem CTests for tensor estimation and tractography. What do you think would have to happen for SPL or LMI to write those? (as in, what can I do to help make it happen)?
>
> Gordon
>
> On Oct 13, 2012, at 9:36, Steve Pieper <pi...@ib...> wrote:
>
>> Hi Gordon -
>>
>> I have a chance to build slicer against the latest teem - I only found
>> one build issue, relate to a deprecated png api call [1].  As you may
>> recall, in slicer we build teem against the png code that comes with
>> vtk (to simplify deployment shared library dependencies).  VTK's
>> internal version is still png 1.0.12.  According to the [2] the old
>> call was removed in 1.4 and the new call was available somewhere in
>> the version 1.0.18 to 1.2.9 timeframe (hard to tell for sure from the
>> comment).
>>
>> The small patch pasted below fixes the build for me and things seem to
>> all work as expected otherwise.  I think it's the right thing for all
>> versions of libpng, but that should be doublechecked.  I did submit an
>> experimental build to the teem dashboard [3] with the patched version.
>>
>> I've tried tensor estimation and tractography in slicer with the new
>> teem, and they look fine, but I didn't do any kind of quantitative
>> comparison.
>>
>> So overall it's good news - thanks for all the work on the new version of teem!
>>
>> -Steve
>>
>> [1] http://teem.svn.sourceforge.net/viewvc/teem/teem/trunk/src/nrrd/formatPNG.c?r1=4618&r2=4617&pathrev=4618
>>
>> [2] http://www.libpng.org/pub/png/src/libpng-1.2.x-to-1.4.x-summary.txt
>>
>> [3] http://my.cdash.org/viewTest.php?onlypassed&buildid=398555
>>
>>
>>
>>
>> pieper@boggs:/media/extra60/slicer4/latest/Slicer-superbuild-i4/teem$ svn diff
>> Index: src/nrrd/formatPNG.c
>> ===================================================================
>> --- src/nrrd/formatPNG.c    (revision 5783)
>> +++ src/nrrd/formatPNG.c    (working copy)
>> @@ -223,7 +223,11 @@
>>     png_set_palette_to_rgb(png);
>>   /* expand grayscale images to 8 bits from 1, 2, or 4 bits */
>>   if (type == PNG_COLOR_TYPE_GRAY && depth < 8)
>> +#if PNG_LIBPNG_VER_MAJOR == 1 && PNG_LIBPNG_VER_MINOR < 2
>> +    png_set_gray_1_2_4_to_8(png);
>> +#else
>>     png_set_expand_gray_1_2_4_to_8(png);
>> +#endif
>>   /* expand paletted or rgb images with transparency to full alpha
>>      channels so the data will be available as rgba quartets */
>>   if (png_get_valid(png, info, PNG_INFO_tRNS))
>>
>>
>>
>>
>> On Tue, Sep 25, 2012 at 4:20 PM, Gordon L. Kindlmann <gl...@uc...> wrote:
>>> Hello,
>>>
>>> If you're using Teem in any project, and you've been working with an svn checkout instead of the Dec 2008 release of 1.10, now would be a good time to do "svn update", rebuild, and see if things are still working as you'd expect.  There have been many updates.  On many files this amounted to the belated addition of "2012" to the copyright statement, but there have been many other more substantial updates.
>>>
>>> There will be many more updates before the 1.11 release, but if something has gone badly then we'd like to know sooner rather than later. Whatever means you have to check the working-ness of your program with Teem should be created and executed, right about now, so that you can do it again easily when we're closer to a 1.11 release.
>>>
>>> My sincere intent is to do a 1.11 release before Christmas this year.  The goal is for 1.11 to not have any API changes that break existing code.  Besides new functionality, there is more use of "const", and ints that should always have been unsigned are now unsigned.  Some odd functions that were never used or used once inside Teem have been removed or relocated and made static.  This work has motivated some thinking about more disruptive API changes that would make sense for Teem 2.0, see teem/src/TODO.txt for more on that.
>>>
>>> The substantial updates have originated with two things:
>>>
>>> (1) First was the creation of some tests (see teem/Testing), which were in turn motivated by some ongoing hacking within the gage library.  Along the way, some serious bugs were found and fixed (see my Sept 7 message "recent Teem work, and remaining work towards release").
>>>
>>> ************************************************************
>>> IF THERE IS SOME PART OF TEEM THAT IS IMPORTANT FOR YOUR WORK,
>>> WHY NOT HELP BY WRITING A TEST FOR IT?
>>> ************************************************************
>>>
>>> Add your tests to one of the directories in teem/Testing, following the example from existing tests. The convention now is to have all test binaries begin with "test_".
>>>
>>> If you didn't want to do a Teem svn update because you were afraid of your program breaking, you have all the necessary motivation to actually test whatever it is you're depending on. Teem will be better with your contribution.
>>>
>>> Tests can be simple or big, and can refer to datasets committed to teem/data (see teem/Testing/nrrd/tload.c for an example of this).  You can also call single unu and tend commands from their C API, which is a little odd but do-able; see teem/Testing/meet/probeSS.c for an example; this runs "tend helix".  Suggestions or utilities for making it easier to run unu/tend from a test are welcome.
>>>
>>> (2) The second source of non-trivial updates was the ongoing effort to get the NrrdIO in ITK re-synched with Teem.  An issue here had been fixing the various warnings that showed up in Visual Studio, since even if no Teem users care about Visual Studio, lots of ITK users do.  I now have a 64-bit Windows VM that I can use for testing.  I've also recently found that the "gcc -Wconversion" flag enables the same warnings and more, and have started on the slow work of fixing these (at least those that show up in NrrdIO).  These warning have revealed many many places (usually in old code) where "int" was used for all array indexing, instead of unsigned int.
>>>
>>> Comments and feedback welcome,
>>> Gordon
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> teem-users mailing list
>>> tee...@li...
>>> https://lists.sourceforge.net/lists/listinfo/teem-users
>>

Re: [Teem-users] updates and helping with tests

[Gordon L. K. <gl...@uc...>]

Thanks for the PNG patch and the kind words. I will try it when I'm back on my laptop.

It would be good if there were Teem CTests for tensor estimation and tractography. What do you think would have to happen for SPL or LMI to write those? (as in, what can I do to help make it happen)?

Gordon

On Oct 13, 2012, at 9:36, Steve Pieper <pi...@ib...> wrote:

> Hi Gordon -
> 
> I have a chance to build slicer against the latest teem - I only found
> one build issue, relate to a deprecated png api call [1].  As you may
> recall, in slicer we build teem against the png code that comes with
> vtk (to simplify deployment shared library dependencies).  VTK's
> internal version is still png 1.0.12.  According to the [2] the old
> call was removed in 1.4 and the new call was available somewhere in
> the version 1.0.18 to 1.2.9 timeframe (hard to tell for sure from the
> comment).
> 
> The small patch pasted below fixes the build for me and things seem to
> all work as expected otherwise.  I think it's the right thing for all
> versions of libpng, but that should be doublechecked.  I did submit an
> experimental build to the teem dashboard [3] with the patched version.
> 
> I've tried tensor estimation and tractography in slicer with the new
> teem, and they look fine, but I didn't do any kind of quantitative
> comparison.
> 
> So overall it's good news - thanks for all the work on the new version of teem!
> 
> -Steve
> 
> [1] http://teem.svn.sourceforge.net/viewvc/teem/teem/trunk/src/nrrd/formatPNG.c?r1=4618&r2=4617&pathrev=4618
> 
> [2] http://www.libpng.org/pub/png/src/libpng-1.2.x-to-1.4.x-summary.txt
> 
> [3] http://my.cdash.org/viewTest.php?onlypassed&buildid=398555
> 
> 
> 
> 
> pieper@boggs:/media/extra60/slicer4/latest/Slicer-superbuild-i4/teem$ svn diff
> Index: src/nrrd/formatPNG.c
> ===================================================================
> --- src/nrrd/formatPNG.c    (revision 5783)
> +++ src/nrrd/formatPNG.c    (working copy)
> @@ -223,7 +223,11 @@
>     png_set_palette_to_rgb(png);
>   /* expand grayscale images to 8 bits from 1, 2, or 4 bits */
>   if (type == PNG_COLOR_TYPE_GRAY && depth < 8)
> +#if PNG_LIBPNG_VER_MAJOR == 1 && PNG_LIBPNG_VER_MINOR < 2
> +    png_set_gray_1_2_4_to_8(png);
> +#else
>     png_set_expand_gray_1_2_4_to_8(png);
> +#endif
>   /* expand paletted or rgb images with transparency to full alpha
>      channels so the data will be available as rgba quartets */
>   if (png_get_valid(png, info, PNG_INFO_tRNS))
> 
> 
> 
> 
> On Tue, Sep 25, 2012 at 4:20 PM, Gordon L. Kindlmann <gl...@uc...> wrote:
>> Hello,
>> 
>> If you're using Teem in any project, and you've been working with an svn checkout instead of the Dec 2008 release of 1.10, now would be a good time to do "svn update", rebuild, and see if things are still working as you'd expect.  There have been many updates.  On many files this amounted to the belated addition of "2012" to the copyright statement, but there have been many other more substantial updates.
>> 
>> There will be many more updates before the 1.11 release, but if something has gone badly then we'd like to know sooner rather than later. Whatever means you have to check the working-ness of your program with Teem should be created and executed, right about now, so that you can do it again easily when we're closer to a 1.11 release.
>> 
>> My sincere intent is to do a 1.11 release before Christmas this year.  The goal is for 1.11 to not have any API changes that break existing code.  Besides new functionality, there is more use of "const", and ints that should always have been unsigned are now unsigned.  Some odd functions that were never used or used once inside Teem have been removed or relocated and made static.  This work has motivated some thinking about more disruptive API changes that would make sense for Teem 2.0, see teem/src/TODO.txt for more on that.
>> 
>> The substantial updates have originated with two things:
>> 
>> (1) First was the creation of some tests (see teem/Testing), which were in turn motivated by some ongoing hacking within the gage library.  Along the way, some serious bugs were found and fixed (see my Sept 7 message "recent Teem work, and remaining work towards release").
>> 
>> ************************************************************
>> IF THERE IS SOME PART OF TEEM THAT IS IMPORTANT FOR YOUR WORK,
>> WHY NOT HELP BY WRITING A TEST FOR IT?
>> ************************************************************
>> 
>> Add your tests to one of the directories in teem/Testing, following the example from existing tests. The convention now is to have all test binaries begin with "test_".
>> 
>> If you didn't want to do a Teem svn update because you were afraid of your program breaking, you have all the necessary motivation to actually test whatever it is you're depending on. Teem will be better with your contribution.
>> 
>> Tests can be simple or big, and can refer to datasets committed to teem/data (see teem/Testing/nrrd/tload.c for an example of this).  You can also call single unu and tend commands from their C API, which is a little odd but do-able; see teem/Testing/meet/probeSS.c for an example; this runs "tend helix".  Suggestions or utilities for making it easier to run unu/tend from a test are welcome.
>> 
>> (2) The second source of non-trivial updates was the ongoing effort to get the NrrdIO in ITK re-synched with Teem.  An issue here had been fixing the various warnings that showed up in Visual Studio, since even if no Teem users care about Visual Studio, lots of ITK users do.  I now have a 64-bit Windows VM that I can use for testing.  I've also recently found that the "gcc -Wconversion" flag enables the same warnings and more, and have started on the slow work of fixing these (at least those that show up in NrrdIO).  These warning have revealed many many places (usually in old code) where "int" was used for all array indexing, instead of unsigned int.
>> 
>> Comments and feedback welcome,
>> Gordon
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> teem-users mailing list
>> tee...@li...
>> https://lists.sourceforge.net/lists/listinfo/teem-users
> 

Re: [Teem-users] updates and helping with tests

[Steve P. <pi...@ib...>]

Hi Gordon -

I have a chance to build slicer against the latest teem - I only found
one build issue, relate to a deprecated png api call [1].  As you may
recall, in slicer we build teem against the png code that comes with
vtk (to simplify deployment shared library dependencies).  VTK's
internal version is still png 1.0.12.  According to the [2] the old
call was removed in 1.4 and the new call was available somewhere in
the version 1.0.18 to 1.2.9 timeframe (hard to tell for sure from the
comment).

The small patch pasted below fixes the build for me and things seem to
all work as expected otherwise.  I think it's the right thing for all
versions of libpng, but that should be doublechecked.  I did submit an
experimental build to the teem dashboard [3] with the patched version.

I've tried tensor estimation and tractography in slicer with the new
teem, and they look fine, but I didn't do any kind of quantitative
comparison.

So overall it's good news - thanks for all the work on the new version of teem!

-Steve

[1] http://teem.svn.sourceforge.net/viewvc/teem/teem/trunk/src/nrrd/formatPNG.c?r1=4618&r2=4617&pathrev=4618

[2] http://www.libpng.org/pub/png/src/libpng-1.2.x-to-1.4.x-summary.txt

[3] http://my.cdash.org/viewTest.php?onlypassed&buildid=398555




pieper@boggs:/media/extra60/slicer4/latest/Slicer-superbuild-i4/teem$ svn diff
Index: src/nrrd/formatPNG.c
===================================================================
--- src/nrrd/formatPNG.c	(revision 5783)
+++ src/nrrd/formatPNG.c	(working copy)
@@ -223,7 +223,11 @@
     png_set_palette_to_rgb(png);
   /* expand grayscale images to 8 bits from 1, 2, or 4 bits */
   if (type == PNG_COLOR_TYPE_GRAY && depth < 8)
+#if PNG_LIBPNG_VER_MAJOR == 1 && PNG_LIBPNG_VER_MINOR < 2
+    png_set_gray_1_2_4_to_8(png);
+#else
     png_set_expand_gray_1_2_4_to_8(png);
+#endif
   /* expand paletted or rgb images with transparency to full alpha
      channels so the data will be available as rgba quartets */
   if (png_get_valid(png, info, PNG_INFO_tRNS))




On Tue, Sep 25, 2012 at 4:20 PM, Gordon L. Kindlmann <gl...@uc...> wrote:
> Hello,
>
> If you're using Teem in any project, and you've been working with an svn checkout instead of the Dec 2008 release of 1.10, now would be a good time to do "svn update", rebuild, and see if things are still working as you'd expect.  There have been many updates.  On many files this amounted to the belated addition of "2012" to the copyright statement, but there have been many other more substantial updates.
>
> There will be many more updates before the 1.11 release, but if something has gone badly then we'd like to know sooner rather than later. Whatever means you have to check the working-ness of your program with Teem should be created and executed, right about now, so that you can do it again easily when we're closer to a 1.11 release.
>
> My sincere intent is to do a 1.11 release before Christmas this year.  The goal is for 1.11 to not have any API changes that break existing code.  Besides new functionality, there is more use of "const", and ints that should always have been unsigned are now unsigned.  Some odd functions that were never used or used once inside Teem have been removed or relocated and made static.  This work has motivated some thinking about more disruptive API changes that would make sense for Teem 2.0, see teem/src/TODO.txt for more on that.
>
> The substantial updates have originated with two things:
>
> (1) First was the creation of some tests (see teem/Testing), which were in turn motivated by some ongoing hacking within the gage library.  Along the way, some serious bugs were found and fixed (see my Sept 7 message "recent Teem work, and remaining work towards release").
>
> ************************************************************
> IF THERE IS SOME PART OF TEEM THAT IS IMPORTANT FOR YOUR WORK,
> WHY NOT HELP BY WRITING A TEST FOR IT?
> ************************************************************
>
> Add your tests to one of the directories in teem/Testing, following the example from existing tests. The convention now is to have all test binaries begin with "test_".
>
> If you didn't want to do a Teem svn update because you were afraid of your program breaking, you have all the necessary motivation to actually test whatever it is you're depending on. Teem will be better with your contribution.
>
> Tests can be simple or big, and can refer to datasets committed to teem/data (see teem/Testing/nrrd/tload.c for an example of this).  You can also call single unu and tend commands from their C API, which is a little odd but do-able; see teem/Testing/meet/probeSS.c for an example; this runs "tend helix".  Suggestions or utilities for making it easier to run unu/tend from a test are welcome.
>
> (2) The second source of non-trivial updates was the ongoing effort to get the NrrdIO in ITK re-synched with Teem.  An issue here had been fixing the various warnings that showed up in Visual Studio, since even if no Teem users care about Visual Studio, lots of ITK users do.  I now have a 64-bit Windows VM that I can use for testing.  I've also recently found that the "gcc -Wconversion" flag enables the same warnings and more, and have started on the slow work of fixing these (at least those that show up in NrrdIO).  These warning have revealed many many places (usually in old code) where "int" was used for all array indexing, instead of unsigned int.
>
> Comments and feedback welcome,
> Gordon
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users

Re: [Teem-users] updates and helping with tests

[Gordon L. K. <gl...@uc...>]

The link on the web page has been fixed.

Do others here on the list have opinions about how to prioritize which pages at http://teem.sourceforge.net are most important to update?

Gordon

On Oct 12, 2012, at 9:33 PM, Gordon L. Kindlmann wrote:

> 
> On Oct 12, 2012, at 5:16 PM, Sean McBride wrote:
> 
>> On Fri, 12 Oct 2012 13:04:37 -0500, Gordon L. Kindlmann said:
>> 
>>> 
>>> A quick update on the following:
>>> 
>>> The synching of ITK's and Teem's NrrdIO has reached the point of an ITK
>>> patch being submitted to gerrit:
>>> 
>>> http://review.source.kitware.com/#/c/7891/
>>> 
>>> Many more Teem updates were done to permit this patch to go through to
>>> this point without (at least for now) without any compiler warnings.
>>> 
>>> Again: If you are using Teem for something (anything), and you want to
>>> have your project's usage of Teem move into the future instead of being
>>> stuck with some unmaintained past version, then you should do an "svn
>>> update" and see if your project still works with the updated Teem.
>> 
>> Is there a teem dashboard?
>> 
>> The link here:
>> <http://teem.sourceforge.net>
>> 
>> seems to go to the VTK dashboard...
>> 
>> Cheers,
>> 
>> --
>> ____________________________________________________________
>> Sean McBride, B. Eng                 se...@ro...
>> Rogue Research                        www.rogue-research.com
>> Mac Software Developer              Montréal, Québec, Canada
>> 
> 
> 
> Wow, the web pages are really completely busted.  Sorry about that- the link is from when NAMIC-associated dashboards were organized differently.  Here's the link:
> 
> http://my.cdash.org/index.php?project=Teem
> 
> Having more different mac builds contributing there would be tremendous :)
> 
> BTW, gerrit patch above includes the removal of signed integer overflow in the sanity checks.
> 
> cheers,
> Gordon
> 
> 
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users
> 

Re: [Teem-users] updates and helping with tests

[Gordon L. K. <gl...@uc...>]

On Oct 12, 2012, at 5:16 PM, Sean McBride wrote:

> On Fri, 12 Oct 2012 13:04:37 -0500, Gordon L. Kindlmann said:
> 
>> 
>> A quick update on the following:
>> 
>> The synching of ITK's and Teem's NrrdIO has reached the point of an ITK
>> patch being submitted to gerrit:
>> 
>> http://review.source.kitware.com/#/c/7891/
>> 
>> Many more Teem updates were done to permit this patch to go through to
>> this point without (at least for now) without any compiler warnings.
>> 
>> Again: If you are using Teem for something (anything), and you want to
>> have your project's usage of Teem move into the future instead of being
>> stuck with some unmaintained past version, then you should do an "svn
>> update" and see if your project still works with the updated Teem.
> 
> Is there a teem dashboard?
> 
> The link here:
> <http://teem.sourceforge.net>
> 
> seems to go to the VTK dashboard...
> 
> Cheers,
> 
> --
> ____________________________________________________________
> Sean McBride, B. Eng                 se...@ro...
> Rogue Research                        www.rogue-research.com
> Mac Software Developer              Montréal, Québec, Canada
> 


Wow, the web pages are really completely busted.  Sorry about that- the link is from when NAMIC-associated dashboards were organized differently.  Here's the link:

http://my.cdash.org/index.php?project=Teem

Having more different mac builds contributing there would be tremendous :)

BTW, gerrit patch above includes the removal of signed integer overflow in the sanity checks.

cheers,
Gordon

Re: [Teem-users] updates and helping with tests

[Sean M. <se...@ro...>]

On Fri, 12 Oct 2012 13:04:37 -0500, Gordon L. Kindlmann said:

>
>A quick update on the following:
>
>The synching of ITK's and Teem's NrrdIO has reached the point of an ITK
>patch being submitted to gerrit:
>
>http://review.source.kitware.com/#/c/7891/
>
>Many more Teem updates were done to permit this patch to go through to
>this point without (at least for now) without any compiler warnings.
>
>Again: If you are using Teem for something (anything), and you want to
>have your project's usage of Teem move into the future instead of being
>stuck with some unmaintained past version, then you should do an "svn
>update" and see if your project still works with the updated Teem.

Is there a teem dashboard?

The link here:
<http://teem.sourceforge.net>

seems to go to the VTK dashboard...

Cheers,

-- 
____________________________________________________________
Sean McBride, B. Eng                 se...@ro...
Rogue Research                        www.rogue-research.com 
Mac Software Developer              Montréal, Québec, Canada

Re: [Teem-users] updates and helping with tests

[Gordon L. K. <gl...@uc...>]

A quick update on the following:

The synching of ITK's and Teem's NrrdIO has reached the point of an ITK patch being submitted to gerrit:

http://review.source.kitware.com/#/c/7891/

Many more Teem updates were done to permit this patch to go through to this point without (at least for now) without any compiler warnings.

Again: If you are using Teem for something (anything), and you want to have your project's usage of Teem move into the future instead of being stuck with some unmaintained past version, then you should do an "svn update" and see if your project still works with the updated Teem.

Adding your own tests (to teem/Testing, email me for svn write access!) is a beautiful way of documenting the Teem behavior that your project depends on.

I've set a tentative 1.11 release date of Dec 19 2012.

Gordon

On Sep 25, 2012, at 3:20 PM, Gordon L. Kindlmann wrote:

> Hello,
> 
> If you're using Teem in any project, and you've been working with an svn checkout instead of the Dec 2008 release of 1.10, now would be a good time to do "svn update", rebuild, and see if things are still working as you'd expect.  There have been many updates.  On many files this amounted to the belated addition of "2012" to the copyright statement, but there have been many other more substantial updates.
> 
> There will be many more updates before the 1.11 release, but if something has gone badly then we'd like to know sooner rather than later. Whatever means you have to check the working-ness of your program with Teem should be created and executed, right about now, so that you can do it again easily when we're closer to a 1.11 release.
> 
> My sincere intent is to do a 1.11 release before Christmas this year.  The goal is for 1.11 to not have any API changes that break existing code.  Besides new functionality, there is more use of "const", and ints that should always have been unsigned are now unsigned.  Some odd functions that were never used or used once inside Teem have been removed or relocated and made static.  This work has motivated some thinking about more disruptive API changes that would make sense for Teem 2.0, see teem/src/TODO.txt for more on that.
> 
> The substantial updates have originated with two things:
> 
> (1) First was the creation of some tests (see teem/Testing), which were in turn motivated by some ongoing hacking within the gage library.  Along the way, some serious bugs were found and fixed (see my Sept 7 message "recent Teem work, and remaining work towards release").  
> 
> ************************************************************
> IF THERE IS SOME PART OF TEEM THAT IS IMPORTANT FOR YOUR WORK,
> WHY NOT HELP BY WRITING A TEST FOR IT?
> ************************************************************
> 
> Add your tests to one of the directories in teem/Testing, following the example from existing tests. The convention now is to have all test binaries begin with "test_".
> 
> If you didn't want to do a Teem svn update because you were afraid of your program breaking, you have all the necessary motivation to actually test whatever it is you're depending on. Teem will be better with your contribution.
> 
> Tests can be simple or big, and can refer to datasets committed to teem/data (see teem/Testing/nrrd/tload.c for an example of this).  You can also call single unu and tend commands from their C API, which is a little odd but do-able; see teem/Testing/meet/probeSS.c for an example; this runs "tend helix".  Suggestions or utilities for making it easier to run unu/tend from a test are welcome.
> 
> (2) The second source of non-trivial updates was the ongoing effort to get the NrrdIO in ITK re-synched with Teem.  An issue here had been fixing the various warnings that showed up in Visual Studio, since even if no Teem users care about Visual Studio, lots of ITK users do.  I now have a 64-bit Windows VM that I can use for testing.  I've also recently found that the "gcc -Wconversion" flag enables the same warnings and more, and have started on the slow work of fixing these (at least those that show up in NrrdIO).  These warning have revealed many many places (usually in old code) where "int" was used for all array indexing, instead of unsigned int.
> 
> Comments and feedback welcome,
> Gordon
> 
> 

Re: [Teem-users] unu cksum, and a question of endianness

[Gordon L. K. <gl...@uc...>]

On Oct 9, 2012, at 12:38 PM, James Bigler wrote:

> On Tue, Oct 9, 2012 at 11:22 AM, Thomas Schultz <tsc...@tu...> wrote:
> On Tue, 2012-10-09 at 11:05 -0600, James Bigler wrote:
> [...]
> > What does it hurt to always compute it in an endian agnostic way (as
> > from the users perspective)?
> 
> From the user's perspective, it is reasonable to assume that running unu
> cksum will produce the same output as running the system's cksum command
> on a detached .raw data file. In this sense, Gordon's current
> implementation conforms to the principle of least astonishment, while
> still giving you the option of an endian-agnostic between-system
> comparison.
> 
> Best /Thomas
> 
> 
> Perhaps I misunderstood what Gordon implemented, but I thought he did the checksum on the bytes in memory.  Nrrd will reorder the bytes from the file's endian format to the system's endian format which could produce a different checksum than if run externally to nrrd if the file's endianness doesn't match the system's endianness.  In order to be consistent with what you mentioned nrrd would have to keep around some flag that indicated the "original" endianness.
> 
> James
> 
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users

Yes, the checksum is computed on bytes in memory.  And, its still the case that nrrdRead() will swap bytes if the endianness on disk is different than the run-time endianness: I saw little value in keeping the array in memory with the wrong endianness.  The endianness on disk should be remembered in nrrdIOState->endian, but unless you've passed your own nrrdIOState to nrrdRead or nrrdLoad that information is gone after nrrdRead or nrrdLoad returns.

And that's ok: assuming that the data was written with local endianness (be it big or little), then "unu cksum" and "cksum" will agree, which I feel is least astonishing.  They will differ if someone saved the data big-endian and you're running on a little-endian machine or vice versa.  IMHO there really is no "original" endianness - the endianness is not an intrinsic property of the array data, its always an accident of the platform, so it doesn't merit representation in the Nrrd struct itself.

Anyway, this is why I added the "unu cksum -pen" option, so that you can annotate the cksum with the endianness used.  

% cksum mode.raw 
3399162503 500000 mode.raw
% unu cksum mode.nhdr
1045014522 500000 mode.nhdr
% unu cksum -pen true mode.nhdr
1045014522(little) 500000 mode.nhdr
% unu cksum -en big -pen true mode.nhdr
3399162503(big) 500000 mode.nhdr

I guess we could also discuss whether the default to -pen should be true or false: true means that the results of unu cksum are annotated and may reduce confusion, but are thus dissimilar from cksum output; false means the the results look just like the output from cksum.

Gordon

Re: [Teem-users] unu cksum, and a question of endianness

[James B. <jam...@gm...>]

On Tue, Oct 9, 2012 at 11:22 AM, Thomas Schultz
<tsc...@tu...>wrote:

> On Tue, 2012-10-09 at 11:05 -0600, James Bigler wrote:
> [...]
> > What does it hurt to always compute it in an endian agnostic way (as
> > from the users perspective)?
>
> From the user's perspective, it is reasonable to assume that running unu
> cksum will produce the same output as running the system's cksum command
> on a detached .raw data file. In this sense, Gordon's current
> implementation conforms to the principle of least astonishment, while
> still giving you the option of an endian-agnostic between-system
> comparison.
>
> Best /Thomas
>
>
Perhaps I misunderstood what Gordon implemented, but I thought he did the
checksum on the bytes in memory.  Nrrd will reorder the bytes from the
file's endian format to the system's endian format which could produce a
different checksum than if run externally to nrrd if the file's endianness
doesn't match the system's endianness.  In order to be consistent with what
you mentioned nrrd would have to keep around some flag that indicated the
"original" endianness.

James

Re: [Teem-users] unu cksum, and a question of endianness

[Thomas S. <tsc...@tu...>]

On Tue, 2012-10-09 at 11:05 -0600, James Bigler wrote:
[...]
> What does it hurt to always compute it in an endian agnostic way (as
> from the users perspective)?

>From the user's perspective, it is reasonable to assume that running unu
cksum will produce the same output as running the system's cksum command
on a detached .raw data file. In this sense, Gordon's current
implementation conforms to the principle of least astonishment, while
still giving you the option of an endian-agnostic between-system
comparison.

Best /Thomas

Re: [Teem-users] unu cksum, and a question of endianness

[James B. <jam...@gm...>]

On Tue, Oct 9, 2012 at 10:09 AM, Gordon L. Kindlmann <gl...@uc...>wrote:

>
> On Oct 9, 2012, at 10:33 AM, Gordon L. Kindlmann wrote:
>
>
> On Oct 9, 2012, at 10:20 AM, James Bigler wrote:
>
>
>
> On Tue, Oct 9, 2012 at 8:43 AM, Gordon L. Kindlmann <gl...@uc...>wrote:
>
>> Hello,
>>
>> As part of doing some more testing, I wanted to have a way of generating
>> a checksum for data, so I wrote airCRC32() (teem/src/air/math.c) and
>> nrrdCRC32() (teem/src/nrrd/arith.c), which calls airCRC32() on nrrd->data,
>> and "unu cksum" which calls nrrdCRC32().  By design, the 32-bit unsigned
>> int returned by "unu ckum tmp.nhdr" will be the same as "cksum tmp.raw" on
>> the underlying raw data.
>>
>> However, the CRC computation is on a simple sequence of bytes, so for
>> multi-byte values (e.g. ints or floats), the result depends on endianness.
>>  That seems like perhaps a problem - you could imagine wanting a way of
>> getting a checksum on nrrd data that would be intrinsic to the data values
>> and not to their byte ordering.  On the other hand, when you save out to a
>> .nhdr/.raw pair, the default behavior is to write out data with local
>> endianness, which the "cksum" tool will also be sensitive to, so the "unu
>> cksum" and "cksum" results would agree.
>>
>> So which is better?  Having "cksum" and "unu cksum" agree, even though
>> they'll generate different answers for the same array values on different
>> endian machines, or, having "unu cksum" return the same value for the same
>> array values, regardless of endianness?
>>
>> Gordon
>>
>>
>> I'd like to think of what would be the least surprising to me.  I would
> expect the checksum to be the same regardless of the machine it was run on.
>
> If I there were a field in the header that said what the checksum should
> be (for data integrity), I would expect that checksum to match regardless
> of what machine I loaded the data on.  In addition if someone listed the
> checksum on a web page I would expect the checksum to match on my machine
> regardless of what endianness of the machine is.  If I want to share the
> checksum, do I now need to provide two checksums, one for big and one for
> little endianness?  That seems like it would be a source of confusion.
>
> I think the important thing is that the checksum provides some assurance
> of the data.  Since nrrd is endian aware, it should swizzle the bits to
> make sure you get the same checksum on big and little endian machines (it
> should probably swizzle the bits on big endian machines since they are less
> common).
>
> James
>
>
> Having a checksum field in the nrrd file header is a great idea; but
> adding it will entail updating the file format, which I am not going to
> touch until the 1.11 release is all done.
>
> So James, you're voting for always doing the CRC computation in
> little-endian, since its the most common.  Makes sense.  But there are some
> other heuristics you could use:
>
> 2) always big-endian, because that's "network order", so (I feel) its the
> closest thing to a platform-independent endianness.
>
> 3a) allow either one to be specified as the ordering to use for the CRC
> computation; but defaulting to the local endianness.  This is what "unu
> save" does now, for example.  If you want to check CRCs with friends, you'd
> tell them to run either "unu cksum -en big" or "unu cksum -en little" (your
> choice).  Then, since the endianness has been explicitly given, you can be
> sure it will give the same everywhere.
>
> 3b) like 3a but default to big endian
>
> 3c) like 3a but default to little endian.
>
> My preference is for one of 3a/b/c.  Any other feelings on this?
>
> Gordon
>
>
> I've implemented and committed choice 3a (see below).  The difference
> between 3a/3b/3c is only the default value given to hestOptAdd; so that's
> very easy to change.
>
> Gordon
>
> unu cksum: Compute 32-bit CRC of nrrd data (same as via "cksum"). Unlike
> other commands, this doesn't produce a nrrd. It only prints to standard out
> the CRC and byte counts for the input nrrd(s), seeking to emulate the
> formatting of cksum output.
> * Uses nrrdCRC32
>
> Usage: unu cksum [-en,--endian <end>] [-pen,--printendian <bool>] <nin1
> ...>
>
>   -en <end> , --endian <end> = Endianness in which to compute CRC; "little"
>               for Intel and friends; "big" for everyone else. Defaults to
>               endianness of this machine; default: "little"
> -pen <bool> , --printendian <bool> = whether or not to indicate after the
> CRC
>               value the endianness with which the CRC was computed; doing
> so
>               clarifies that the CRC result depends on endianness and may
>               remove confusion in comparing results on platforms of
> different
>               endianness (bool); default: "false"
>  <nin1 ...> = input nrrd(s) (1 or more strings)
>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users
>
>
I think people having to think about the endianness is a bad idea.  The
wonderful thing about nrrd is that I never had to think about the
endianness.  It was encoded in the file, so I never had to worry about it.
Now whoever uses the checksum (produces/consumes) needs to worry about the
endianness of the machine where it was generated from and the current
machine.  I think having a knob to compute it little or big endian is just
one more knob people have to worry about.  Less knobs == less confusion.

What does it hurt to always compute it in an endian agnostic way (as from
the users perspective)?

BTW, I wasn't suggesting that the checksum be part of the NRRD spec, that
was simply an example of how the checksum could be computed.

James

Re: [Teem-users] unu cksum, and a question of endianness

[Gordon L. K. <gl...@uc...>]

On Oct 9, 2012, at 10:33 AM, Gordon L. Kindlmann wrote:

> 
> On Oct 9, 2012, at 10:20 AM, James Bigler wrote:
> 
>> 
>> 
>> On Tue, Oct 9, 2012 at 8:43 AM, Gordon L. Kindlmann <gl...@uc...> wrote:
>> Hello,
>> 
>> As part of doing some more testing, I wanted to have a way of generating a checksum for data, so I wrote airCRC32() (teem/src/air/math.c) and nrrdCRC32() (teem/src/nrrd/arith.c), which calls airCRC32() on nrrd->data, and "unu cksum" which calls nrrdCRC32().  By design, the 32-bit unsigned int returned by "unu ckum tmp.nhdr" will be the same as "cksum tmp.raw" on the underlying raw data.
>> 
>> However, the CRC computation is on a simple sequence of bytes, so for multi-byte values (e.g. ints or floats), the result depends on endianness.  That seems like perhaps a problem - you could imagine wanting a way of getting a checksum on nrrd data that would be intrinsic to the data values and not to their byte ordering.  On the other hand, when you save out to a .nhdr/.raw pair, the default behavior is to write out data with local endianness, which the "cksum" tool will also be sensitive to, so the "unu cksum" and "cksum" results would agree.
>> 
>> So which is better?  Having "cksum" and "unu cksum" agree, even though they'll generate different answers for the same array values on different endian machines, or, having "unu cksum" return the same value for the same array values, regardless of endianness?
>> 
>> Gordon
>> 
>> 
>> I'd like to think of what would be the least surprising to me.  I would expect the checksum to be the same regardless of the machine it was run on.
>> 
>> If I there were a field in the header that said what the checksum should be (for data integrity), I would expect that checksum to match regardless of what machine I loaded the data on.  In addition if someone listed the checksum on a web page I would expect the checksum to match on my machine regardless of what endianness of the machine is.  If I want to share the checksum, do I now need to provide two checksums, one for big and one for little endianness?  That seems like it would be a source of confusion.
>> 
>> I think the important thing is that the checksum provides some assurance of the data.  Since nrrd is endian aware, it should swizzle the bits to make sure you get the same checksum on big and little endian machines (it should probably swizzle the bits on big endian machines since they are less common).
>> 
>> James
> 
> Having a checksum field in the nrrd file header is a great idea; but adding it will entail updating the file format, which I am not going to touch until the 1.11 release is all done.
> 
> So James, you're voting for always doing the CRC computation in little-endian, since its the most common.  Makes sense.  But there are some other heuristics you could use:
> 
> 2) always big-endian, because that's "network order", so (I feel) its the closest thing to a platform-independent endianness.
> 
> 3a) allow either one to be specified as the ordering to use for the CRC computation; but defaulting to the local endianness.  This is what "unu save" does now, for example.  If you want to check CRCs with friends, you'd tell them to run either "unu cksum -en big" or "unu cksum -en little" (your choice).  Then, since the endianness has been explicitly given, you can be sure it will give the same everywhere.
> 
> 3b) like 3a but default to big endian
> 
> 3c) like 3a but default to little endian.
> 
> My preference is for one of 3a/b/c.  Any other feelings on this?
> 
> Gordon

I've implemented and committed choice 3a (see below).  The difference between 3a/3b/3c is only the default value given to hestOptAdd; so that's very easy to change.

Gordon

unu cksum: Compute 32-bit CRC of nrrd data (same as via "cksum"). Unlike
other commands, this doesn't produce a nrrd. It only prints to standard out
the CRC and byte counts for the input nrrd(s), seeking to emulate the
formatting of cksum output.
* Uses nrrdCRC32

Usage: unu cksum [-en,--endian <end>] [-pen,--printendian <bool>] <nin1 ...>

  -en <end> , --endian <end> = Endianness in which to compute CRC; "little"
              for Intel and friends; "big" for everyone else. Defaults to
              endianness of this machine; default: "little"
-pen <bool> , --printendian <bool> = whether or not to indicate after the CRC
              value the endianness with which the CRC was computed; doing so
              clarifies that the CRC result depends on endianness and may
              remove confusion in comparing results on platforms of different
              endianness (bool); default: "false"
 <nin1 ...> = input nrrd(s) (1 or more strings)

Re: [Teem-users] unu cksum, and a question of endianness

[Gordon L. K. <gl...@uc...>]

On Oct 9, 2012, at 10:20 AM, James Bigler wrote:

> 
> 
> On Tue, Oct 9, 2012 at 8:43 AM, Gordon L. Kindlmann <gl...@uc...> wrote:
> Hello,
> 
> As part of doing some more testing, I wanted to have a way of generating a checksum for data, so I wrote airCRC32() (teem/src/air/math.c) and nrrdCRC32() (teem/src/nrrd/arith.c), which calls airCRC32() on nrrd->data, and "unu cksum" which calls nrrdCRC32().  By design, the 32-bit unsigned int returned by "unu ckum tmp.nhdr" will be the same as "cksum tmp.raw" on the underlying raw data.
> 
> However, the CRC computation is on a simple sequence of bytes, so for multi-byte values (e.g. ints or floats), the result depends on endianness.  That seems like perhaps a problem - you could imagine wanting a way of getting a checksum on nrrd data that would be intrinsic to the data values and not to their byte ordering.  On the other hand, when you save out to a .nhdr/.raw pair, the default behavior is to write out data with local endianness, which the "cksum" tool will also be sensitive to, so the "unu cksum" and "cksum" results would agree.
> 
> So which is better?  Having "cksum" and "unu cksum" agree, even though they'll generate different answers for the same array values on different endian machines, or, having "unu cksum" return the same value for the same array values, regardless of endianness?
> 
> Gordon
> 
> 
> I'd like to think of what would be the least surprising to me.  I would expect the checksum to be the same regardless of the machine it was run on.
> 
> If I there were a field in the header that said what the checksum should be (for data integrity), I would expect that checksum to match regardless of what machine I loaded the data on.  In addition if someone listed the checksum on a web page I would expect the checksum to match on my machine regardless of what endianness of the machine is.  If I want to share the checksum, do I now need to provide two checksums, one for big and one for little endianness?  That seems like it would be a source of confusion.
> 
> I think the important thing is that the checksum provides some assurance of the data.  Since nrrd is endian aware, it should swizzle the bits to make sure you get the same checksum on big and little endian machines (it should probably swizzle the bits on big endian machines since they are less common).
> 
> James

Having a checksum field in the nrrd file header is a great idea; but adding it will entail updating the file format, which I am not going to touch until the 1.11 release is all done.

So James, you're voting for always doing the CRC computation in little-endian, since its the most common.  Makes sense.  But there are some other heuristics you could use:

2) always big-endian, because that's "network order", so (I feel) its the closest thing to a platform-independent endianness.

3a) allow either one to be specified as the ordering to use for the CRC computation; but defaulting to the local endianness.  This is what "unu save" does now, for example.  If you want to check CRCs with friends, you'd tell them to run either "unu cksum -en big" or "unu cksum -en little" (your choice).  Then, since the endianness has been explicitly given, you can be sure it will give the same everywhere.

3b) like 3a but default to big endian

3c) like 3a but default to little endian.

My preference is for one of 3a/b/c.  Any other feelings on this?

Gordon

Re: [Teem-users] unu cksum, and a question of endianness

[James B. <jam...@gm...>]

On Tue, Oct 9, 2012 at 8:43 AM, Gordon L. Kindlmann <gl...@uc...>wrote:

> Hello,
>
> As part of doing some more testing, I wanted to have a way of generating a
> checksum for data, so I wrote airCRC32() (teem/src/air/math.c) and
> nrrdCRC32() (teem/src/nrrd/arith.c), which calls airCRC32() on nrrd->data,
> and "unu cksum" which calls nrrdCRC32().  By design, the 32-bit unsigned
> int returned by "unu ckum tmp.nhdr" will be the same as "cksum tmp.raw" on
> the underlying raw data.
>
> However, the CRC computation is on a simple sequence of bytes, so for
> multi-byte values (e.g. ints or floats), the result depends on endianness.
>  That seems like perhaps a problem - you could imagine wanting a way of
> getting a checksum on nrrd data that would be intrinsic to the data values
> and not to their byte ordering.  On the other hand, when you save out to a
> .nhdr/.raw pair, the default behavior is to write out data with local
> endianness, which the "cksum" tool will also be sensitive to, so the "unu
> cksum" and "cksum" results would agree.
>
> So which is better?  Having "cksum" and "unu cksum" agree, even though
> they'll generate different answers for the same array values on different
> endian machines, or, having "unu cksum" return the same value for the same
> array values, regardless of endianness?
>
> Gordon
>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> teem-users mailing list
> tee...@li...
> https://lists.sourceforge.net/lists/listinfo/teem-users
>

I'd like to think of what would be the least surprising to me.  I would
expect the checksum to be the same regardless of the machine it was run on.

If I there were a field in the header that said what the checksum should be
(for data integrity), I would expect that checksum to match regardless of
what machine I loaded the data on.  In addition if someone listed the
checksum on a web page I would expect the checksum to match on my machine
regardless of what endianness of the machine is.  If I want to share the
checksum, do I now need to provide two checksums, one for big and one for
little endianness?  That seems like it would be a source of confusion.

I think the important thing is that the checksum provides some assurance of
the data.  Since nrrd is endian aware, it should swizzle the bits to make
sure you get the same checksum on big and little endian machines (it should
probably swizzle the bits on big endian machines since they are less
common).

James

Re: [Teem-users] unu cksum, and a question of endianness

[Thomas S. <tsc...@tu...>]

Hi,

On Tue, 2012-10-09 at 09:43 -0500, Gordon L. Kindlmann wrote:

> So which is better?  Having "cksum" and "unu cksum" agree, even though
>  they'll generate different answers for the same array values on
>  different endian machines, or, having "unu cksum" return the same
>  value for the same array values, regardless of endianness?

I see potential use cases for both alternatives - is it an option to
make it an option (the current behavior being the default)?

Best /Thomas

[Teem-users] unu cksum, and a question of endianness

[Gordon L. K. <gl...@uc...>]

Hello,

As part of doing some more testing, I wanted to have a way of generating a checksum for data, so I wrote airCRC32() (teem/src/air/math.c) and nrrdCRC32() (teem/src/nrrd/arith.c), which calls airCRC32() on nrrd->data, and "unu cksum" which calls nrrdCRC32().  By design, the 32-bit unsigned int returned by "unu ckum tmp.nhdr" will be the same as "cksum tmp.raw" on the underlying raw data.

However, the CRC computation is on a simple sequence of bytes, so for multi-byte values (e.g. ints or floats), the result depends on endianness.  That seems like perhaps a problem - you could imagine wanting a way of getting a checksum on nrrd data that would be intrinsic to the data values and not to their byte ordering.  On the other hand, when you save out to a .nhdr/.raw pair, the default behavior is to write out data with local endianness, which the "cksum" tool will also be sensitive to, so the "unu cksum" and "cksum" results would agree.

So which is better?  Having "cksum" and "unu cksum" agree, even though they'll generate different answers for the same array values on different endian machines, or, having "unu cksum" return the same value for the same array values, regardless of endianness?

Gordon

[Teem-users] updates and helping with tests

[Gordon L. K. <gl...@uc...>]

Hello,

If you're using Teem in any project, and you've been working with an svn checkout instead of the Dec 2008 release of 1.10, now would be a good time to do "svn update", rebuild, and see if things are still working as you'd expect.  There have been many updates.  On many files this amounted to the belated addition of "2012" to the copyright statement, but there have been many other more substantial updates.

There will be many more updates before the 1.11 release, but if something has gone badly then we'd like to know sooner rather than later. Whatever means you have to check the working-ness of your program with Teem should be created and executed, right about now, so that you can do it again easily when we're closer to a 1.11 release.

My sincere intent is to do a 1.11 release before Christmas this year.  The goal is for 1.11 to not have any API changes that break existing code.  Besides new functionality, there is more use of "const", and ints that should always have been unsigned are now unsigned.  Some odd functions that were never used or used once inside Teem have been removed or relocated and made static.  This work has motivated some thinking about more disruptive API changes that would make sense for Teem 2.0, see teem/src/TODO.txt for more on that.

The substantial updates have originated with two things:

(1) First was the creation of some tests (see teem/Testing), which were in turn motivated by some ongoing hacking within the gage library.  Along the way, some serious bugs were found and fixed (see my Sept 7 message "recent Teem work, and remaining work towards release").  

************************************************************
IF THERE IS SOME PART OF TEEM THAT IS IMPORTANT FOR YOUR WORK,
WHY NOT HELP BY WRITING A TEST FOR IT?
************************************************************

Add your tests to one of the directories in teem/Testing, following the example from existing tests. The convention now is to have all test binaries begin with "test_".

If you didn't want to do a Teem svn update because you were afraid of your program breaking, you have all the necessary motivation to actually test whatever it is you're depending on. Teem will be better with your contribution.

Tests can be simple or big, and can refer to datasets committed to teem/data (see teem/Testing/nrrd/tload.c for an example of this).  You can also call single unu and tend commands from their C API, which is a little odd but do-able; see teem/Testing/meet/probeSS.c for an example; this runs "tend helix".  Suggestions or utilities for making it easier to run unu/tend from a test are welcome.

(2) The second source of non-trivial updates was the ongoing effort to get the NrrdIO in ITK re-synched with Teem.  An issue here had been fixing the various warnings that showed up in Visual Studio, since even if no Teem users care about Visual Studio, lots of ITK users do.  I now have a 64-bit Windows VM that I can use for testing.  I've also recently found that the "gcc -Wconversion" flag enables the same warnings and more, and have started on the slow work of fixing these (at least those that show up in NrrdIO).  These warning have revealed many many places (usually in old code) where "int" was used for all array indexing, instead of unsigned int.

Comments and feedback welcome,
Gordon

[Teem-users] nrrdSlice (unu slice) and nrrdProject (unu project) on 1-D data

[Gordon L. K. <gl...@uc...>]

Hello all,

I finally got around to tweaking the implementation of nrrdSlice (called by unu slice) and nrrdProject (called by unu project) so that you can work on 1-D arrays.  The results in these cases are single-sample 1-D arrays, which is unusual, but more useful than generating an error in these cases.  Adding the special case of slicing/projecting a 1-D back to another 1-D array was a far less disruptive change than, say, allowing nrrds to have dimension 0.  Do an svn update on a Teem trunk checkout to see this change.

Examples with attached image:

# find value[0,200,120] 
unu slice -i skull.png -a 0 -p 0 | unu slice -a 0 -p 200 | unu slice -a 0 -p 120 | unu save -f text
178

# find average value of whole image
unu axmerge -i skull.png -a 0 1 | unu project -a 0 -m mean -t float | unu save -f text
52.15559

Prior to this update, you'd have to "unu axinsert -a 1 | " prior to the final slice or project, which got to be annoying after about a decade.

Behavior on higher-dimensional arrays should be completely unchanged.  Try it out and let me know if it works as expected.

Gordon