getdata-devel — General discussion of GetData development.

[getdata-devel] GetData-0.11.0 released

[D. V. W. <ge...@ke...>]

Hi! I'm probably as surprised as you are, but GetData-0.11.0 has
been released!

You can get it here:

  https://github.com/ketiltrout/getdata/releases/tag/v0.11.0

The primary reason for a new release is to fix a use-after-free
vulnerability assigned CVE-2021-20204.  All users are encouraged to
upgrade.

The security advisory is here:

  https://github.com/ketiltrout/getdata/security/advisories/GHSA-3m22-hpcj-m3pp

There are a few other bug fixes and one new function (gd_open_limit() which
you can use to try to avoid running out of file descriptors) in this
release, almost all of which have been sitting in the repository for
years and probably should have been released some time ago!


Where is GetData these days?
============================

Also, speaking of repositories, as you may have guessed from the above
URLs, I've moved getdata to GitHub:

  https://github.com/ketiltrout/getdata

I think it should be an improvement.



The future of this mailing list
===============================

Submissions to this list are almost entirely spam at this point, so if
you've attempted to contact me via this list in the past, and I've
ignored you, it's not because I didn't want to talk to you, it's because
your message got lost in the mess.  Apologies!

For the moment, I think it's best to think of this as an annoucement
list, though I'm not sure I'm convinced that's even worth it in the long
term.

For bug reports and other questions in general, I'd strongly suggest
using GitHub issues instead.  Similarly, to keep up to date with
GetData, it's almost certainly going to be better to watch the project
there than to monitor this list.


Cheers, and have fun!
=====================

It's been a fair bit of time since I've had to do this, so hopefully I
got it right.  If I haven't and this release doesn't work for you, make
an issue for it and we'll sort it out.

-don
-- 
D. V. Wiebe
ge...@ke...
https://github.com/ketiltrout/getdata

[getdata-devel] GetData vulnerability (CVE-2021-20204) and problems with Debian bugfix

[Graeme S. <gsm...@th...>]

Hi all,

I'm a long-time happy GetData user. Thanks for all your efforts 
producing a widely useful package.

I wanted to point out the following tale of woe, starting with the 
following use-after-free problem from June 2021:

https://nvd.nist.gov/vuln/detail/CVE-2021-20204

The CVE leads to responses from both Red Hat and Debian:

https://bugzilla.redhat.com/show_bug.cgi?id=1956348
https://lists.debian.org/debian-lts-announce/2021/05/msg00015.html

Debian produced the following patch:

https://salsa.debian.org/science-team/libgetdata/-/commit/61275e4c051090ce11467207eb361a6d81c405d9

...which, unfortunately, breaks much of libgetdata (as can be seen in 
the regression tests after the patch is applied.) There is an active bug 
report here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992437

This is where the trail gets cold. Debian's packaged libgetdata 0.10 is 
still broken, Red Hat doesn't seem to have patched the CVE, and there 
does not appear to be a CVE fix in the public getdata repositories.

Are getdata devs aware of the CVE, and/or formulating a response? The 
easiest way to break up the logjam would be to fix the CVE in 
libgetdata, and for Debian to remove their problematic patch when they 
update.

Thanks again for all your hard work - I am guessing maintaining this 
package is a volunteer position, and I'm grateful for whatever time you 
can put into it.

best,
Graeme

Re: [getdata-devel] Zipped Dirfile support patch

[Matthew P. <ge...@mp...>]

Another minor update. The previous version of the patch prematurely closed
file descriptors for the LUTs used for LINTERP fields, which sporadically
caused issues. A revised patch with this bug fixed is attached.

-Matthew Petroff

On Thu, Feb 6, 2020 at 11:52 AM Matthew Petroff <ge...@mp...>
wrote:

> A minor update. The previous version of the patch had a couple file
> descriptor leaks that have now been fixed. Checks with Valgrind show no
> remaining file descriptor leaks (and no memory leaks). A revised patch is
> attached.
>
> -Matthew Petroff
>
> On Thu, Jan 30, 2020 at 1:26 PM Matthew Petroff <ge...@mp...>
> wrote:
>
>> Dear all,
>>
>> Attached is a patch that allows for reading Dirfiles that are in an
>> uncompressed
>> Zip file. I'm sending it to this list now in case it is useful for anyone
>> else
>> (CLASS has recently started using it). It should apply cleanly to SVN
>> r1175.
>> Development of the patch was motivated by a need to reduce the total file
>> count
>> for FLAC-encoded Dirfiles, to alleviate the backup and data transfer
>> overheads
>> that result from having a very large number of small files.
>>
>> Below is some documentation (also included in the patch) for the
>> functionality:
>>
>> Separate from the Dirfile encoding scheme, GetData will read Dirfiles
>> contained
>> in uncompressed Zip files. This functionality is meant for reading
>> archival
>> data, so writing to these Zip files is not supported. Using the Info-ZIP
>> `zip`
>> utility, a Zip file can be created by running `zip -r0 ../dirfile.zip *`
>> from
>> within the root of an existing Dirfile. All encoding schemes are
>> supported by
>> this functionality except for the two encoding schemes that already use
>> Zip
>> files, *zzip* and *zzslim*. The encoding scheme must be specified using
>> the
>> /ENCODING directive, even if the Dirfile is unencoded. For /INCLUDE
>> directives
>> and LINTERP field look up table files, only relative paths are supported
>> and
>> only without `./` and `../` syntax.
>>
>> Although Zip files are most commonly created using _Deflate_ compression,
>> the
>> Zip standard (ISO/IEC 21320-1) also supports _Store_ compression, i.e., no
>> compression at all. GetData's Zip file support requires _Store_
>> compression for
>> all data files, although either _Store_ compression or _Deflate_
>> compression
>> can be used for any *format* files or any LINTERP field look up table
>> files.
>> With _Store_ compression, a Zip file effectively concatenates a Dirfile's
>> individual files together into a single file. Since a Zip file contains an
>> offset table, unlike a tarball, random reads are supported without the
>> need to load the entire file from disk.
>>
>> -Matthew Petroff
>>
>>
>>

Re: [getdata-devel] Zipped Dirfile support patch

[Matthew P. <ge...@mp...>]

A minor update. The previous version of the patch had a couple file
descriptor leaks that have now been fixed. Checks with Valgrind show no
remaining file descriptor leaks (and no memory leaks). A revised patch is
attached.

-Matthew Petroff

On Thu, Jan 30, 2020 at 1:26 PM Matthew Petroff <ge...@mp...>
wrote:

> Dear all,
>
> Attached is a patch that allows for reading Dirfiles that are in an
> uncompressed
> Zip file. I'm sending it to this list now in case it is useful for anyone
> else
> (CLASS has recently started using it). It should apply cleanly to SVN
> r1175.
> Development of the patch was motivated by a need to reduce the total file
> count
> for FLAC-encoded Dirfiles, to alleviate the backup and data transfer
> overheads
> that result from having a very large number of small files.
>
> Below is some documentation (also included in the patch) for the
> functionality:
>
> Separate from the Dirfile encoding scheme, GetData will read Dirfiles
> contained
> in uncompressed Zip files. This functionality is meant for reading archival
> data, so writing to these Zip files is not supported. Using the Info-ZIP
> `zip`
> utility, a Zip file can be created by running `zip -r0 ../dirfile.zip *`
> from
> within the root of an existing Dirfile. All encoding schemes are supported
> by
> this functionality except for the two encoding schemes that already use Zip
> files, *zzip* and *zzslim*. The encoding scheme must be specified using the
> /ENCODING directive, even if the Dirfile is unencoded. For /INCLUDE
> directives
> and LINTERP field look up table files, only relative paths are supported
> and
> only without `./` and `../` syntax.
>
> Although Zip files are most commonly created using _Deflate_ compression,
> the
> Zip standard (ISO/IEC 21320-1) also supports _Store_ compression, i.e., no
> compression at all. GetData's Zip file support requires _Store_
> compression for
> all data files, although either _Store_ compression or _Deflate_
> compression
> can be used for any *format* files or any LINTERP field look up table
> files.
> With _Store_ compression, a Zip file effectively concatenates a Dirfile's
> individual files together into a single file. Since a Zip file contains an
> offset table, unlike a tarball, random reads are supported without the
> need to load the entire file from disk.
>
> -Matthew Petroff
>
>
>

[getdata-devel] Zipped Dirfile support patch

[Matthew P. <ge...@mp...>]

Dear all,

Attached is a patch that allows for reading Dirfiles that are in an
uncompressed
Zip file. I'm sending it to this list now in case it is useful for anyone
else
(CLASS has recently started using it). It should apply cleanly to SVN r1175.
Development of the patch was motivated by a need to reduce the total file
count
for FLAC-encoded Dirfiles, to alleviate the backup and data transfer
overheads
that result from having a very large number of small files.

Below is some documentation (also included in the patch) for the
functionality:

Separate from the Dirfile encoding scheme, GetData will read Dirfiles
contained
in uncompressed Zip files. This functionality is meant for reading archival
data, so writing to these Zip files is not supported. Using the Info-ZIP
`zip`
utility, a Zip file can be created by running `zip -r0 ../dirfile.zip *`
from
within the root of an existing Dirfile. All encoding schemes are supported
by
this functionality except for the two encoding schemes that already use Zip
files, *zzip* and *zzslim*. The encoding scheme must be specified using the
/ENCODING directive, even if the Dirfile is unencoded. For /INCLUDE
directives
and LINTERP field look up table files, only relative paths are supported and
only without `./` and `../` syntax.

Although Zip files are most commonly created using _Deflate_ compression,
the
Zip standard (ISO/IEC 21320-1) also supports _Store_ compression, i.e., no
compression at all. GetData's Zip file support requires _Store_ compression
for
all data files, although either _Store_ compression or _Deflate_ compression
can be used for any *format* files or any LINTERP field look up table files.
With _Store_ compression, a Zip file effectively concatenates a Dirfile's
individual files together into a single file. Since a Zip file contains an
offset table, unlike a tarball, random reads are supported without the
need to load the entire file from disk.

-Matthew Petroff

[getdata-devel] Implicit index vector for MPLEX data streams?

[Graeme S. <gsm...@th...>]

Hi,

I am a long-time casual user of libgetdata/dirfile. It has been quite a 
while since I looked at either the library or the standard, and I am 
happy to see them both growing and maturing. Congratulations and thanks.

I have successfully shifted older code from a very large number of 
separate RAW files to a single MPLEX file. The results scale better 
(fewer gd_putdata calls, less stress on the VM, more striped I/O). All 
in all, it's been a pleasure to fix a long-standing problem with my code.

However, I now have to carry around an <index> vector for my MPLEX 
stream that is just a modulo-2^n counter. (That is, my MPLEX'd data 
stream is always ordered predictably, but I still need to tell 
libgetdata that.) This index counter takes up a significant amount of 
disk space, and I would prefer not to carry it around. Is it possible to 
synthesize this counter using some dirfile magic?

Here's a longer explanation in case it's not clear. My older format file 
looked like this:

foo_c1 RAW INT32 1
foo_c2 RAW INT32 1
[...]
foo_c256 RAW INT32 1

After multiplexing these data streams into a single file, I can use

foo_mplex RAW INT32 256
/HIDDEN foo_mplex

channel RAW UINT8 256
/HIDDEN channel

foo_c1 MPLEX foo_mplex channel 0 256
foo_c2 MPLEX foo_mplex channel 1 256
[...]
foo_c256 MPLEX foo_mplex channel 255 256

The only wrinkle is the "channel" file; even in UINT8s, it wastes a 
large fraction of the overall disk space.

I'm sure this is not an unique requirement and I thought I'd ask if I 
was missing something in the standards, or if I am approaching things 
from the wrong perspective.

thanks,
Graeme

Re: [getdata-devel] Make check failure: getdata-0.10.0

[D. V. W. <ge...@ke...>]

Hi Jeff,

Well, it certainly shouldn't be segfaulting, but I'm not sure if the bug
is in the test or the library, yet.

Disabing modules shouldn't make a difference.

What version of libpcre are you using?  Can you also send me your
config.log?

Cheers,
-don

>    Hello,
> 
>    I have successfully built getdata-0.10.0 with the new dynamic modules
>    behavior disabled (relevant library isn’t yet available on the shared
>    cluster I’m using).  The build proceeds fine, but make check has one
>    failure (copied below): a seg fault on match_pcre_js.
> 
>    Is this a known issue?  Any ideas?
> 
>    Thanks,
> 
>    Jeff
> 
>    PASS: match_pcre
> 
>    PASS: match_pcre_bad
> 
>    PASS: match_pcre_caseless
> 
>    PASS: match_pcre_ext
> 
>    libgetdata: Bad regular expression at offset 1: PCRE does not support
>    \L, \l, \N, \U, or \u
> 
>    n = 0 (expected 4)
> 
>    error = -24 (expected 0)
> 
>    entry_list = (nil) (expected non-NULL)
> 
>    /bin/sh: line 5: 19492 Segmentation fault      ${dir}$tst
> 
>    FAIL: match_pcre_js
> 
>    PASS: match_pcre_utf8
> 
>    PASS: match_regex
> 
>    PASS: match_regex_bad
> 
>    PASS: match_regex_ext
> 
>    PASS: match_regex_icase
> 
>    --------------------
>    Jeffrey Filippini
>    Assistant Professor of Physics and Astronomy
>    University of Illinois at Urbana-Champaign
>    405 Loomis Lab
>    1110 W Green St
>    Urbana, IL 61801
>    +1 (217) 244-4820
>    [1]jp...@il...
>    --------------------
> 
> References
> 
>    1. mailto:jp...@il...



> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Subject: confirm 5084172385121a18c5e81b861b18dc038748b06b
> Sender: get...@li...
> From: get...@li...
> Date: Tue, 12 Sep 2017 19:03:35 +0000
> Message-ID: <mai...@li...>
> 
> If you reply to this message, keeping the Subject: header intact,
> Mailman will discard the held message.  Do this if the message is
> spam.  If you reply to this message and include an Approved: header
> with the list password in it, the message will be approved for posting
> to the list.  The Approved: header can also appear in the first line
> of the body of the reply.


-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Make check failure: getdata-0.10.0

[Filippini, J. P <jp...@il...>]

Hello,

I have successfully built getdata-0.10.0 with the new dynamic modules behavior disabled (relevant library isn’t yet available on the shared cluster I’m using).  The build proceeds fine, but make check has one failure (copied below): a seg fault on match_pcre_js.  

Is this a known issue?  Any ideas?

Thanks,

Jeff

> PASS: match_pcre
> PASS: match_pcre_bad
> PASS: match_pcre_caseless
> PASS: match_pcre_ext
> libgetdata: Bad regular expression at offset 1: PCRE does not support \L, \l, \N, \U, or \u
> n = 0 (expected 4)
> error = -24 (expected 0)
> entry_list = (nil) (expected non-NULL)
> /bin/sh: line 5: 19492 Segmentation fault      ${dir}$tst
> FAIL: match_pcre_js
> PASS: match_pcre_utf8
> PASS: match_regex
> PASS: match_regex_bad
> PASS: match_regex_ext
> PASS: match_regex_icase

--------------------
Jeffrey Filippini
Assistant Professor of Physics and Astronomy
University of Illinois at Urbana-Champaign

405 Loomis Lab
1110 W Green St
Urbana, IL 61801
+1 (217) 244-4820
jp...@il... <mailto:jp...@il...>
--------------------

[getdata-devel] GetData tutorial

[W. M. <wad...@gm...>]

Hello,

I'm completely new to GetData and am very keen on using it. Can anyone
suggest an introductory guide or tutorial to help me get started? The
documentation at:

http://getdata.sourceforge.net/getdata.html

looks very complicated and I can't make sense of it. Any help will be
greatly appreciated.

Thanks in advance,
Wadud.

-- 
web: http://miahw.wordpress.com
mobile: +447905 755604
gpg: BDFB 2E29 B22F

[getdata-devel] GetData-0.10.0 released.

[D. V. W. <ge...@ke...>]

Hi all,

GetData-0.10.0 has been released.  You can get it from your local
SourceForge repository:

  https://sourceforge.net/projects/getdata/files/getdata/0.10.0/

This release introduces a new Dirfile Standards Version (also numbered
10), which adds three new field types (INDIR, SARRAY, SINDIR), and also
field namespaces.  This is the first update to the Dirfile Standards
since 2012.

GetData-0.10.0 also adds capability for regular expression matching
against the entry list and fixes bugs found since the release of
GetData-0.9.4.

Full release notes can be found at the bottom of the page linked above,
but downstream packagers may be most interested in the following:

 * In the standard autotools build system, encodings which use external
   libraries for compression (gzip, bzip2, flac, lzma, slim, zzip,
   zzslim) are now by default built as dynamically loaded modules. To
   recover the old build behaviour, which put everything into the core
   GetData library binary, pass --disable-modules to ./configure. Using
   modules introduces a runtime dependency on GNU libltdl.

Cheers and have fun,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] GetData-0.9.4 released

[D. V. W. <ge...@ke...>]

Hi all,

GetData-0.9.4 has been released to fix a few bugs found after the
release of 0.9.3.  Most notably, it fixes one bug that prevented the
compilation of the Python bindings under Python3.  It also fixes a
couple of bugs related to reading FLAC-compressed data, an a few other
things.   You can get it from your local SourceForge mirror, which also
has a complete list of changes:

  https://sourceforge.net/projects/getdata/files/getdata/0.9.4

Have fun,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] GetData-0.9.3 released.

[D. V. W. <ge...@ke...>]

Hi all,

GetData version 0.9.3 has been released.  You can get it from your local
SourceForge mirror:

  https://sourceforge.net/projects/getdata/files/getdata/0.9.3/

The primary enchancement for this release is added support in the
bindings for PHP version 7 and Python version 3.  This release also
fixes a few minor bugs found since the release of 0.9.2.

PHP 5 is still supported, as is Python 2 from version 2.4 onwards.
Support for Python 2.3 has been dropped.  Furthermore, Python 2 requires
Unicode support to be enabled.  The earliest supported Python 3 release
is version 3.2.

Another change of potential interest to downstream packagers:

  * The default install directory for the IDL, Perl, PHP, and Python
    bindings has changed: they are now all installed under ${exec_prefix}
    when possible.  The install directories can still be overridden by
    ./configure options.

Full release notes are provided at the bottom of the page linked above.
Release notes are also available in the file called NEWS in the GetData
source release.

Have fun,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] Patch for Python 3 Support

[D. V. W. <ge...@ke...>]

On Mon, Apr 04, 2016 at 10:22:05AM -0400, Matthew Petroff wrote:
> Attached is a patch to add Python 3 support to the Python bindings.
> Most of the changes are wrapped in preprocessor conditionals and are
> related to the int/long and str/unicode changes in Python 3 [1]. The
> only visible change for Python 2 is allowing Unicode strings to be
> accepted; this breaks support for versions <2.6. If desired, I could
> add additional preprocessor conditionals to avoid breaking support.
> Currently, bindings are only built for one Python version; it would be
> nice to automatically build bindings for both Python 2 and 3, but I'm
> not particularly familiar with Autotools. The patch has been tested
> with Python 2.7 and 3.4 on Linux Mint 17.3 (Ubuntu 14.04 base).
> 
> -Matthew Petroff

Thanks for the patch!

I finally got around to applying it, and it worked well, but then I
realised pygetdata couldn't automatically UTF-8 decode all the strings
returned by the C library because Dirfiles are vague about their
character encoding.  So, that left me with pygetdata reutning PyBytes
objects in Python3, which I didn't like either.  So I added the
capability in pygetdata for the caller to specify what character
encoding to use for strings, which I think is the best that can be
done.

So, you can do something like:

  import pygetdata
  pygetdata.character_encoding = 'utf-8'

and proceed.  If character_encoding is None (the default), the current
locale's default character encoding is used to encode Unicode strings
passed to the bindings, but no decoding of strings returned from the C
library is done.

I've committed the final result of this in SVN revision 1063, which adds
Python3 and PyUnicode support.  It should be released as 0.9.3 at some
point.

I've tested it a bit in Python2.4, 2.7, 3.2 (the earliest supported
Python3 version), and 3.4.  Python 2.3 support has been dropped due to
bugs in the 2.3 branch we're not interested in working around, but we're
still okay with 2.4 and 2.5, so long as Unicode support is enabled in
those versions.

It still needs a bit of polish.

Cheers,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Patch for Python 3 Support

[Matthew P. <ge...@mp...>]

Attached is a patch to add Python 3 support to the Python bindings.
Most of the changes are wrapped in preprocessor conditionals and are
related to the int/long and str/unicode changes in Python 3 [1]. The
only visible change for Python 2 is allowing Unicode strings to be
accepted; this breaks support for versions <2.6. If desired, I could
add additional preprocessor conditionals to avoid breaking support.
Currently, bindings are only built for one Python version; it would be
nice to automatically build bindings for both Python 2 and 3, but I'm
not particularly familiar with Autotools. The patch has been tested
with Python 2.7 and 3.4 on Linux Mint 17.3 (Ubuntu 14.04 base).

-Matthew Petroff

[1] https://docs.python.org/3/howto/cporting.html

[getdata-devel] GetData-0.9.2.1 released

[D. V. W. <ge...@ke...>]

Greetings,

To avoid obliging everyone who wants the test suite to work to manually
patch it to fix the bug reported earlier this week, I've released a new
version of GetData (0.9.2.1) with the fix applied:

https://sourceforge.net/projects/getdata/files/getdata/0.9.2.1/

All this new release does is fix that one bug in the test suite.  The
library and bindings are unchanged from the first 0.9.2 release, all
the way down to them reporting the same version number (i.e. 0.9.2, not
0.9.2.1).

If you were one of the lucky people who were not affected by the test
suite bug (either because you are on a platform that's not affected, or
else maybe because you don't care about the test suite), upgrading from
the first 0.9.2 release to this new one won't gain you anything.

Thanks to Dinar Valeev and Christian Trippe at openSUSE for the report
and verification of the fix.

Cheers,
-Don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] GetData-0.9.2 released.

[Christian T. <ct...@op...>]

Hi!

Am Donnerstag, 24. März 2016, 19:22:57 CEST schrieb D. V. Wiebe:
> On Thu, Mar 24, 2016 at 05:10:27PM -0700, D. V. Wiebe wrote:
> > > "Hi,
> > > according to our build system, the test which was broken on BE systems,
> > > now broken on intel 32bits.
> > > 
> > > [  902s] E.m[0] = -9.25596534238386e+61 (expected 3.1)
> > > [  902s] FAIL: alter_entry_scalar3r
> > > [  902s] PASS: alter_entry_scalar4
> > > 

> It's the test that's now broken.  The library is fine.  I've committed
> the following fix:
> 
> === CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ====
> --- test/alter_entry_scalar3r.c (revision 1053)
> +++ test/alter_entry_scalar3r.c (revision 1054)
> @@ -50,7 +50,7 @@
>    n = gd_entry(D, "data", &E);
> 
>    CHECKI(n,0);
> -  CHECKF(E.EN(lincom,m)[0], 3.1);
> +  CHECKF(E.EN(recip,dividend), 3.1);
>    CHECKP(E.scalar[0]);
>    gd_free_entry_strings(&E);
> 
> === CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ====
> 

I can confirm this fixes the problem.

Thanks
Christian

Re: [getdata-devel] GetData-0.9.2 released.

[D. V. W. <dv...@ke...>]

On Thu, Mar 24, 2016 at 05:10:27PM -0700, D. V. Wiebe wrote:
> > "Hi,
> > according to our build system, the test which was broken on BE systems,
> > now broken on intel 32bits.
> > 
> > [  902s] E.m[0] = -9.25596534238386e+61 (expected 3.1)
> > [  902s] FAIL: alter_entry_scalar3r
> > [  902s] PASS: alter_entry_scalar4
> > 
> > Build outlook you can see on OBS [1]
> 
> Thanks for the report.  I've been able to reproduce the problem here
> I'll take a look.
> 
> Thanks for the report,
> -don

It's the test that's now broken.  The library is fine.  I've committed
the following fix:

=== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ====
--- test/alter_entry_scalar3r.c (revision 1053)
+++ test/alter_entry_scalar3r.c (revision 1054)
@@ -50,7 +50,7 @@
   n = gd_entry(D, "data", &E);
 
   CHECKI(n,0);
-  CHECKF(E.EN(lincom,m)[0], 3.1);
+  CHECKF(E.EN(recip,dividend), 3.1);
   CHECKP(E.scalar[0]);
   gd_free_entry_strings(&E);
 
=== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ==== CUT ====

Cheers,
-don
-- 
Don Wiebe                                   dv...@ph...
Department of Physics and Astronomy
University of British Columbia              Hennings 204
6224 Agricultural Road                      Tele: +1-604-822-2585
University Endowment Lands, BC
Canada   V6T 1Z1                            http://ketiltrout.net/

Re: [getdata-devel] GetData-0.9.2 released.

[D. V. W. <ge...@ke...>]

On Thu, Mar 24, 2016 at 01:11:28PM +0100, Dinar Valeev wrote:
> Hi, seems my message is blocked to getdata-devel. Let me quote my reply here:

Apologies for that.  I've unstuck it.

> "Hi,
> according to our build system, the test which was broken on BE systems,
> now broken on intel 32bits.
> 
> [  902s] E.m[0] = -9.25596534238386e+61 (expected 3.1)
> [  902s] FAIL: alter_entry_scalar3r
> [  902s] PASS: alter_entry_scalar4
> 
> Build outlook you can see on OBS [1]

Thanks for the report.  I've been able to reproduce the problem here
I'll take a look.

Thanks for the report,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] GetData-0.9.2 released.

[Dinar V. <di...@gm...>]

On Thu, Mar 24, 2016 at 2:39 AM, D. V. Wiebe <ge...@ke...> wrote:
> Hi all,
>
> We've released GetData-0.9.2:
>
> https://sourceforge.net/projects/getdata/files/getdata/0.9.2/
>
> It fixes bugs found after the release of 0.9.1, including a segfault
> introduced in 0.9.1, plus bugs affecting big-endian systems.  Full
> release notes can be found at the link above.  Sincere thanks to
> Dan Horák for help with fixing the big-endian bugs.
Hi,
according to our build system, the test which was broken on BE systems,
now broken on intel 32bits.

[  902s] E.m[0] = -9.25596534238386e+61 (expected 3.1)
[  902s] FAIL: alter_entry_scalar3r
[  902s] PASS: alter_entry_scalar4

Build outlook you can see on OBS [1]

[1] https://build.opensuse.org/package/show/home:k0da:branches:KDE:Extra/getdata
>
> Share and enjoy,
> -don
> --
> D. V. Wiebe
> ge...@ke...
> http://getdata.sourceforge.net/
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
> _______________________________________________
> getdata-devel mailing list
> get...@li...
> https://lists.sourceforge.net/lists/listinfo/getdata-devel
>

[getdata-devel] GetData-0.9.2 released.

[D. V. W. <ge...@ke...>]

Hi all,

We've released GetData-0.9.2:

https://sourceforge.net/projects/getdata/files/getdata/0.9.2/

It fixes bugs found after the release of 0.9.1, including a segfault
introduced in 0.9.1, plus bugs affecting big-endian systems.  Full
release notes can be found at the link above.  Sincere thanks to
Dan Horák for help with fixing the big-endian bugs.

Share and enjoy,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] alter_entry_scalar3c test fails on big endian platforms

[Dan H. <da...@da...>]

On Fri, 4 Mar 2016 15:51:29 -0800
"D. V. Wiebe" <ge...@ke...> wrote:

> On Fri, Mar 04, 2016 at 11:29:06PM +0100, Dan Horák wrote:
> > this is captured build log including "make check" output -
> > http://fedora.danny.cz/tmp/.build-0.9.1-1.fc25.log.gz
> > 
> > If the code plays with bit-fields in C, then the bits are used
> > differently between little and big endian arches. You can find the
> > details in eg. PPC ELF ABI document [1].
> > 
> > 	With regards,
> > 
> > 		Dan
> 
> Thanks Dan,
> 
> that was helpful.  I think I've found the problem.  I wish it were
> something as interesting as bit twiddling, but it's just trying to

ok, just saw some "bit" handling in the failing test case :-)

> put a 64-bit integer into an int.  The bug seems to have been
> introduced in 0.6.0 in 2009; I'm surprised it wasn't found earlier.
> 
> Regardless, I've been working towards a 0.9.2 release.  I'll fix this,
> and there will likely be a release next week.

thanks


		Dan

Re: [getdata-devel] alter_entry_scalar3c test fails on big endian platforms

[D. V. W. <ge...@ke...>]

On Fri, Mar 04, 2016 at 11:29:06PM +0100, Dan Horák wrote:
> this is captured build log including "make check" output -
> http://fedora.danny.cz/tmp/.build-0.9.1-1.fc25.log.gz
> 
> If the code plays with bit-fields in C, then the bits are used
> differently between little and big endian arches. You can find the
> details in eg. PPC ELF ABI document [1].
> 
> 	With regards,
> 
> 		Dan

Thanks Dan,

that was helpful.  I think I've found the problem.  I wish it were
something as interesting as bit twiddling, but it's just trying to put a
64-bit integer into an int.  The bug seems to have been introduced in
0.6.0 in 2009; I'm surprised it wasn't found earlier.

Regardless, I've been working towards a 0.9.2 release.  I'll fix this,
and there will likely be a release next week.

Cheers,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] alter_entry_scalar3c test fails on big endian platforms

[Dan H. <da...@da...>]

Hi Don,

On Fri, 4 Mar 2016 13:42:25 -0800
"D. V. Wiebe" <ge...@ke...> wrote:

> On Fri, Mar 04, 2016 at 12:29:39PM +0100, Dan Horák wrote:
> > Hello,
> > 
> > when building getdata 0.9.0 and 0.9.1 on Fedora big endian platforms
> > (ppc64, s390, s390x), the alter_entry_scalar3c test fails.
> > 
> > [sharkcz@devel10 test]$ ./alter_entry_scalar3c 
> > E.bitnum = 0 (expected 3)
> > 
> > An example from our s390 buildsystem -
> > http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=2133825
> > 
> > If you need access to such platform or more information, please let
> > me know.
> > 
> > 
> > Thanks
> > 
> > 	Dan
> 
> Hi Dan,
> 
> Thanks for the report.  I'll see what I can figure out.
> 
> The most useful thing would be if you could turn on the debugging
> messages (./configure --enable-debug) and provide me the output of
> "make check" which will produce about 10MB of text.

this is captured build log including "make check" output -
http://fedora.danny.cz/tmp/.build-0.9.1-1.fc25.log.gz

If the code plays with bit-fields in C, then the bits are used
differently between little and big endian arches. You can find the
details in eg. PPC ELF ABI document [1].

[1]
http://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html#BITFIELD


	With regards,

		Dan

Re: [getdata-devel] alter_entry_scalar3c test fails on big endian platforms

[D. V. W. <ge...@ke...>]

On Fri, Mar 04, 2016 at 12:29:39PM +0100, Dan Horák wrote:
> Hello,
> 
> when building getdata 0.9.0 and 0.9.1 on Fedora big endian platforms
> (ppc64, s390, s390x), the alter_entry_scalar3c test fails.
> 
> [sharkcz@devel10 test]$ ./alter_entry_scalar3c 
> E.bitnum = 0 (expected 3)
> 
> An example from our s390 buildsystem -
> http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=2133825
> 
> If you need access to such platform or more information, please let me
> know.
> 
> 
> Thanks
> 
> 	Dan

Hi Dan,

Thanks for the report.  I'll see what I can figure out.

The most useful thing would be if you could turn on the debugging
messages (./configure --enable-debug) and provide me the output of "make
check" which will produce about 10MB of text.

Thanks,
-don
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] alter_entry_scalar3c test fails on big endian platforms

[Dan H. <da...@da...>]

Hello,

when building getdata 0.9.0 and 0.9.1 on Fedora big endian platforms
(ppc64, s390, s390x), the alter_entry_scalar3c test fails.

[sharkcz@devel10 test]$ ./alter_entry_scalar3c 
E.bitnum = 0 (expected 3)

An example from our s390 buildsystem -
http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=2133825

If you need access to such platform or more information, please let me
know.


	Thanks

		Dan