getdata-devel — General discussion of GetData development.

Re: [getdata-devel] getdata implementation questions

[Ross W. <ros...@gm...>]

Awesome thanks for that.

I have one more question.  I'm missing the reason why fragments are
useful - Is it just to split up a potentially large format file?

Cheers

Ross

On Thu, Mar 10, 2011 at 7:05 PM, D. V. Wiebe <ge...@ke...> wrote:
> On Thu, Mar 10, 2011 at 11:05:07AM -0600, Ross Williamson wrote:
>> Hi everyone
>>
>> So I'm kind of a novice at dirFiles and getdata so please excuse the
>> following if they are simple:
>>
>> 1) Say I had an array of temperatures - e.g. double lakeshore[8].  In
>> order to put each of those temperatures in the dirfile format would I
>> create 8 individual "fragments" (is that the right word?) for each one
>> using gd_add_raw() or is there a better way to do this?
>
> You'd create 8 individual fields, and probably store them in one
> framgent (which is just a fancy name for the text file containing the
> metadata for the fields):
>
>  char *name[8][12];
>  for (i = 0; i < 8; ++i) {
>    sprintf(name[i], "lakeshore%i", i);
>    gd_add_raw(dirfile, name[i], GD_FLOAT64, 0);
>  }
>
>
>> 2) Nested dirFiles - The example shows using nested dirFiles - How do
>> I initialize/put an entry (in C code) in the toplevel format file to
>> tell it about the subdirectory?
>
> Does the subdirfile already exist, or do you want to create it?  If it
> already exists, it's simple:
>
>  int new_fragment = gd_include(dirfile, "subdirfile/format", 0, 0);
>
> This adds an INCLUDE line in the top level format file (fragment index 0)
> referencing the subdirfile's format file (and, also loads whatever's in
> that subdirfile).  gd_include() returns the fragment index of the new
> fragment, which you'll want to remember, if you plan on doing anything
> with it from within your program.
>
> If the subdirfile doesn't exist yet, you first have to create it:
>
>  mkdir("subdirfile", 0666);
>
> and then create and add a new format file fragment:
>
>  int new_fragment = gd_include(dirfile, "subdirfile/format", 0,
>      GD_CREAT);
>
>
>> 3) Is there any way for dirFiles to follow each other? Consider a
>> situation where you have a dirFile directory that is about 2G and you
>> want to back it up to tape.  You close down that dirFile directory and
>> open up another one.  Is there anyway way if you open up the first
>> dirFile directory to tell it to follow into the second one once it
>> hits the end of the data?
>
> There isn't, although if it would be useful I could add it to the TODO
> as something to think about.  It's not immediately clear that the
> Dirfile Specification is completely compatible with such a thing.
>
> You can sort of fake it by using a FRAMEOFFSET directive (see
> gd_alter_frameoffset()) in the second dirfile to advance its
> frame numbers to past the end of the first.  Whatever's calling GetData
> would still have to do the work of stitching the two dirfiles together,
> though.
>
> Cheers,
> -dvw
> --
> D. V. Wiebe
> ge...@ke...
> http://getdata.sourceforge.net/
>



-- 
Ross Williamson
University of Chicago
Department of Astronomy & Astrophysics
773-834-9785 (office)
312-504-3051 (Cell)

Re: [getdata-devel] C examples

[Barth N. <net...@as...>]

This is a great help.  Putting it on the web page is a great idea!

cbn


On Thu, Mar 10, 2011 at 7:47 PM, D. V. Wiebe <ge...@ke...> wrote:
> On Tue, Mar 08, 2011 at 03:36:07PM -0600, Ross Williamson wrote:
>> Erm I'm dumb
>>
>> Is there any simple examples out there in creating a dirfile and
>> writing data to it.  I can do it Python but not C/C++
>>
>> Any help much appreciated
>>
>> Cheers
>>
>> Ross
>
> That's a good idea.  I should make something like that...
>
> There's an API reference here:
>
> http://getdata.sourceforge.net/getdata.html
>
> which provide an overview of the C functions available.  The
> nitty-gritty details of these functions are given in the UNIX manual
> pages installed with GetData.  (ie. "man gd_open".)
>
> But for now, does the following help?  It should maybe work:
>
> /* Link with -lm -lgetdata */
> #include <getdata.h>
> #include <math.h>
> #include <stdio.h>
>
> int main(void)
> {
>  /* create a new dirfile (or just open it if it already exists)
>   * - You can't create a new readonly dirfile (what would you do
>   *   with it?), so you should always specify GD_RDRW whith GD_CREAT
>   * - GD_UNENCODED tells GetData that you don't want to "encode",
>   *   (i.e. compress) the data stored in the dirfile.  Usually GetData
>   *   can figure out the encoding of a dirfile, but not a new one.
>   */
>  DIRFILE *D = gd_open("/tmp/dirfile-test", GD_RDWR | GD_CREAT |
>                      GD_UNENCODED);
>
>  /* The dirfile creates a single format file fragment, the main
>   * fragment which will be written as "/tmp/dirfile-test/format".
>   *
>   * It's referred to by fragment_index zero.  Let's add some fields
>   * to the dirfile.
>   */
>
>  /* This creates a new field called "data" holding raw data on disk.
>   * It will result in an empty file /tmp/dirfile-test/data1 being
>   * created.
>   * - GD_FLOAT64 is the data type (in this case double precision
>   *   numbers) of the data stored to disk.
>   * - Dirfiles are broken into "frames", which contain one or more
>   *   samples of each field in the database.  The "1" here tells
>   *   GetData that we have one sample of this per frame per field.
>   * - The final argument, 0, indicates that the metadata for this
>   *   field should be stored in the primary fragment (ie. the
>   *   file /tmp/dirfile-test/format).
>   */
>  gd_add_raw(D, "data1", GD_FLOAT64, 1, 0);
>
>  /* If you grok the GetData format specification syntax (it's just a
>   * text file, after all) you can add field specifictions directly.
>   * Here's the an second raw field, with the same parameters, also
>   * added to format file fragment #0
>   */
>  gd_add_spec(D, "data2 RAW FLOAT64 1", 0);
>
>  /* And, just because it's fun, how about we create a derived field
>   * (ie. a field *NOT* associated with a data file on disk.  This field
>   * will contain the sum of the first two fields we defined.
>   * - This creates a LINCOM field (a linear combination of one or more
>   *   input fields)
>   * - The third parameter ("2"), is the number of input fields, in our
>   *   case two.
>   * - The fourth parameter is the names of the input fields.
>   * - The fifth and sixth parameters are the gains and offsets
>   *   associated with the input fields.  Really what GetData will
>   *   compute when you read sum is:
>   *      sum[i] = data1[i] * gain[0] + offset[0] + data2[i] * gain[1] +
>   *               offset[1]
>   */
>  const char *field_names[] = { "data1", "data2" };
>  double gains[] = { 1, 1 };
>  double offsets[] = { 0, 0 };
>  gd_add_lincom(D, "sum", 2, field_names, gains, offsets, 0);
>
>  /* now let's "acquire" some data and write it to the dirfile. */
>  double data1[1000];
>  float data2[1000];
>  int i;
>
>  for (i = 0; i < 1000; ++i) {
>    data1[i] = sin(i / 1000.);
>    data2[i] = log(i);
>  }
>
>  /* Here's the first write.
>   * - The first two zeros tell GetData where to write: frame 0, sample 0.
>   * - The next zero and 1000 tell GetData the amount of data to write:
>   *   zero frames plus 1000 samples.  (Since there's only one sample
>   *   per frame, we could have also specified this as 1000 frames plus
>   *   zero samples.)
>   * - Then there's the data_type (GD_FLOAT64) of the array passed to
>   *   putdata, and then finally the data itself.
>   */
>  gd_putdata(D, "data1", 0, 0, 0, 1000, GD_FLOAT64, data1);
>
>  /* Here's the second write.  Almost identical.
>   * - The data type here though is GD_FLOAT32, to match the float array
>   *   data2 which we passed it.  GetData will take care of converting
>   *   the data from the input type (float), into the storage type
>   *   (double) which we specified when we created the field.
>   */
>  gd_putdata(D, "data2", 0, 0, 0, 1000, GD_FLOAT32, data2);
>
>  /* Now if we want, we can read back some data.  This fetches one
>   * sample of data, starting at frame 100. */
>  double datum;
>  gd_getdata(D, "data1", 100, 0, 0, 1, GD_FLOAT64, &datum);
>  printf("sin 0.1 = %g\n", datum);
>
>  /* We can also read the derived channel: */
>  gd_getdata(D, "sum", 100, 0, 0, 1, GD_FLOAT64, &datum);
>  printf("sin 0.1 + log(100) = %g\n", datum);
>
>  /* Depending on vagueries of the underlying filesystem backing
>   * this database, some of the data may have already been writen to
>   * disk.  None of the metadata has though.  We can force GetData to
>   * write all the remaining data and all the metadata to disk by
>   * performing an explicit flush (using gd_flush), but this will also
>   * occur when we close the dirfile at the end:
>   */
>   gd_close(D);
>
>   return 0;
> }
>
> --
> D. V. Wiebe
> ge...@ke...
> http://getdata.sourceforge.net/
>
> ------------------------------------------------------------------------------
> Colocation vs. Managed Hosting
> A question and answer guide to determining the best fit
> for your organization - today and in the future.
> http://p.sf.net/sfu/internap-sfd2d
> _______________________________________________
> getdata-devel mailing list
> get...@li...
> https://lists.sourceforge.net/lists/listinfo/getdata-devel
>
>



-- 
C. Barth Netterfield
University of Toronto
416-845-0946

Re: [getdata-devel] getdata implementation questions

[D. V. W. <ge...@ke...>]

On Thu, Mar 10, 2011 at 11:05:07AM -0600, Ross Williamson wrote:
> Hi everyone
> 
> So I'm kind of a novice at dirFiles and getdata so please excuse the
> following if they are simple:
> 
> 1) Say I had an array of temperatures - e.g. double lakeshore[8].  In
> order to put each of those temperatures in the dirfile format would I
> create 8 individual "fragments" (is that the right word?) for each one
> using gd_add_raw() or is there a better way to do this?

You'd create 8 individual fields, and probably store them in one
framgent (which is just a fancy name for the text file containing the
metadata for the fields):

  char *name[8][12];
  for (i = 0; i < 8; ++i) {
    sprintf(name[i], "lakeshore%i", i);
    gd_add_raw(dirfile, name[i], GD_FLOAT64, 0);
  }


> 2) Nested dirFiles - The example shows using nested dirFiles - How do
> I initialize/put an entry (in C code) in the toplevel format file to
> tell it about the subdirectory?

Does the subdirfile already exist, or do you want to create it?  If it
already exists, it's simple:

  int new_fragment = gd_include(dirfile, "subdirfile/format", 0, 0);

This adds an INCLUDE line in the top level format file (fragment index 0)
referencing the subdirfile's format file (and, also loads whatever's in
that subdirfile).  gd_include() returns the fragment index of the new
fragment, which you'll want to remember, if you plan on doing anything
with it from within your program.

If the subdirfile doesn't exist yet, you first have to create it:

  mkdir("subdirfile", 0666);

and then create and add a new format file fragment:

  int new_fragment = gd_include(dirfile, "subdirfile/format", 0,
      GD_CREAT);


> 3) Is there any way for dirFiles to follow each other? Consider a
> situation where you have a dirFile directory that is about 2G and you
> want to back it up to tape.  You close down that dirFile directory and
> open up another one.  Is there anyway way if you open up the first
> dirFile directory to tell it to follow into the second one once it
> hits the end of the data?

There isn't, although if it would be useful I could add it to the TODO
as something to think about.  It's not immediately clear that the
Dirfile Specification is completely compatible with such a thing.

You can sort of fake it by using a FRAMEOFFSET directive (see
gd_alter_frameoffset()) in the second dirfile to advance its
frame numbers to past the end of the first.  Whatever's calling GetData
would still have to do the work of stitching the two dirfiles together,
though.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] C examples

[D. V. W. <ge...@ke...>]

On Tue, Mar 08, 2011 at 03:36:07PM -0600, Ross Williamson wrote:
> Erm I'm dumb
> 
> Is there any simple examples out there in creating a dirfile and
> writing data to it.  I can do it Python but not C/C++
> 
> Any help much appreciated
> 
> Cheers
> 
> Ross

That's a good idea.  I should make something like that...

There's an API reference here:

http://getdata.sourceforge.net/getdata.html

which provide an overview of the C functions available.  The
nitty-gritty details of these functions are given in the UNIX manual
pages installed with GetData.  (ie. "man gd_open".)

But for now, does the following help?  It should maybe work:

/* Link with -lm -lgetdata */
#include <getdata.h>
#include <math.h>
#include <stdio.h>

int main(void)
{
  /* create a new dirfile (or just open it if it already exists)
   * - You can't create a new readonly dirfile (what would you do
   *   with it?), so you should always specify GD_RDRW whith GD_CREAT
   * - GD_UNENCODED tells GetData that you don't want to "encode",
   *   (i.e. compress) the data stored in the dirfile.  Usually GetData
   *   can figure out the encoding of a dirfile, but not a new one.
   */
  DIRFILE *D = gd_open("/tmp/dirfile-test", GD_RDWR | GD_CREAT |
                      GD_UNENCODED);

  /* The dirfile creates a single format file fragment, the main
   * fragment which will be written as "/tmp/dirfile-test/format".
   *
   * It's referred to by fragment_index zero.  Let's add some fields
   * to the dirfile.
   */

  /* This creates a new field called "data" holding raw data on disk.
   * It will result in an empty file /tmp/dirfile-test/data1 being
   * created.
   * - GD_FLOAT64 is the data type (in this case double precision
   *   numbers) of the data stored to disk.
   * - Dirfiles are broken into "frames", which contain one or more
   *   samples of each field in the database.  The "1" here tells
   *   GetData that we have one sample of this per frame per field.
   * - The final argument, 0, indicates that the metadata for this
   *   field should be stored in the primary fragment (ie. the
   *   file /tmp/dirfile-test/format).
   */
  gd_add_raw(D, "data1", GD_FLOAT64, 1, 0);

  /* If you grok the GetData format specification syntax (it's just a
   * text file, after all) you can add field specifictions directly.
   * Here's the an second raw field, with the same parameters, also
   * added to format file fragment #0
   */
  gd_add_spec(D, "data2 RAW FLOAT64 1", 0);

  /* And, just because it's fun, how about we create a derived field
   * (ie. a field *NOT* associated with a data file on disk.  This field
   * will contain the sum of the first two fields we defined.
   * - This creates a LINCOM field (a linear combination of one or more
   *   input fields)
   * - The third parameter ("2"), is the number of input fields, in our
   *   case two.
   * - The fourth parameter is the names of the input fields.
   * - The fifth and sixth parameters are the gains and offsets
   *   associated with the input fields.  Really what GetData will
   *   compute when you read sum is:
   *      sum[i] = data1[i] * gain[0] + offset[0] + data2[i] * gain[1] +
   *               offset[1] 
   */
  const char *field_names[] = { "data1", "data2" };
  double gains[] = { 1, 1 };
  double offsets[] = { 0, 0 };
  gd_add_lincom(D, "sum", 2, field_names, gains, offsets, 0);

  /* now let's "acquire" some data and write it to the dirfile. */
  double data1[1000];
  float data2[1000];
  int i;

  for (i = 0; i < 1000; ++i) {
    data1[i] = sin(i / 1000.);
    data2[i] = log(i);
  }

  /* Here's the first write.
   * - The first two zeros tell GetData where to write: frame 0, sample 0.
   * - The next zero and 1000 tell GetData the amount of data to write:
   *   zero frames plus 1000 samples.  (Since there's only one sample
   *   per frame, we could have also specified this as 1000 frames plus
   *   zero samples.)
   * - Then there's the data_type (GD_FLOAT64) of the array passed to
   *   putdata, and then finally the data itself.
   */
  gd_putdata(D, "data1", 0, 0, 0, 1000, GD_FLOAT64, data1);

  /* Here's the second write.  Almost identical.
   * - The data type here though is GD_FLOAT32, to match the float array
   *   data2 which we passed it.  GetData will take care of converting
   *   the data from the input type (float), into the storage type
   *   (double) which we specified when we created the field.
   */
  gd_putdata(D, "data2", 0, 0, 0, 1000, GD_FLOAT32, data2);

  /* Now if we want, we can read back some data.  This fetches one
   * sample of data, starting at frame 100. */
  double datum;
  gd_getdata(D, "data1", 100, 0, 0, 1, GD_FLOAT64, &datum);
  printf("sin 0.1 = %g\n", datum);

  /* We can also read the derived channel: */
  gd_getdata(D, "sum", 100, 0, 0, 1, GD_FLOAT64, &datum);
  printf("sin 0.1 + log(100) = %g\n", datum);

  /* Depending on vagueries of the underlying filesystem backing
   * this database, some of the data may have already been writen to
   * disk.  None of the metadata has though.  We can force GetData to
   * write all the remaining data and all the metadata to disk by
   * performing an explicit flush (using gd_flush), but this will also
   * occur when we close the dirfile at the end:
   */
   gd_close(D);

   return 0;
}

-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] getdata implementation questions

[Ross W. <ros...@gm...>]

Hi everyone

So I'm kind of a novice at dirFiles and getdata so please excuse the
following if they are simple:

1) Say I had an array of temperatures - e.g. double lakeshore[8].  In
order to put each of those temperatures in the dirfile format would I
create 8 individual "fragments" (is that the right word?) for each one
using gd_add_raw() or is there a better way to do this?

2) Nested dirFiles - The example shows using nested dirFiles - How do
I initialize/put an entry (in C code) in the toplevel format file to
tell it about the subdirectory?

3) Is there any way for dirFiles to follow each other? Consider a
situation where you have a dirFile directory that is about 2G and you
want to back it up to tape.  You close down that dirFile directory and
open up another one.  Is there anyway way if you open up the first
dirFile directory to tell it to follow into the second one once it
hits the end of the data?

Cheers,

Ross

-- 
Ross Williamson
University of Chicago
Department of Astronomy & Astrophysics
773-834-9785 (office)
312-504-3051 (Cell)

[getdata-devel] C examples

[Ross W. <ros...@gm...>]

Erm I'm dumb

Is there any simple examples out there in creating a dirfile and
writing data to it.  I can do it Python but not C/C++

Any help much appreciated

Cheers

Ross

-- 
Ross Williamson
University of Chicago
Department of Astronomy & Astrophysics
773-834-9785 (office)
312-504-3051 (Cell)

Re: [getdata-devel] Question about the the unversioned libraries contained in getdata

[Christian T. <ct...@gm...>]

Am Freitag, 25. Februar 2011, 21:44:02 schrieb D. V. Wiebe:
> On Thu, Feb 03, 2011 at 09:03:45PM -0800, D. V. Wiebe wrote:
> > On Thu, Jan 27, 2011 at 07:20:10PM -0500, Matthew D Truch wrote:
> > > On Thu, Jan 20, 2011 at 07:21:46PM +0100, Christian Trippe wrote:
> > > > I am trying to get getdata available for the next openSUSE
> > > > version. In my attempt to package it, I have some trouble with
> > > > the following libraries and would like to have your feedback:
> > > > 
> > > > /usr/lib/libgetdatabzip2-0.7.1.so
> > > > 
> > > >  /usr/lib/libgetdatagzip-0.7.1.so
> > > >  /usr/lib/libgetdatalzma-0.7.1.so
> > > > 
> > > > As openSUSE requests to package each (versioned) library in a
> > > > separate package, I asked for help  about these libs on the
> > > > openSUSE packaging mailinglist. There it was suggested that
> > > > these libs maybe plugins, which should then be installed into
> > > > an own subdir.
> > > 
> > > Yes, those libraries listed above are plugins to the getdata library
> > > (which provide the ability to read dirfiles compressed with bzip2,
> > > gzip, and lzma, respectively).
> > 
> > I see no problem moving them into a subdirectory, if that's what SuSE
> > would prefer, but I won't have time to do that until I get back to North
> > America (ie. end of the month-ish).
> > 
> > -dvw
> 
> The modules are now (in SVN) installed into $libdir/getdata by default.  Is
> that going to work for openSUSE?
> 
Thanks a lot that is great. 

openSUSE 11.4 will contain getdata. However there I had to make the workaround 
to package each of these plugins as a separate package as we are already 
behind RC2.

Christian

Re: [getdata-devel] Question about the the unversioned libraries contained in getdata

[Matthew D T. <ma...@tr...>]

> > > > As openSUSE requests to package each (versioned) library in a separate 
> > > > package, I asked for help  about these libs on the openSUSE packaging 
> > > > mailinglist. There it was suggested that these libs maybe plugins, which 
> > > > should then be installed into an own subdir.
> > > 
> > > Yes, those libraries listed above are plugins to the getdata library
> > > (which provide the ability to read dirfiles compressed with bzip2, gzip,
> > > and lzma, respectively).  
> > 
> > I see no problem moving them into a subdirectory, if that's what SuSE
> > would prefer, but I won't have time to do that until I get back to North
> > America (ie. end of the month-ish).
> > 
> > -dvw
> 
> The modules are now (in SVN) installed into $libdir/getdata by default.  Is that
> going to work for openSUSE?

FWIW, although Fedora Packaging doesn't require such a subdir, it is
considered "best practice", so totally fine by me.  

-- 
"Only two things are infinite, the universe, and human stupidity, and I'm not sure about the former -- Einstein."
--------------------------
Matthew Truch
Department of Physics and Astronomy
University of Pennsylvania
ma...@tr...
http://matt.truch.net/

Re: [getdata-devel] Question about the the unversioned libraries contained in getdata

[D. V. W. <ge...@ke...>]

On Thu, Feb 03, 2011 at 09:03:45PM -0800, D. V. Wiebe wrote:
> On Thu, Jan 27, 2011 at 07:20:10PM -0500, Matthew D Truch wrote:
> > On Thu, Jan 20, 2011 at 07:21:46PM +0100, Christian Trippe wrote:
> > > I am trying to get getdata available for the next openSUSE version. In my 
> > > attempt to package it, I have some trouble with the following libraries and 
> > > would like to have your feedback:
> > > 
> > > /usr/lib/libgetdatabzip2-0.7.1.so
> > >  /usr/lib/libgetdatagzip-0.7.1.so
> > >  /usr/lib/libgetdatalzma-0.7.1.so
> > > 
> > > As openSUSE requests to package each (versioned) library in a separate 
> > > package, I asked for help  about these libs on the openSUSE packaging 
> > > mailinglist. There it was suggested that these libs maybe plugins, which 
> > > should then be installed into an own subdir.
> > 
> > Yes, those libraries listed above are plugins to the getdata library
> > (which provide the ability to read dirfiles compressed with bzip2, gzip,
> > and lzma, respectively).  
> 
> I see no problem moving them into a subdirectory, if that's what SuSE
> would prefer, but I won't have time to do that until I get back to North
> America (ie. end of the month-ish).
> 
> -dvw

The modules are now (in SVN) installed into $libdir/getdata by default.  Is that
going to work for openSUSE?

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] --prefix under OSX

[Peter K. <syn...@gm...>]

When I use --prefix /opt/local under OSX
libgetdata++.dylib links against a libgetata
in /usr/local which does not exists:

otool -L /opt/local/libgetata++.dylib

I don't know where the user local comes from but in
binding/cxx/.lib/libgetdata++.lai I found two
/usr/local entries which lock responsible for the
wrong path.

Peter

Re: [getdata-devel] Question about the the unversioned libraries contained in getdata

[D. V. W. <ge...@ke...>]

On Thu, Jan 27, 2011 at 07:20:10PM -0500, Matthew D Truch wrote:
> On Thu, Jan 20, 2011 at 07:21:46PM +0100, Christian Trippe wrote:
> > I am trying to get getdata available for the next openSUSE version. In my 
> > attempt to package it, I have some trouble with the following libraries and 
> > would like to have your feedback:
> > 
> > /usr/lib/libgetdatabzip2-0.7.1.so
> >  /usr/lib/libgetdatagzip-0.7.1.so
> >  /usr/lib/libgetdatalzma-0.7.1.so
> > 
> > As openSUSE requests to package each (versioned) library in a separate 
> > package, I asked for help  about these libs on the openSUSE packaging 
> > mailinglist. There it was suggested that these libs maybe plugins, which 
> > should then be installed into an own subdir.
> 
> Yes, those libraries listed above are plugins to the getdata library
> (which provide the ability to read dirfiles compressed with bzip2, gzip,
> and lzma, respectively).  

I see no problem moving them into a subdirectory, if that's what SuSE
would prefer, but I won't have time to do that until I get back to North
America (ie. end of the month-ish).

-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] Question about the the unversioned libraries contained in getdata

[Matthew D T. <ma...@tr...>]

On Thu, Jan 20, 2011 at 07:21:46PM +0100, Christian Trippe wrote:
> I am trying to get getdata available for the next openSUSE version. In my 
> attempt to package it, I have some trouble with the following libraries and 
> would like to have your feedback:
> 
> /usr/lib/libgetdatabzip2-0.7.1.so
>  /usr/lib/libgetdatagzip-0.7.1.so
>  /usr/lib/libgetdatalzma-0.7.1.so
> 
> As openSUSE requests to package each (versioned) library in a separate 
> package, I asked for help  about these libs on the openSUSE packaging 
> mailinglist. There it was suggested that these libs maybe plugins, which 
> should then be installed into an own subdir.

Yes, those libraries listed above are plugins to the getdata library
(which provide the ability to read dirfiles compressed with bzip2, gzip,
and lzma, respectively).  

-- 
"One in every seven days is a Thursday."
--------------------------
Matthew Truch
Department of Physics and Astronomy
University of Pennsylvania
ma...@tr...
http://matt.truch.net/

[getdata-devel] Question about the the unversioned libraries contained in getdata

[Christian T. <ct...@gm...>]

Hi,

I am trying to get getdata available for the next openSUSE version. In my 
attempt to package it, I have some trouble with the following libraries and 
would like to have your feedback:

/usr/lib/libgetdatabzip2-0.7.1.so
 /usr/lib/libgetdatagzip-0.7.1.so
 /usr/lib/libgetdatalzma-0.7.1.so

As openSUSE requests to package each (versioned) library in a separate 
package, I asked for help  about these libs on the openSUSE packaging 
mailinglist. There it was suggested that these libs maybe plugins, which 
should then be installed into an own subdir.

You can find the start of the thread at
http://lists.opensuse.org/opensuse-packaging/2011-01/msg00095.html

So my question mainly is, is it correct that these libraries are plugins?

Thanks for your help
Christian

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

>>>> Now becoming more stable with the msvc support,
>>>> could you grant me write access? I would like
> 
> Done.

OK, I've added all mingw/msvc stuff to the getdata
repository and removed the cmake files from kst.

Peter

Re: [getdata-devel] msvc test results

[D. V. W. <ge...@ke...>]

On Thu, Dec 02, 2010 at 08:28:34PM +0100, Peter K?mmel wrote:
> Here are first test results, ctest output:
> 
> 99% tests passed, 7 tests failed out of 649
> 
> Total Test time (real) = 294.27 sec
> 
> The following tests FAILED:
>         339 - test_get_off64 (Failed)
>         428 - test_open_eaccess (Failed)
>         430 - test_open_notdirfile (Failed)
>         431 - test_parent (OTHER_FAULT)
>         565 - test_put_off64 (Failed)
>         577 - test_put_sbit (OTHER_FAULT)
>         620 - test_trunc_rofs (Failed)
> Errors while running CTest

Any indication why the other tests fail?  If the MSVCRT doesn't have
an off64_t type, (which I think is the case), you should just ignore
the {get|put}_off64 test.

Two of the failed tests: open_eaccess, trunc_rofs, rely on the
filesystem supporting chmod()'ing away write access.  However, in cases
where it doesn't, it should cause these tests to return 77 (indicating
the test was skipped).  open_notdirfile might fall into this category,
too.

> I assume OTHER_FAULT tests only fail because there is still a dirfile folder.

That's odd; the test should be deleting it after it's done, as all the
other tests do.

> Test which do not compile:
> 
> No suppoerted ATM with msvc:
> bzip_get bzip_get_get bzip_move_from bzip_nframes
> gzip_get gzip_get_get gzip_move_from gzip_nframes
> lzma_get lzma_nframes
> legacy_get legacy_get_put legacy_get_rofs legacy_nframes legacy_nonexistent legacy_put legacy_spf
> slim_get slim_nframes slim_put
> xz_get xz_nframes

Are you building the legacy API into the library (by defining
GD_LEGACY_API in config.h)?

The bindings tests presumably don't compile because you haven't
specified the location of the external compressor binaries (gzip, bzip2,
&c. via the preprocessor symbols GZIP, BZIP2, &c.)  That shouldn't be
fatal, but probably is in the current set up.

The way I had things set up, the autotools stuff basically just didn't
bother building these tests when support was missing.  I think this is
the wrong approach.  I've changed the Makefile to always build these
tests, but in cases where the test wouldn't compile normally, the
body of the test is replaced with a simple "return 77", which causes
the test to be skipped.  If you avoid defining TEST_GZIP, TEST_BZIP2,
&c. in config.h, these tests should be skipped, now.  Does that help?

Regardless, I think it improves things under *NIX, where which tests
were or were not built could get the test suite into trouble at times.
If it *does* help you, I can do the same thing for the legacy tests and
the Linux-specific LFS transitional API tests (get64 get_off64
nframes64 nframes_off64 put64 put_off64), which MSVC shouldn't care
about.  Then you could just build everything, and the tests themselves
would sort out whether they need to skip themselves or not.

[...]
> Index: test/test.h
> ===================================================================
> --- test/test.h	(Revision 492)
> +++ test/test.h	(Arbeitskopie)
> @@ -70,3 +70,10 @@
>      (unsigned long long)(n),(unsigned long long)(v))
>  #define CHECKXi(i,n,v) CHECKi(i,(n) != (v),n,"0x%" PRIX64,"0x%" PRIX64,\
>      (unsigned long long)(n),(unsigned long long)(v))
> +
> +
> +#ifdef _MSC_VER
> +#define mkdir(x,y) \
> +  system("rmdir /q/s "__TEST__"dirfile"); \
> +  _mkdir(x)
> +#endif

Since I was in the neighbourhood: Applied.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] MinGW patches

[D. V. W. <ge...@ke...>]

On Tue, Dec 07, 2010 at 10:53:45PM +0100, Peter K?mmel wrote:
> On 07.12.2010 04:16, D. V. Wiebe wrote:
> > On Fri, Dec 03, 2010 at 05:45:38PM +0100, Peter K?mmel wrote:
> >>>
> >>> How should we proceed with the new files?
> >>>
> >>
> >> Now becoming more stable with the msvc support,
> >> could you grant me write access? I would like
> >> to move the cmake build system from the kst svn
> >> into getdata/cmake. Where should I add the mingw
> >> implementations needed by msvc? In src/msvc?
> >>
> >> Peter

Done.

[...]
> Wait, I assume we could find a better solution than to switch to
> AC_CONFIG_FILES. The only problem is, that the macros GD_C89_API and
> GD_NO_C99_API are needed, so I could set GD_C89_API in the build system
> and add in getdata.h.in the needed defines:
> 
> #undef GD_NO_C99_API
> 
> #if defined(GD_NO_C99_API) && ! defined(GD_C89_API)
> # define GD_C89_API
> #endif
> 
> +#ifdef GD_C89_API
> +# define GD_NO_C99_API
> +#endif
> 
> It's a bit redundant, but much simpler than to change the whole configure process.

I'm okay with that.

> > Regarding the location of the msvc sources, I'd just stick them in src/.
> > Unless there's some particular reason to ghetto-ise them, I see no
> > particular benefit to putting them in a separate directory.
> 
> Good, but then I have to think about the duplicated headers (dirent.h and libgen.h)
> Having the headers in a separate folder makes it easy to exclude them when building with gcc/mingw.

I was wondering about that myself.  You could just rename them, but I'm
not opposed to a msvc directory, either.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

On 07.12.2010 04:16, D. V. Wiebe wrote:
> On Fri, Dec 03, 2010 at 05:45:38PM +0100, Peter K?mmel wrote:
>>>
>>> How should we proceed with the new files?
>>>
>>
>> Now becoming more stable with the msvc support,
>> could you grant me write access? I would like
>> to move the cmake build system from the kst svn
>> into getdata/cmake. Where should I add the mingw
>> implementations needed by msvc? In src/msvc?
>>
>> Peter
> 
> Apologies for the delay in response.  Just so I'm on the right page,
> your SF user name is syntheticpp, yes?

No problem, and yes: syntheticpp.

> 
> I've applied and committed most of the stuff you've sent me.  The

Thanks, also for the improvements of the patch.

> biggest exception was the getdata.h.in stuff.  The problem is, as things
> are now, autoconf builds getdata.h from getdata.h.in with
> AC_CONFIG_HEADERS rather than AC_CONFIG_FILES, which means it treats it
> like config.h.in (ie. it replaces #undef's with #define's as
> appropriate.)  For some reason I can't fathom, it doesn't to @-var
> substitutions in this mode.
> 
> I think I can convert it to a standard AC_CONFIG_FILES type input, which
> will involve converting all the #undef's to @-vars, which I take is
> easier for you to deal with.  It will require me to muck about with the
> configure and the Makefiles a bit.  (The original reason to make it a
> CONFIG_HEADER is that the Makefile rules are slightly more clever.
> Making getdata.h a simple CONFIG_FILE results in *everything* being
> rebuilt every time ./configure is run, even if nothing in getdata.h
> changed.)  I'll take care of it when I get a moment.

Wait, I assume we could find a better solution than to switch to
AC_CONFIG_FILES. The only problem is, that the macros GD_C89_API and
GD_NO_C99_API are needed, so I could set GD_C89_API in the build system
and add in getdata.h.in the needed defines:

#undef GD_NO_C99_API

#if defined(GD_NO_C99_API) && ! defined(GD_C89_API)
# define GD_C89_API
#endif

+#ifdef GD_C89_API
+# define GD_NO_C99_API
+#endif

It's a bit redundant, but much simpler than to change the whole configure process.



> 
> Regarding the location of the msvc sources, I'd just stick them in src/.
> Unless there's some particular reason to ghetto-ise them, I see no
> particular benefit to putting them in a separate directory.

Good, but then I have to think about the duplicated headers (dirent.h and libgen.h)
Having the headers in a separate folder makes it easy to exclude them when building with gcc/mingw.

> 
> In other news, I simply deleted all the unistd.h's from the test suite.
> (Changes like that can be made with sed, which makes it trivial.) unistd.h
> should be included by internal.h (when appropriate), anyways, so I don't
> think your stub header should be needed.

Yes I saw your commit log. I only added unistd.h because I thought it wouldn't
be a problem to remove all the includes automatically which I was too lazy to.

> 
> Also, regarding the stat64 stuff in internal.h, instead of the NO_STAT64
> &c. defines you added, doesn't it make more sense just to have struct
> stat and stat() as defaults?  That's what I've committed.  Let me know
> if it doesn't work.

Yes, also a good fix. But have to test it the next days.

Many thanks,
Peter

> 
> I haven't looked at the test suite stuff, yet.
> Cheers,
> -dvw

[getdata-devel] GetData 0.7.1 released.

[D. V. W. <ge...@ke...>]

GetData 0.7.1 has been released.  It may be downloaded from your local
SourceForge mirror:

http://sourceforge.net/projects/getdata/files/getdata/0.7.1/

GetData 0.7.1 is a bug-fix release to the recent GetData 0.7.0 release.
It fixes several memory managment issues (a few segmentation faults and
more memory leaks), as well as a few other bugs found after the release
of 0.7.0.

The release notes contain a complete list of the changes from the previous
release:

http://sourceforge.net/projects/getdata/files/getdata/0.7.1/README.txt/view

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

> 
> How should we proceed with the new files?
> 

Now becoming more stable with the msvc support,
could you grant me write access? I would like
to move the cmake build system from the kst svn
into getdata/cmake. Where should I add the mingw
implementations needed by msvc? In src/msvc?

Peter

[getdata-devel] msvc test results

[Peter K. <syn...@gm...>]

Here are first test results, ctest output:

99% tests passed, 7 tests failed out of 649

Total Test time (real) = 294.27 sec

The following tests FAILED:
        339 - test_get_off64 (Failed)
        428 - test_open_eaccess (Failed)
        430 - test_open_notdirfile (Failed)
        431 - test_parent (OTHER_FAULT)
        565 - test_put_off64 (Failed)
        577 - test_put_sbit (OTHER_FAULT)
        620 - test_trunc_rofs (Failed)
Errors while running CTest


I assume OTHER_FAULT tests only fail because there is still a dirfile folder.


Test which do not compile:

No suppoerted ATM with msvc:
bzip_get bzip_get_get bzip_move_from bzip_nframes
gzip_get gzip_get_get gzip_move_from gzip_nframes
lzma_get lzma_nframes
legacy_get legacy_get_put legacy_get_rofs legacy_nframes legacy_nonexistent legacy_put legacy_spf
slim_get slim_nframes slim_put
xz_get xz_nframes

Should be fixed:
endian_alter endian_get endian_move
get64 get_uint64 get_endian64 get_int64
include_cb include_pc
index_domain index_range
move_data_endian
nframes64
open_cb_abort open_cb_cont open_cb_ignore open_cb_invalid open_cb_rescan
put64 put_endian64
		

99% is really great, and 649 test is a good code coverage!

Attached the updated patch.

Cheers,
Peter

[getdata-devel] MSVC C++

[Peter K. <syn...@gm...>]

Here a patch for the C++ binding, all is obvious.

Peter

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

svn status:
M       src\flush.c
A       src\msvc
A       src\msvc\dirname.c
A       src\msvc\dirent.c
A       src\msvc\libgen.h
A       src\msvc\dirent.h
A       src\msvc\unistd.h
M       src\putdata.c
M       src\parse.c
M       src\mod.c
M       src\internal.h
M       src\getdata.h.in
M       src\raw.c

Attached a cleaned up patch for existing getdata files.
I think it could be committed, but you have to update the
automake process a bit because of HAVE_UNISTD_H, HAVE_STDINT_H,
and the GD_NO_C99_API macro.

How should we proceed with the new files?

(I've added a empty unistd.h file because all tests need it.)

Peter

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

> Thanks for the patch.  I've applied (although not committed) most of it,
> further comments below.


Now also most of the tests compile.
For mingw I get this result:
74% tests passed, 174 tests failed out of 664

but most of the msvc tests trigger an assert (hope always at the same line).


>> ..\..\getdata\src\getdata.c(151): error C2440: 'function' : cannot convert from 'double *' to '_complex'
>>
>>>
>>> not using msvc in ansi mode: there is already a typedef struct _complex, we must fix the complex macros.
>>>
> 
> Not quite sure I understand.  If you aren't using ANSI mode, but MSVC
> doesn't understand the _Complex keyword, what is the typedef struct
> _complex?  Can't you just enable ANSI mode, which will remove references
> to complex and _Complex from the source, and then it won't matter if
> MSVC has something called _complex?

math.h:

#ifndef _COMPLEX_DEFINED
struct _complex {
        double x,y; /* real and imaginary parts */ } ;

#if     !__STDC__ && !defined (__cplusplus)
/* Non-ANSI name for compatibility */
#define complex _complex
#endif
#define _COMPLEX_DEFINED
#endif


But above error could be fixed by including math.h first.


> 
> [...]
>> ..\..\getdata\src\open.c(51): error C2065: 'DIR' : undeclared identifier
> [...]
>> ..\..\getdata\src\open.c(173): error C2065: 'dir_error' : undeclared identifier
>>
>>>
>>> no dirent.h, could use kdewindows' implementation
>>>
> 
> That's odd; how does this work on MinGW then, which also uses the
> MSVCRT?  Do they implement dirent.h themselves?  Could we borrow that?
> AFAICT, the corresponding MSVCRT idiom is to use _findfirst, _findnext,
> _findclose.   
> 
> I'm rather leery of inserting a kdewindows dependency, even though I'm
> sure it would be a robust way to get around this.

I've added cygwin/mingw files for fixing such errors (dirent.h, dirname)

> 
> [...]
>> ..\..\getdata\src\putdata.c(532): error C2036: 'void *' : unknown size
>>
>>>
>>> seems void* is not a synonym for a pointer, could it be replaced by a typedef?
> 
> Clearly my fault: I should be casting that void pointer to a char* before
> performing math with it.

Yes, that fixes it.

> 
> [...]
>> Index: src/open.c
>> ===================================================================
>> --- src/open.c	(Revision 489)
>> +++ src/open.c	(Arbeitskopie)
>> @@ -39,10 +39,13 @@
>>  #include <libgen.h>
>>  #endif
>>  
>> +
>> +
>>  /* attempt to open or create a new dirfile - set error appropriately */
>>  static FILE* _GD_CreateDirfile(DIRFILE* D, const char* format_file,
>>      const char* filedir)
>>  {
>> +  FILE* fp = NULL;
>>    struct stat statbuf;
>>    char fullname[FILENAME_MAX];
>>    DIR* dir;
>> @@ -50,7 +53,6 @@
>>    struct dirent* lamb;
>>    int dir_error = 0;
>>    int format_error = 0;
>> -  FILE* fp = NULL;
>>  
>>    dtrace("%p, \"%s\", \"%s\"", D, format_file, filedir);
> 
> This doesn't appear to do anything useful.  Can you confirm.

Yes, don't need it any more.

> 
>> Index: src/flush.c
>> ===================================================================
>> --- src/flush.c	(Revision 489)
>> +++ src/flush.c	(Arbeitskopie)
> [...]
>> Index: src/ascii.c
>> ===================================================================
>> --- src/ascii.c	(Revision 489)
>> +++ src/ascii.c	(Arbeitskopie)
> [...]
> 
> Both applied.  (I don't know why I thought unistd.h was a C Standard
> header.  Given that UNI[X] appears right in the name.)
> 
>> Index: src/internal.h
>> ===================================================================
>> --- src/internal.h	(Revision 489)
>> +++ src/internal.h	(Arbeitskopie)
>> @@ -27,18 +27,32 @@
> [...]
>>  #include <string.h>
>> +#include <errno.h>
>> +#include <fcntl.h>
>>  
>>  #ifdef __APPLE__
>>  typedef off_t off64_t;
>>  #endif
>>  
>> +#ifdef _MSC_VER
>> +typedef size_t ssize_t;
>> +typedef int mode_t;
>> +#define fsync _commit
>> +#define snprintf _snprintf
>> +#endif
> 
> Applied.
> 
>>  #ifdef GD_NO_C99_API
>>  #  define GD_DCOMPLEXP_t double *
>>  #  define GD_DCOMPLEXA(v) double v[2]
>>  #  define GD_DCOMPLEXV(v) double v[][2]
>> +#ifndef _MSC_VER 
>>  #  define cabs(z)  sqrt((z)[0] * (z)[0] + (z)[1] * (z)[1])
>>  #  define carg(z)  atan2((z)[1], (z)[0])
>> +#endif
>>  #  define creal(z) ((z)[0])
>>  #  define cimag(z) ((z)[1])
>>  #  define _gd_a2c _gd_c2c
>> @@ -86,6 +100,7 @@
>>  #endif
>>  #endif
> 
> I don't understand.  MSVC doesn't define creal and cimag, but they
> do have cabs and carg?

When math.h is inculded first, this is not needed any more.


> 
> [...]
>>  
>> +#ifdef _MSC_VER
>> +#define __gd_unused
>> +#else
>>  #define __gd_unused __attribute__ (( unused ))
>> +#endif
>>  
> 
> Yeah, I should have known better.  Applied.
> 
> [...]
>>  
>> -#if defined __CYGWIN__ || defined __APPLE__
>> +#if defined __CYGWIN__ || defined __APPLE__  || _MSC_VER
>>  #  define gd_stat64 stat
>>  #elif HAVE_STAT64
>>  #  define gd_stat64 stat64
>> @@ -265,7 +284,7 @@
>>  typedef struct stat64 gd_stat64_t;
>>  #elif HAVE_STRUCT___STAT64
>>  typedef struct __stat64 gd_stat64_t;
>> -#elif defined __CYGWIN__ || defined __APPLE__
>> +#elif defined __CYGWIN__ || defined __APPLE__ || _MSC_VER
>>  typedef struct stat gd_stat64_t;
>>  #endif
> 
> Applied, although doesn't the MSVCRT contain _stat64 and struct __stat64?
> Couldn't you just define HAVE_STRUCT___STAT64 and HAVE__STAT64 to use
> that instead?

there is a _stat64 and struct _stat64. I've introduced HAVE_STRUCT__STAT64,
see patch.

> 
> [...]
>>  int _GD_SlimClose(struct _gd_raw_file* file);
>>  off64_t _GD_SlimSize(struct _gd_raw_file* file, gd_type_t data_type);
>>  
>> -static inline int entry_cmp(const void *a, const void *b)
>> +#ifdef _MSC_VER
>> +#define _gd_static_inline static
>> +#else
>> +#define _gd_static_inline static inline
>> +#endif
>> +
>> +_gd_static_inline int entry_cmp(const void *a, const void *b)
>>  {
>>    return strcmp((*(gd_entry_t**)a)->field, (*(gd_entry_t**)b)->field);
> 
> Happily applied, although I'm curious: does MSVC not recognise inline,
> or does "static inline" do something different in it?

msvc doesn't like the inline.

> 
>> Index: src/getdata.h.in
>> ===================================================================
>> --- src/getdata.h.in	(Revision 489)
>> +++ src/getdata.h.in	(Arbeitskopie)
>> @@ -56,10 +56,18 @@
>>   *    size_t, off_t, __REDIRECT, &c., if present
>>   */
>>  #include <sys/types.h>
>> +
>> +#ifdef HAVE_UNISTD_H
>>  #include <unistd.h>
>> +#endif
>>  
>> +#ifdef HAVE_STDINT_H
>> +#include <stdint.h>
>> +#endif
> 
> This is a problem.  getdata.h is a public header, and doesn't know
> anything about what goes on in the internal config.h.  So, I've
> basically put this code in internal.h.  I'll have to figure out what I
> wanted from unistd.h and either replace it with a C Standard header, or
> else wrap it in an appropriate define (I suppose !defined(_MSC_VER)
> would work, but I'd like to figure out why unistd is there in the first
> place.)
> 
>>  /* If this symbol is defined here, the C99-API is not present in the library */
>> -#undef GD_NO_C99_API
>> +// TODO check for C99 #undef GD_NO_C99_API
> 
> Not sure what the intent is here.  In the autoconf system, configure is
> the thing that figures out whether we're compiling in ANSI mode or not.
> If we are, during the getdat.h.in -> getdata.h conversion, it defines
> GD_NO_C99_API here, to make sure applications can't access the
> non-existant C99 API.  In non-ANSI mode, this gets left undefined by
> configure.

At the moment I only take a renamed copy of getdata.h.in, so it is not
possible define GD_NO_C99_API when using msvc. Is there a other way to mark
GD_NO_C99_API as used by automake. Cmake could replace things like  $GD_NO_C99_API
or @GD_NO_C99_API@.

#if @GD_ENABLE_C99_API@
#undef GD_NO_C99_API
#endif

And why should it be necessary to undef GD_NO_C99_API at this place?

> 
>>  #if defined(GD_NO_C99_API) && ! defined(GD_C89_API)
>>  # define GD_C89_API
>> @@ -225,9 +233,15 @@
>>  /* GD_NO_ENTRY is not part of this count */
>>  #define GD_N_ENTYPES 14
>>  
>> +#ifdef _MSC_VER // TODO
>> +typedef short _gd_int16_t;
>> +typedef unsigned short _gd_uint16_t;
>> +typedef __int64 _gd_int64_t;
>> +#else
>>  #undef _gd_int16_t
>>  #undef _gd_uint16_t
>>  #undef _gd_int64_t
>> +#endif
> 
> Applied.
> 
> [...]
>> Index: src/raw.c
>> ===================================================================
>> --- src/raw.c	(Revision 489)
>> +++ src/raw.c	(Arbeitskopie)
> [...]
> 
> Applied.
> 
> Cheers,
> -dvw

I'll have a look at may current changes again.

Peter

Re: [getdata-devel] MinGW patches

[D. V. W. <ge...@ke...>]

On Tue, Nov 30, 2010 at 06:26:49PM +0100, Peter K?mmel wrote:
> 
> > I've committed the re-arrangement to trunk, and it'll be in the 0.7.1
> > bug-fix release which I'm hoping to put out at the end of the week.  Please
> > give it a whirl when you have a chance and let me know what MSVC thinks
> > of it.
> 
> Great, most files compile now (see comments in the msvc output).
> Attached a patch with some small fixes (not mentioned for committing).
> 
> Peter

Thanks for the patch.  I've applied (although not committed) most of it,
further comments below.

[...]
> ..\..\getdata\src\getdata.c(121): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(121): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(122): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(122): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(123): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(123): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(124): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(124): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(125): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(125): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(126): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(126): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(127): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(127): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(128): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(128): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(129): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(129): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(130): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(130): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(150): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(150): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> ..\..\getdata\src\getdata.c(151): error C2440: 'function' : cannot convert from 'float *' to '_complex'
> ..\..\getdata\src\getdata.c(151): error C2440: 'function' : cannot convert from 'double *' to '_complex'
> 
> >
> > not using msvc in ansi mode: there is already a typedef struct _complex, we must fix the complex macros.
> >

Not quite sure I understand.  If you aren't using ANSI mode, but MSVC
doesn't understand the _Complex keyword, what is the typedef struct
_complex?  Can't you just enable ANSI mode, which will remove references
to complex and _Complex from the source, and then it won't matter if
MSVC has something called _complex?

[...]
> ..\..\getdata\src\open.c(51): error C2065: 'DIR' : undeclared identifier
[...]
> ..\..\getdata\src\open.c(173): error C2065: 'dir_error' : undeclared identifier
> 
> >
> > no dirent.h, could use kdewindows' implementation
> >

That's odd; how does this work on MinGW then, which also uses the
MSVCRT?  Do they implement dirent.h themselves?  Could we borrow that?
AFAICT, the corresponding MSVCRT idiom is to use _findfirst, _findnext,
_findclose.   

I'm rather leery of inserting a kdewindows dependency, even though I'm
sure it would be a robust way to get around this.

[...]
> ..\..\getdata\src\putdata.c(532): error C2036: 'void *' : unknown size
> 
> >
> > seems void* is not a synonym for a pointer, could it be replaced by a typedef?

Clearly my fault: I should be casting that void pointer to a char* before
performing math with it.

[...]
> Index: src/open.c
> ===================================================================
> --- src/open.c	(Revision 489)
> +++ src/open.c	(Arbeitskopie)
> @@ -39,10 +39,13 @@
>  #include <libgen.h>
>  #endif
>  
> +
> +
>  /* attempt to open or create a new dirfile - set error appropriately */
>  static FILE* _GD_CreateDirfile(DIRFILE* D, const char* format_file,
>      const char* filedir)
>  {
> +  FILE* fp = NULL;
>    struct stat statbuf;
>    char fullname[FILENAME_MAX];
>    DIR* dir;
> @@ -50,7 +53,6 @@
>    struct dirent* lamb;
>    int dir_error = 0;
>    int format_error = 0;
> -  FILE* fp = NULL;
>  
>    dtrace("%p, \"%s\", \"%s\"", D, format_file, filedir);

This doesn't appear to do anything useful.  Can you confirm.

> Index: src/flush.c
> ===================================================================
> --- src/flush.c	(Revision 489)
> +++ src/flush.c	(Arbeitskopie)
[...]
> Index: src/ascii.c
> ===================================================================
> --- src/ascii.c	(Revision 489)
> +++ src/ascii.c	(Arbeitskopie)
[...]

Both applied.  (I don't know why I thought unistd.h was a C Standard
header.  Given that UNI[X] appears right in the name.)

> Index: src/internal.h
> ===================================================================
> --- src/internal.h	(Revision 489)
> +++ src/internal.h	(Arbeitskopie)
> @@ -27,18 +27,32 @@
[...]
>  #include <string.h>
> +#include <errno.h>
> +#include <fcntl.h>
>  
>  #ifdef __APPLE__
>  typedef off_t off64_t;
>  #endif
>  
> +#ifdef _MSC_VER
> +typedef size_t ssize_t;
> +typedef int mode_t;
> +#define fsync _commit
> +#define snprintf _snprintf
> +#endif

Applied.

>  #ifdef GD_NO_C99_API
>  #  define GD_DCOMPLEXP_t double *
>  #  define GD_DCOMPLEXA(v) double v[2]
>  #  define GD_DCOMPLEXV(v) double v[][2]
> +#ifndef _MSC_VER 
>  #  define cabs(z)  sqrt((z)[0] * (z)[0] + (z)[1] * (z)[1])
>  #  define carg(z)  atan2((z)[1], (z)[0])
> +#endif
>  #  define creal(z) ((z)[0])
>  #  define cimag(z) ((z)[1])
>  #  define _gd_a2c _gd_c2c
> @@ -86,6 +100,7 @@
>  #endif
>  #endif

I don't understand.  MSVC doesn't define creal and cimag, but they
do have cabs and carg?

[...]
>  
> +#ifdef _MSC_VER
> +#define __gd_unused
> +#else
>  #define __gd_unused __attribute__ (( unused ))
> +#endif
>  

Yeah, I should have known better.  Applied.

[...]
>  
> -#if defined __CYGWIN__ || defined __APPLE__
> +#if defined __CYGWIN__ || defined __APPLE__  || _MSC_VER
>  #  define gd_stat64 stat
>  #elif HAVE_STAT64
>  #  define gd_stat64 stat64
> @@ -265,7 +284,7 @@
>  typedef struct stat64 gd_stat64_t;
>  #elif HAVE_STRUCT___STAT64
>  typedef struct __stat64 gd_stat64_t;
> -#elif defined __CYGWIN__ || defined __APPLE__
> +#elif defined __CYGWIN__ || defined __APPLE__ || _MSC_VER
>  typedef struct stat gd_stat64_t;
>  #endif

Applied, although doesn't the MSVCRT contain _stat64 and struct __stat64?
Couldn't you just define HAVE_STRUCT___STAT64 and HAVE__STAT64 to use
that instead?

[...]
>  int _GD_SlimClose(struct _gd_raw_file* file);
>  off64_t _GD_SlimSize(struct _gd_raw_file* file, gd_type_t data_type);
>  
> -static inline int entry_cmp(const void *a, const void *b)
> +#ifdef _MSC_VER
> +#define _gd_static_inline static
> +#else
> +#define _gd_static_inline static inline
> +#endif
> +
> +_gd_static_inline int entry_cmp(const void *a, const void *b)
>  {
>    return strcmp((*(gd_entry_t**)a)->field, (*(gd_entry_t**)b)->field);

Happily applied, although I'm curious: does MSVC not recognise inline,
or does "static inline" do something different in it?

> Index: src/getdata.h.in
> ===================================================================
> --- src/getdata.h.in	(Revision 489)
> +++ src/getdata.h.in	(Arbeitskopie)
> @@ -56,10 +56,18 @@
>   *    size_t, off_t, __REDIRECT, &c., if present
>   */
>  #include <sys/types.h>
> +
> +#ifdef HAVE_UNISTD_H
>  #include <unistd.h>
> +#endif
>  
> +#ifdef HAVE_STDINT_H
> +#include <stdint.h>
> +#endif

This is a problem.  getdata.h is a public header, and doesn't know
anything about what goes on in the internal config.h.  So, I've
basically put this code in internal.h.  I'll have to figure out what I
wanted from unistd.h and either replace it with a C Standard header, or
else wrap it in an appropriate define (I suppose !defined(_MSC_VER)
would work, but I'd like to figure out why unistd is there in the first
place.)

>  /* If this symbol is defined here, the C99-API is not present in the library */
> -#undef GD_NO_C99_API
> +// TODO check for C99 #undef GD_NO_C99_API

Not sure what the intent is here.  In the autoconf system, configure is
the thing that figures out whether we're compiling in ANSI mode or not.
If we are, during the getdat.h.in -> getdata.h conversion, it defines
GD_NO_C99_API here, to make sure applications can't access the
non-existant C99 API.  In non-ANSI mode, this gets left undefined by
configure.

>  #if defined(GD_NO_C99_API) && ! defined(GD_C89_API)
>  # define GD_C89_API
> @@ -225,9 +233,15 @@
>  /* GD_NO_ENTRY is not part of this count */
>  #define GD_N_ENTYPES 14
>  
> +#ifdef _MSC_VER // TODO
> +typedef short _gd_int16_t;
> +typedef unsigned short _gd_uint16_t;
> +typedef __int64 _gd_int64_t;
> +#else
>  #undef _gd_int16_t
>  #undef _gd_uint16_t
>  #undef _gd_int64_t
> +#endif

Applied.

[...]
> Index: src/raw.c
> ===================================================================
> --- src/raw.c	(Revision 489)
> +++ src/raw.c	(Arbeitskopie)
[...]

Applied.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] MinGW patches

[Peter K. <syn...@gm...>]

> I've committed the re-arrangement to trunk, and it'll be in the 0.7.1
> bug-fix release which I'm hoping to put out at the end of the week.  Please
> give it a whirl when you have a chance and let me know what MSVC thinks
> of it.

Great, most files compile now (see comments in the msvc output).
Attached a patch with some small fixes (not mentioned for committing).

Peter




------ Build started: Project: ZERO_CHECK, Configuration: Debug Win32 ------
  Checking Build System
  CMake does not need to re-run because Q:/getdata/_zehner/CMakeFiles/generate.stamp is up-to-date.
  CMake does not need to re-run because Q:/getdata/_zehner/src/CMakeFiles/generate.stamp is up-to-date.
------ Build started: Project: getdata, Configuration: Debug Win32 ------
  Building Custom Rule Q:/kst/kst/misc/getdata-windows/cmake/src/CMakeLists.txt
  CMake does not need to re-run because Q:\getdata\_zehner\src\CMakeFiles\generate.stamp is up-to-date.
  add.c
  ascii.c
  close.c
  common.c
  compat.c
  constant.c
  del.c
  encoding.c
  endian.c
  entry.c
  field_list.c
  flimits.c
  flush.c
  fragment.c
  getdata.c
..\..\getdata\src\getdata.c(121): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(121): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(122): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(122): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(123): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(123): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(124): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(124): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(125): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(125): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(126): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(126): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(127): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(127): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(128): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(128): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(129): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(129): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(130): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(130): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(150): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(150): error C2440: 'function' : cannot convert from 'double *' to '_complex'
..\..\getdata\src\getdata.c(151): error C2440: 'function' : cannot convert from 'float *' to '_complex'
..\..\getdata\src\getdata.c(151): error C2440: 'function' : cannot convert from 'double *' to '_complex'

>
> not using msvc in ansi mode: there is already a typedef struct _complex, we must fix the complex macros.
>

  globals.c
  gzip.c
  include.c
  index.c
  meta_list.c
  Generating Code...
  Compiling...
  move.c
  name.c
  native.c
  nfields.c
  nframes.c
  nmeta.c
  open.c
..\..\getdata\src\open.c(51): error C2065: 'DIR' : undeclared identifier
..\..\getdata\src\open.c(51): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(52): error C2143: syntax error : missing ';' before 'type'
..\..\getdata\src\open.c(53): error C2143: syntax error : missing ';' before 'type'
..\..\getdata\src\open.c(54): error C2143: syntax error : missing ';' before 'type'
..\..\getdata\src\open.c(55): error C2143: syntax error : missing ';' before 'type'
..\..\getdata\src\open.c(61): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(65): error C2065: 'dir_error' : undeclared identifier
..\..\getdata\src\open.c(67): error C2065: 'dir_error' : undeclared identifier
..\..\getdata\src\open.c(73): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(73): error C2065: 'dir_error' : undeclared identifier
..\..\getdata\src\open.c(81): error C2065: 'dir_error' : undeclared identifier
..\..\getdata\src\open.c(81): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(88): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(89): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(96): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(113): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(125): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(132): error C2065: 'dirfile_end' : undeclared identifier
..\..\getdata\src\open.c(133): error C2065: 'dirfile_end' : undeclared identifier
..\..\getdata\src\open.c(133): error C2100: illegal indirection
..\..\getdata\src\open.c(135): error C2065: 'dirfile_end' : undeclared identifier
..\..\getdata\src\open.c(138): error C2065: 'lamb' : undeclared identifier
..\..\getdata\src\open.c(138): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(139): error C2065: 'dirfile_end' : undeclared identifier
..\..\getdata\src\open.c(139): error C2065: 'lamb' : undeclared identifier
..\..\getdata\src\open.c(139): error C2223: left of '->d_name' must point to struct/union
..\..\getdata\src\open.c(139): error C2198: 'strcpy' : too few arguments for call
..\..\getdata\src\open.c(143): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(152): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(159): error C2065: 'dir' : undeclared identifier
..\..\getdata\src\open.c(163): error C2065: 'format_error' : undeclared identifier
..\..\getdata\src\open.c(173): error C2065: 'dir_error' : undeclared identifier

>
> no dirent.h, could use kdewindows' implementation
>

  parse.c
  protect.c
  putdata.c
..\..\getdata\src\putdata.c(532): error C2036: 'void *' : unknown size

>
> seems void* is not a synonym for a pointer, could it be replaced by a typedef?
>

  raw.c
  spf.c
  string.c
  types.c
  Generating Code...
------ Build started: Project: ALL_BUILD, Configuration: Debug Win32 ------
  Building Custom Rule Q:/kst/kst/misc/getdata-windows/cmake/CMakeLists.txt
  CMake does not need to re-run because Q:\getdata\_zehner\CMakeFiles\generate.stamp is up-to-date.
------ Skipped Build: Project: INSTALL, Configuration: Debug Win32 ------
Project not selected to build for this solution configuration
========== Build: 2 succeeded, 1 failed, 0 up-to-date, 1 skipped ==========