getdata-devel — General discussion of GetData development.

Re: [getdata-devel] Next step

[Barth N. <bar...@gm...>]

Peter,

That would be great.

The first step is to port the getdata library, which dirfile uses.
getdata lives at

http://getdata.sourceforge.net/

You will probably want to become part of
get...@li..., and check with Don about what
status things are at in the port.

My son looked at it a year ago, and found that it was using some
large-file libc extensions which weren't in windows, or something like
that.

In parallel, we should be looking into preparing for a kst2 KDE release.

On Tue, Jul 20, 2010 at 6:04 PM, Peter Kümmel <syn...@gm...> wrote:
> Hi Barth,
>
> I thought about starting with porting dirfile to Windows.
> What do you think, is this a good idea?
> Am I right that at the moment this datasource
> is the most important one?
>
> Peter


-- 
C. Barth Netterfield
University of Toronto
416-845-0946

[getdata-devel] GetData 0.6.3 released

[D. V. W. <ge...@ke...>]

GetData 0.6.3 has been released. It may be downloaded from your local
SourceForge mirror:

http://sourceforge.net/projects/getdata/files/getdata/0.6.3/

This minor bugfix release fixes a memory leak in the Python bindings.
All Python users should upgrade.

The release notes contain a complete list of the changes from the
previous release:

http://sourceforge.net/projects/getdata/files/getdata/0.6.3/RELEASE_NOTES-0.6.3/view

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] Simpy Python Question

[D. V. W. <ge...@ke...>]

On Mon, Apr 12, 2010 at 10:05:25PM -0500, Ross Williamson wrote:
>    Hi There
>    Simple question about the Python bindings using putdata.  I have a piece
>    of code that is sampling some temperatures once a second.  I open a
>    dirfile (using CREAT, RDWR, UNENCODED).  
>    I get my data and then write each temperture to the dirfile using putdata
>    then call flush().
>    The problem is that when I call putdata a second later it does not append
>    the data, but overwrites the data in each of the files in the dirfile
>    directory.
>    How do I tell it to append rather than overwrite the data in there?
>    Let me know if you need some code snippets
>    Thanks
>    Ross

(Apologies for the delayed response.)

There's no explicit "append" mode.  Rather, you must explicitly tell
putdata() where in the dirfile to store each chunk of data you give it.
That is to say, getdata has no concept of a "write pointer".  All writes
must be accompanied by an address.  (The same thing goes for reads.)

Unfortunately, the Python bindings make this less than blatantly
obvious, since putdata() requires neither first_frame, nor first_sample,
and if both are omitted they both default to zero.  (Maybe it should
complain in that case?)

So, the answer to your question is: you must explictly specify the
frame or sample at which you want to write each bit of data you shove
into putdata() by using either first_frame=<blah>, or
first_sample=<blah>.  If you specify both these, it will add them
together (taking into account samples-per-frame).

Hope that makes sense,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Simpy Python Question

[Ross W. <ros...@gm...>]

Hi There

Simple question about the Python bindings using putdata.  I have a piece of
code that is sampling some temperatures once a second.  I open a dirfile
(using CREAT, RDWR, UNENCODED).

I get my data and then write each temperture to the dirfile using putdata
then call flush().

The problem is that when I call putdata a second later it does not append
the data, but overwrites the data in each of the files in the dirfile
directory.

How do I tell it to append rather than overwrite the data in there?

Let me know if you need some code snippets

Thanks

Ross

[getdata-devel] Fwd: getdata on macosx

[Barth N. <net...@as...>]

----------  Forwarded Message  ----------

Subject: getdata on macosx
Date: March 16, 2010
From: Carlo Contaldi <c.c...@im...>
To: "MacTavish, Carolyn Judith" <cma...@as...>, "C. Barth 
Netterfield" <net...@as...>

I just got getdata to compile on mac osx - just needed to add a 
definition of off64_t in getdata.h

e.g.

#ifndef off64_t
typedef __darwin_off_t off64_t;
#endif

and comment out the definition of off_t in internal.h, it is already 
defined and was creating a conflict. Can someone add these things into 
the sourceforge version?

Carlo

-------------------------------------------------------

[getdata-devel] GetData 0.6.2 released

[D. V. W. <ge...@ke...>]

GetData 0.6.2 has been released.  It may be downloaded from your local
SourceForge mirror:

http://sourceforge.net/projects/getdata/files/getdata/0.6.2/

This is a minor bugfix release.  It fixes a bug when using an ASCII file
as the reference field for a dirfile.

The release notes contain a complete list of the (one) change from the
previous release:

http://sourceforge.net/projects/getdata/files/getdata/0.6.2/RELEASE_NOTES-0.6.2/view

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] GetData 0.6.1 released.

[D. V. W. <ge...@ke...>]

GetData 0.6.1 has been released.  It may be downloaded from your local
SourceForge mirror:

http://sourceforge.net/projects/getdata/files/getdata/0.6.1/

This is a minor bugfix release.  It mostly fixes a few bugs in the build
system; but if your copy of GetData built without issue, you don't have
to worry about them.  It also fixes potential segfaults in the Legacy
API and the IDL bindings.

This release also removes the uncessesary inclusion of <complex.h> in
getdata.h, which was causing trouble for some users.

The release notes contain a complete list of changes from the previous
release:

http://sourceforge.net/projects/getdata/files/getdata/0.6.1/RELEASE_NOTES-0.6.1/view

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] getdata on windows

[D. V. W. <ge...@ke...>]

On Tue, Feb 09, 2010 at 11:54:21AM +0100, Johan de Haas wrote:
> Can I use getdata on a windows pc.
> I want to use it in a python program.
> 
> 
> Johan

It might be persuaded to work under Cygwin, but I don't know of any
attempts to try it.  GetData's interface with the operating system isn't
very spectacular, I can't forsee any reason why it shouldn't work under
Windows.

A few months ago, an attempt was made to compile GetData using MinGW.  I
don't know how it turned out.

I'm hoping to get it working under Windows for the upcoming 0.7.0
release.

-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] getdata on windows

[Johan de H. <j.t...@tu...>]

Can I use getdata on a windows pc.
I want to use it in a python program.


Johan

[getdata-devel] GetData 0.6.0 released

[D. V. W. <ge...@ke...>]

GetData 0.6.0 has been released.  Thanks to MDT and CBN for testing
pre-release candidates.  It may be downloaded from your local
SourceForge mirror:

http://sourceforge.net/projects/getdata/files/getdata/0.6.0/

Most notably, GetData 0.6.0 marks a change in licensing of the GetData
library.  GetData is now distributed under the GNU Lesser General Public
License (LGPL), version 2.1 or, at your option, any later version.

This release co-incides with a new version of the Dirfile Standards
(Version Seven).  The new standards include support for complex valued
data, a polynomial derived field (POLYNOM), and a signed bit field
(SBIT), and a few other minor tweaks.

As a result, GetData can now handle complex valued data, and by default
exports an API which uses the C99 `complex' macro.  (A C89 API is also
available, for those who don't wan't to use C99).

GetData 0.6.0 also introduces bindings to the library for Python and the
Interactive Data Language (IDL).

Furthermore, in addition to bug fixes, changes in GetData 0.6.0 include:
* a new, more useful, but API breaking, parser callback handler
  signature;
* a speedier LINCOM calculator for LINCOMs of two or three fields (~20%
  improvement)
* some rudimentary attempts at making nicer looking (in the human
  readable sense) format files (see the GD_PRETTY_PRINT flag).
* due to changes introduced in Standards Version 7, dirfile_add_spec()
  and dirfile_alter_spec() can now add metafields.
* CONST scalar field codes used in the definition of a derived field are
  now available in the gd_entry_t object.  Because of this,
  dirfile_add(), dirfile_alter_entry(), and dirfile_madd() can be used
  to manipulate non-literal field parameters.
* A reverse-lookup function, get_framenum(), is available to calculate a
  frame number given a data value.  Naturally, it only works on
  monotonic vectors.
* The test-suites for the various bindings have been greatly expanded.
  As a result, they are significanlty less buggy than in previous
  releases.

The release notes contain a complete list of changes from the previous
release:

http://sourceforge.net/projects/getdata/files/getdata/0.6.0/RELEASE_NOTES-0.6.0/view

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] Memory Leak

[D. V. W. <ge...@ke...>]

On Thu, Oct 29, 2009 at 08:22:42PM -0400, Barth Netterfield wrote:
> I'm having some memory leaks with getdata when opening and closing files:
> 
> First, what looks like an error:
> in bindings/cxx/dirfile.c, in the destructor:
> 
> should
> 
> if (D == NULL)
>     dirfile_close(D);
> 
> 
> instead be
> 
> if (D != NULL)
>     dirfile_close(D);

Fixed.  (Actually I just removed the conditional; dirfile_close is one
of the very few functions which will accept NULL as a dirfile pointer.)

> Second:
> valgrind complains that in _GD_ParseRaw(), 
> 
>  E->e->filebase = malloc(FILENAME_MAX);
>  
> is never freed.  dirfile_close is getting called.

Okay, fixed that one too.  Thanks.

Let me know if you find anything else.
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Memory Leak

[Barth N. <net...@as...>]

I'm having some memory leaks with getdata when opening and closing files:

First, what looks like an error:
in bindings/cxx/dirfile.c, in the destructor:

should

if (D == NULL)
    dirfile_close(D);


instead be

if (D != NULL)
    dirfile_close(D);


Second:
valgrind complains that in _GD_ParseRaw(), 

 E->e->filebase = malloc(FILENAME_MAX);
 
is never freed.  dirfile_close is getting called.

This is important because I create a dirfile object every time a new extant 
file or directory is typed in the file selector, to check to see if it is a 
valid dirfile.

cbn

Re: [getdata-devel] getdata with minGW

[D. V. W. <ge...@ke...>]

On Thu, Sep 03, 2009 at 09:00:30PM -0400, Joshua Netterifled wrote:
> Hello,
> 
> I'll try explicitly casting all the references to edata in ascii.c to FILE*.

Let me know if it works.

> In any case, the includes did not fix the problem, unfortunately. Any other 
> ideas?

Hmmm, nothing profound.  If you're getting a size unknown error instead
of a syntax error, it would seem to indicate that the "struct stat64"
is declared in the included header files, but not defined, which seems
odd to me.

I'm surprised "struct __stat64" would still have this problem.  I'd
investigate sys/stat.h and see what it actually defines.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] getdata with minGW

[D. V. W. <ge...@ke...>]

On Thu, Sep 03, 2009 at 04:01:21PM -0400, Joshua Netterifled wrote:
> Hello,
> 
> Has anyone had any luck porting getdata to MinGW?

Oooo, sounds scary.  GetData likely assumes all sorts of funny (and
hitherto undiscovered) things about the POSIXity of the operating
system it's running on, but I guess you'll discover that, if it's the
case.

> First of all, the definition of _gd_raw_file was unusable in MinGW. I am too 
> much of a newbie to know why edata was defined as void*. Would not FILE * work 
> fine? If so, starting at line 156, this is my change:
> 
> struct _gd_raw_file {
>   char* name;
>   int fp;
> #if defined(mingw32_HOST_OS) || defined(__MINGW32__)
>   FILE *edata;
> #else
>   void* edata;
> #endif
>   int encoding;
> };

edata is a void pointer because the different encoding schemes use it to
point to different file-related data, about which GetData itself is
agnostic.  For example, the gzip encoding scheme uses it to point to a
gzFile object.

If you don't use any of these external encodings, the only thing left
that uses edata is the ascii encoder, which indeed uses edata as a FILE
pointer.

If all the references to edata in ascii.c are typecast explicitly to a
FILE* does that fix the problem?

> Secondly, on line 249 of encoding.c, I get the error "storage size of 
> 'statbuf' isn't known", and I am clueless as to how to fix it. (as a side-
> note, in minGW,  "stat64" should be "__stat64", but this is not the problem)

You mean:

  struct __stat64 statbuf;

still gives you a problem?

It's possible this is because I forgot to include some required
headers in encoding.c.  Does it work if you include <sys/types.h> and
<unistd.h> along with <sys/stat.h>?

Are you also required to rename the function call from stat64() to
_stat64()?

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] getdata with minGW

[Joshua N. <drm...@gm...>]

Hello,

I'll try explicitly casting all the references to edata in ascii.c to FILE*.

In any case, the includes did not fix the problem, unfortunately. Any other 
ideas?

And yes, I had already renamed the function call.

Thank you for your time,
Josh

[getdata-devel] getdata with minGW

[Joshua N. <drm...@gm...>]

Hello,

Has anyone had any luck porting getdata to MinGW?

If not, being inexperienced, I would like some help.

First of all, the definition of _gd_raw_file was unusable in MinGW. I am too 
much of a newbie to know why edata was defined as void*. Would not FILE * work 
fine? If so, starting at line 156, this is my change:

struct _gd_raw_file {
  char* name;
  int fp;
#if defined(mingw32_HOST_OS) || defined(__MINGW32__)
  FILE *edata;
#else
  void* edata;
#endif
  int encoding;
};

Secondly, on line 249 of encoding.c, I get the error "storage size of 
'statbuf' isn't known", and I am clueless as to how to fix it. (as a side-
note, in minGW,  "stat64" should be "__stat64", but this is not the problem)

Thank you for your time,
Josh

[getdata-devel] python getdata bindings

[Michael M. <mmi...@as...>]

Hi DVW et al,

Since you expressed an interest in python bindings for GetData, I'm
attaching such a python module.  Hopefully it works well enough that it
can be incorporated into the GetData release.

This module should work as-is on python 2.5+; I believe it will work on
older pythons if the ctypes add-on module is installed, but I have not
tested this yet.  

The interface is not especially python-ish, it should be close to an
exact duplicate of the C library API.  Included in the package is a
program, linearizer.py, that serves as a usage example.

I should thank Brendan Crill for providing some useful input and
feedback in the development of this module.

...Michael Milligan
Observational Cosmology Group
University of Minnesota

-- 
Key fingerprint = 9F6B E8F5 206F 35E9 FABB  9EAD 398D CD42 D1CE 8C87

[getdata-devel] GetData 0.5.0 released

[D. V. W. <ge...@ke...>]

GetData 0.5.0 has been released.  Again, thanks to Matthew Truch for his
help in beta testing.  It may be downloaded form your local SourceForge
mirror:

http://sourceforge.net/project/showfiles.php?group_id=229287

GetData 0.5.0 is expected to be API-complete, by which I mean, although
functionality may still be missing, no further additions to the list of
public function provided by GetData is expected.

In addition to various bug fixes, changes in GetData 0.5.0 include:
* The ability to delete fields, and modify field metadata
* The ability to change frameoffset, encoding, byte ordering, or
  reference field
* An extra flag, GD_IGNORE_DUPS, to make the parser more forgiving when
  reading format files with duplicate fields.
* The ability to move fields between format file fragments, and to
  remove fragments from a dirfile.
* Read-only support for bzip2 and gzip compressed dirfiles
* A callback function can now be used to more gracefully handle syntax
  errors in format files.
* Support for GetData modules (which is useful to package distributors,
  but can be ignored by users compiling directly from source.)

The Dirfile Standards are unchanged from GetData 0.4.x (Standards
Version 6).

The release notes contain a complete list of changes from the previous
release:

http://sourceforge.net/project/shownotes.php?group_id=229287&release_id=654262

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

Re: [getdata-devel] Getdata IDL interface

[D. V. W. <ge...@ke...>]

On Sun, Dec 14, 2008 at 11:53:58PM +0100, Samuel Leach wrote:
> Hello everyone,
> 
> I've just started using the getdata library and I was wondering if
> anyone has already coded up an IDL dirfile interface ?
> 
> Thanks for any information,
> 
> Sam Leach

I believe Gaelen Marsden at UBC wrote an IDL library for reading
dirfiles some time ago.  I don't know what state it's in, but I
suspect they wouldn't support the latest Dirfile Standards that
the GetData Project now writes.

IIRC, his IDL routines weren't a GetData wrapper, but a stand-alone
library for reading dirfiles.

-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Getdata IDL interface

[Samuel L. <le...@si...>]

Hello everyone,

I've just started using the getdata library and I was wondering if
anyone has already coded up an IDL dirfile interface ?

Thanks for any information,

Sam Leach

-- 
Samuel Leach
Astrophysics Sector
SISSA-ISAS
via Beirut 4
34014 Trieste, Italy.

Tel: +39-040-3787527
Fax: +39-040-3787528
http://people.sissa.it/~leach/

Re: [getdata-devel] getdata python

[D. V. W. <ge...@ke...>]

On Wed, Dec 10, 2008 at 01:56:49AM -0800, Brendan Crill wrote:
> Hi Getdata people,
> 
> I've been working on and using a python wrapper for getdata and I'd like to 
> make it available to others.  Do you think this is something that should be 
> added to the getdata package, or should I release and maintain it somewhere 
> else?
> 
> -- Brendan

A python wrapper sounds cool!  I'd love to have it in the getdata
package.  Could you send me a copy of it, so I'd have an idea of what
that would require? 

-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] getdata python

[Brendan C. <Bre...@jp...>]

Hi Getdata people,

I've been working on and using a python wrapper for getdata and I'd like to 
make it available to others.  Do you think this is something that should be 
added to the getdata package, or should I release and maintain it somewhere 
else?

-- Brendan
===========================
Dr. Brendan Crill
Cryogenic Instrument and Technology Development Group
Jet Propulsion Laboratory
Mail stop 79-24
4800 Oak Grove Drive
Pasadena CA 91109
818 354-5416 voice, 818 590-8276 cell
818 393 4878 fax
===========================

Re: [getdata-devel] Getdata assumes uniform encoding

[D. V. W. <ge...@ke...>]

On Thu, Nov 20, 2008 at 01:03:54PM -0500, Adam D. Hincks wrote:
> Hi Don,
> 
> Ran into a hitch reading a dirfile that contained some fields unencoded 
> and some fields slim-encoded.  It appears that the first time 
> get_nframes64() is called it uses the encoding of the reference field to 
> fix all the rest of the fields' encoding.
> 
> I made the following hack in _GD_DoRaw() in getdata.c so that I could read 
> the dirfile, but I suspect we'll want a more elegant and robust solution 
> in the long run:

It sounds to me like you have both slim encoded and unencoded RAW fields
defined in the same format file fragment.  That doesn't work, as you've
discovered.

The Standards indicate that all RAW fields defined in a given fragment
must have the same encoding.  To have the situation you desire, you have
to define the slim encoded RAW fields in one fragment and the unencoded
RAW fields in a different fragment, and then INCLUDE one fragment in the
other.  Derived fields may be defined anywhere.

-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/

[getdata-devel] Getdata assumes uniform encoding

[Adam D. H. <ah...@Pr...>]

Hi Don,

Ran into a hitch reading a dirfile that contained some fields unencoded 
and some fields slim-encoded.  It appears that the first time 
get_nframes64() is called it uses the encoding of the reference field to 
fix all the rest of the fields' encoding.

I made the following hack in _GD_DoRaw() in getdata.c so that I could read 
the dirfile, but I suspect we'll want a more elegant and robust solution 
in the long run:

<snip>
       /* Figure out the encoding subtype, if required */
       if (E->e->encoding == GD_ENC_UNKNOWN) {
         //_GD_ResolveEncoding(E->e->file,
         //    D->fragment[E->fragment_index].flags, E->e);
         _GD_ResolveEncoding(E->e->file, GD_AUTO_ENCODED, E->e);
       }
</snip>

Adam

[getdata-devel] kst 1.x and the GetData Project

[D. V. W. <ge...@ke...>]

I have updated the kst 1.x in SVN to optionally use the modern GetData
library.  It works thusly: if a suitably modern GetData library is found
at configure time, it will be used.  Otherwise the old getdata library
provided with the kst sources will be used (which supports dirfiles only
up to Standards Version 4).  GetData 0.4.2 (just released) or better is
required by kst 1.x.

The GetData support in kst 1.x uses the legacy API.  As a result, when
compiling kst 1.x against the GetData, don't be alarmed by the plethora
of "deprecated" warnings.

Cheers,
-dvw
-- 
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/