getdata-devel Mailing List for GetData (Page 3)
Scientific Database Format
Brought to you by:
ketiltrout
Menu
▾
▴
getdata-devel — General discussion of GetData development.
You can subscribe to this list here.
| 2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(10) |
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2009 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2010 |
Jan
|
Feb
(4) |
Mar
(1) |
Apr
(3) |
May
|
Jun
|
Jul
(21) |
Aug
(1) |
Sep
(16) |
Oct
(2) |
Nov
(12) |
Dec
(11) |
| 2011 |
Jan
(2) |
Feb
(5) |
Mar
(42) |
Apr
(1) |
May
|
Jun
|
Jul
(5) |
Aug
|
Sep
(4) |
Oct
(4) |
Nov
(7) |
Dec
(9) |
| 2012 |
Jan
|
Feb
|
Mar
|
Apr
(3) |
May
(1) |
Jun
|
Jul
(9) |
Aug
(1) |
Sep
|
Oct
(3) |
Nov
|
Dec
(5) |
| 2013 |
Jan
(2) |
Feb
|
Mar
(9) |
Apr
(3) |
May
(1) |
Jun
|
Jul
|
Aug
(3) |
Sep
(3) |
Oct
(1) |
Nov
(1) |
Dec
|
| 2014 |
Jan
(4) |
Feb
(7) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
|
Dec
(1) |
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(7) |
Jun
|
Jul
|
Aug
(6) |
Sep
(6) |
Oct
(1) |
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
(6) |
Mar
(11) |
Apr
(2) |
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
(1) |
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
|
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
| 2020 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
(1) |
Dec
|
|
From: Seth H. <set...@gm...> - 2015-05-06 17:12:34
|
Hi Don-
I believe that there are a couple bugs in _GD_MungeCode when metafields are
used. This is crashing in my use case but as it is a memory error, I
haven't gotten the test case to raise an error independently of valgrind.
With the attached patches, the valgrind error goes away, all tests pass and
my use case is also stable.
First:
The slash pointer points to the metafield after the '/' marker but len_sub
includes the '/' in its length calculation. While correct for allocation,
when nptr is incremented after the slash pointer is concatenated, it moves
the pointer beyond the end of the allocated string. The NULL-termination
then writes past the allocated memory by 1 byte.
Second:
The line `new_code[len_newpx + len + len_newsx] = '/';` duplicates the
behavior of the line immediately before it without name spaces and appears
to write into the middle of the new_code string when name spaces are in use.
I'm attaching a proposed patch for name.c as well as a revision to
test/madd_string.c that exposes the first issue when run using valgrind.
Do let me know if I am misunderstanding the intended behavior of the code
in the second issue.
Regards-
Seth
Index: src/name.c
===================================================================
--- src/name.c (revision 961)
+++ src/name.c (working copy)
@@ -102,7 +102,7 @@
*/
if ((slash = (char*)memchr(ptr, '/', len))) {
len_sub = len + (ptr - slash);
- len = slash++ - ptr;
+ len = slash - ptr;
}
/* Verify the suffix is present */
@@ -181,8 +181,6 @@
}
if (slash) {
- *(nptr++) = '/';
- new_code[len_newpx + len + len_newsx] = '/';
strcpy(nptr, slash);
nptr += len_sub;
}
Index: test/madd_string.c
===================================================================
--- test/madd_string.c (revision 961)
+++ test/madd_string.c (working copy)
@@ -34,7 +34,7 @@
{
const char *filedir = "dirfile";
const char *format = "dirfile/format";
- int error, ge_error, r = 0;
+ int error, mf_error, ge_error, r = 0;
gd_entry_t e;
char val[1000];
DIRFILE *D;
@@ -45,7 +45,13 @@
gd_madd_string(D, "new", "meta", "A string.");
error = gd_error(D);
+ gd_metaflush(D);
+ mf_error = gd_error(D);
+
/* check */
+ CHECKI(error, GD_E_OK);
+ CHECKI(mf_error, GD_E_OK);
+
gd_entry(D, "new/meta", &e);
ge_error = gd_error(D);
CHECKI(ge_error, 0);
@@ -62,6 +68,5 @@
unlink(format);
rmdir(filedir);
- CHECKI(error, GD_E_OK);
return r;
}
|
|
From: D. V. W. <ge...@ke...> - 2014-12-08 00:37:50
|
GetData-0.8.6 has been released. It may be downloaded from your local SourceForge mirror: https://sourceforge.net/projects/getdata/files/getdata/0.8.6/ This release fixes a number of issues discovered after the release of GetData-0.8.5. Full release notes are provided at the bottom of the release page linked above as well as in the file called NEWS included in the GetData release archives. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Steve M. <Ste...@na...> - 2014-09-19 19:13:29
|
Funny, this method didn't work when saving my dirfile on a different drive
(same machine, but new drive is SSD and the old was not).
However, getting the modified date (instead of the length) seems to do the
trick in this situation (I'm now querying both length and mod date just in
case). Also, querying just the "Reference" file is sufficient.
Steve
On Thu, Sep 18, 2014 at 3:53 PM, Steve Maher <Ste...@na...>
wrote:
> To followup on this issue, to get around kst not seeing dirfile updates on
> Windows, I just do the following (Java) after writing new data.
>
> private void triggerKstUpdate(String archiveDirectory)
> {
> for (File file : new File(archiveDirectory).listFiles())
> {
> file.length();
> }
> }
>
> Apparently asking Windows to calculate the file lengths triggers an update
> of the file metadata.
>
> Steve
>
>
> On Fri, Aug 23, 2013 at 4:22 PM, Steve Maher <Ste...@na...>
> wrote:
>
>>
>> On Fri, Aug 23, 2013 at 2:01 PM, Peter Kümmel <syn...@gm...>
>> wrote:
>>
>> Hi Steve, sorry for not having any tips?
>>>
>>>
>>
>>> At least one ;)
>>> Have you noticed the "Update type" on the first page of the wizard?
>>>
>>
>> Yup. Neither the Time Interval nor the Change Detection nor the No
>> Update ;) cause an update to work. Interestingly, the trick of doing a
>> cygwin "ls" (or refresh in Windows Explorer) tickles only the Time Interval
>> mode to work and doesn't help the Change Detection mode.
>>
>> Steve
>>
>> _______________________________________________
>> Kst mailing list
>> Ks...@kd...
>> https://mail.kde.org/mailman/listinfo/kst
>>
>>
>
|
|
From: Steve M. <Ste...@na...> - 2014-09-18 19:53:52
|
To followup on this issue, to get around kst not seeing dirfile updates on
Windows, I just do the following (Java) after writing new data.
private void triggerKstUpdate(String archiveDirectory)
{
for (File file : new File(archiveDirectory).listFiles())
{
file.length();
}
}
Apparently asking Windows to calculate the file lengths triggers an update
of the file metadata.
Steve
On Fri, Aug 23, 2013 at 4:22 PM, Steve Maher <Ste...@na...>
wrote:
>
> On Fri, Aug 23, 2013 at 2:01 PM, Peter Kümmel <syn...@gm...> wrote:
>
> Hi Steve, sorry for not having any tips?
>>
>>
>
>> At least one ;)
>> Have you noticed the "Update type" on the first page of the wizard?
>>
>
> Yup. Neither the Time Interval nor the Change Detection nor the No Update
> ;) cause an update to work. Interestingly, the trick of doing a cygwin
> "ls" (or refresh in Windows Explorer) tickles only the Time Interval mode
> to work and doesn't help the Change Detection mode.
>
> Steve
>
> _______________________________________________
> Kst mailing list
> Ks...@kd...
> https://mail.kde.org/mailman/listinfo/kst
>
>
|
|
From: Joy D. <did...@gm...> - 2014-02-20 04:44:24
|
That does fix the issue. Thanks a lot. On Wed, Feb 19, 2014 at 11:36 PM, D. V. Wiebe <ge...@ke...>wrote: > Thanks for the extra info, that was very helpful. I think you're running > afoul of two bugs that were fixed in GetData-0.8.4, namely: > > * BUG FIX: Trying to read sample zero of a gzipped RAW field the second > time > now works as expected. Previously, the request to reposition the I/O > pointer back to zero would be ignored by the gzip framework. Reported > by > Alexandra Rahlin. > > and: > > * PYTHON BUG FIX: Numpy arrays returned by dirfile.getdata() are now the > correct length. Previously they would always be the length requested > (or > the length of the dirfile, if no length was explicitly given) even if > fewer > samples were returned. Extra elements would contain uninitialised > memory. > > If you can't upgrade to a newer version of GetData, you might be able to > work > around it by skipping the zero'th sample and starting your read at sample > one. > > Let me know if that doesn't fix it, > > -don > -- > D. V. Wiebe > ge...@ke... > http://getdata.sourceforge.net/ > -- ------------------------------------------------------------------------------- Joy Didier Columbia Astrophysics Lab Physics PhD Candidate (212) 854-9230 (lab) (212) 854-8121 (fax) -------------------------------------------------------------------------------- |
|
From: D. V. W. <ge...@ke...> - 2014-02-20 04:36:59
|
Thanks for the extra info, that was very helpful. I think you're running
afoul of two bugs that were fixed in GetData-0.8.4, namely:
* BUG FIX: Trying to read sample zero of a gzipped RAW field the second time
now works as expected. Previously, the request to reposition the I/O
pointer back to zero would be ignored by the gzip framework. Reported by
Alexandra Rahlin.
and:
* PYTHON BUG FIX: Numpy arrays returned by dirfile.getdata() are now the
correct length. Previously they would always be the length requested (or
the length of the dirfile, if no length was explicitly given) even if fewer
samples were returned. Extra elements would contain uninitialised memory.
If you can't upgrade to a newer version of GetData, you might be able to work
around it by skipping the zero'th sample and starting your read at sample one.
Let me know if that doesn't fix it,
-don
--
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/
|
|
From: Joy D. <did...@gm...> - 2014-02-20 04:21:21
|
Last clarification:
This actually happens to any channel that I try to read more than once: the
first time around the channel is read properly, but all the following times
the channel is read erroneously.
Could the probelm be the the code is trying to unzip the channel at every
call of D.getdata, but the channel is already unzipped?
=============================================================================
D = gd.dirfile("ex_dirfile_3")
In [35]: D.getdata("third_channel")
Out[35]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,
85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
dtype=int8)
In [36]: D.getdata("third_channel")
Out[36]:
array([ -8, -42, 52, 85, -63, 127, 0, 0, 80, 98, 97, 2, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 112, 0, 0, 0], dtype=int8)
In [37]: D.getdata("flag0")
Out[37]:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
In [38]: D.getdata("flag0")
Out[38]:
array([ 140468334942744, 140468334942744, 43474376,
7943899258573201248, 140468348568752, 28397752,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
7943922258700201224, 32733584, 35756152,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
14808977229581140125, 35795952, 35804752,
0, 0, 0,
16249379669116094257, 140468348018048, 28396456,
13497258184966649490, 35796000, 35728200,
13497258184966649491, 35796048, 35727264,
142441597423371540, 35796864, 28396456,
0, 0, 0,
0, 0, 0,
2314054222116390903, 140468347760000, 35804624,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
17647712777910934205, 140468348361064, 35804688,
0, 0, 0,
0, 0, 0,
784, 80, 0,
32917232], dtype=uint64)
On Wed, Feb 19, 2014 at 11:01 PM, Joy Didier <did...@gm...> wrote:
> Investigating the bug more I've found that the problem is not which
> channel I read, but in which order I read them, between the bitfield and
> it's raw parent. See the printscreen below.
>
> The first channel read with D.getdata("channel_name") returns the correct
> values, but the second channel read with D.getdata returns bogus values, if
> the channels read have a bitfield/parent relation.
> If I close the dirfile and then reopen it, the first channel I read is
> correct and the second channel I read is incorrect.
>
> I've checked and this does not concern other raw channels that don't have
> an associated bitfield (I can read these raw channels first, or last, and
> they are still read correctly).
>
> This may invalidate the test of dirfile2ascii since I suspect it closes
> the dirfile after every call. It can also explain why the bug appeared the
> other way around in KST: I must have been loading the bitfield first,
> before the raw channel.
>
>
>
> ===================================================================================
> In [9]: D = gd.dirfile("ex_dirfile")
>
> In [10]: D.getdata("raw_flag")
> Out[10]:
> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3,
> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>
> In [11]: D.getdata("flag0")
> Out[11]:
> array([ 140468334942856, 140468334942856, 45150976,
> 45150976, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 641, 140468334941848,
> 140468334941848, 140468348018048, 28396456,
> 13497258184966649490, 35796000, 35728200,
> 13497258184966649491, 0, 0,
> 0, 1, 43002700,
> 0, 1, 0,
> 0, 0, 0,
> 2314054222116390903, 140468347760000, 35804624,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 17647712777910934205, 140468348361064, 35804688,
> 0, 0, 0,
> 0, 0, 0,
> 43137904, 257, 45155872,
> 140468334942088], dtype=uint64)
>
> In [12]: D.close()
>
> In [13]: D = gd.dirfile("ex_dirfile")
>
> In [14]: D.getdata("flag0")
> Out[14]:
> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
>
> In [15]: D.getdata("raw_flag")
> Out[15]:
> array([ -8, -42, 52, 85, -63, 127, 0, 0, -128, -98, -84,
> 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 112, 0, 0,
> 0], dtype=int8)
>
> In [16]: D.close()
> ============================================================
>
>
>
> On Wed, Feb 19, 2014 at 10:29 PM, Joy Didier <did...@gm...> wrote:
>
>> Hi,
>> Here is a similar print screen. I'm using 0.8.2.
>>
>> I'm running debian jessie/sid:
>> Linux version 3.9-1-amd64 (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP
>> Debian 3.9.4-1
>>
>> I just tried dirfile2ascii and it returns the correct values for both
>> raw_flag and flag0.
>> Joy
>>
>> ====================================================================
>> $ ls -l ex_dirfile/
>> total 8
>> -rw-r--r-- 1 joy ebex 263 Feb 19 00:27 format
>> -rw-r--r-- 1 joy ebex 35 Feb 19 00:27 raw_flag.gz
>> $ python
>> Python 2.7.5+ (default, Sep 17 2013, 15:31:50)
>> [GCC 4.8.1] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import pygetdata as gd
>> >>> D = gd.dirfile('ex_dirfile')
>> >>> D.getdata('raw_flag')
>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
>> 1,
>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2,
>> 2,
>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
>> 3,
>> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>> >>> D.getdata('flag0')
>> array([ 140462112111368, 140462112111368, 26196784,
>> 26196784, 25650002, 10,
>> 140462112113216, 8, 0,
>> 4507079348539146979, 140462124693968, 9446016,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 10710030370915244967, 140462124691504, 9446016,
>> 0, 0, 0,
>> 0, 0, 0,
>> 2880483376040711751, 140462125738096, 24267584,
>> 16077188030449198219, 20838272, 9446016,
>> 0, 0, 0,
>> 11123187229275223373, 140462124461968, 19069888,
>> 0, 0, 0,
>> 16606360166094669967, 140462124569872, 9446016,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 1, 140462124908820,
>> 140462124908820, 1, 6009525,
>> 0, 1, 0,
>> 9805731381512495190, 20838656, 24150544,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 0, 0, 0,
>> 784, 784, 0,
>> 0], dtype=uint64)
>> >>> gd.__version__
>> (0, 8, 2, '')
>> =======================================================================
>>
>>
>>
>>
>> On Wed, Feb 19, 2014 at 8:37 PM, D. V. Wiebe <ge...@ke...>wrote:
>>
>>> On Wed, Feb 19, 2014 at 01:58:44AM -0500, Joy Didier wrote:
>>> > Hi,
>>> > I think I've found a bug with getdata when reading gzip encoded
>>> channels
>>> > and using bitfields.
>>> > The attached dirfile was written using pygetdata and gzip encoding.
>>> > It has a 100 sample long int8 "raw_flag" channel that increments
>>> from 0 to
>>> > 3, and then back to zero.
>>> > It also has a bitfield "flag0" corresponding to bit 0 of "raw_flag".
>>> > When trying to read the data, I see the following bugs:
>>> > * reading the data using pygetdata: the raw_flag channel reads
>>> properly,
>>> > but the bitfield channel reads incorrect values. The incorrect
>>> values are
>>> > different every time I read the data.
>>> > * reading the data using KST2: the raw_flag channel reads zeros,
>>> but the
>>> > bitfield channel reads proper values. If I delete the bitfield from
>>> the
>>> > format file, then KST2 reads the raw_flag channel properly.
>>> > If I gunzip the channel, and get rid of the ENCODING directive in
>>> the
>>> > format file, all the channels read properly through KST and
>>> pygetdata.
>>> > Thanks,
>>> > Joy
>>>
>>> Thanks for the report, but I can't seem to reproduce the problem here
>>> with
>>> 0.8.5 and python (I haven't tried kst yet):
>>>
>>> $ ls -l test-dirfile
>>> total 8
>>> -rw------- 1 dvw dvw 263 Feb 19 17:09 format
>>> -rw------- 1 dvw dvw 35 Feb 19 17:09 raw_flag.gz
>>> $ python
>>> Python 2.7.5 (default, May 29 2013, 03:26:28)
>>> [GCC 4.8.0] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>> >>> import pygetdata as gd
>>> >>> D = gd.dirfile('test-dirfile')
>>> >>> D.getdata('raw_flag')
>>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
>>> 1, 1,
>>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
>>> 2, 2,
>>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3,
>>> 3, 3,
>>> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>>> >>> D.getdata('flag0')
>>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
>>> 1, 1,
>>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
>>> 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
>>> 1, 1,
>>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
>>> >>> gd.__version__
>>> (0, 8, 5, '')
>>> >>>
>>>
>>> Can you read the data correctly with, say, dirfile2ascii, which uses the
>>> C library directly? What architecture are you running on?
>>>
>>> Cheers,
>>> -dvw
>>> --
>>> D. V. Wiebe
>>> ge...@ke...
>>> http://getdata.sourceforge.net/
>>>
>>
>>
>>
>> --
>>
>>
>> -------------------------------------------------------------------------------
>> Joy Didier
>> Columbia Astrophysics Lab
>> Physics PhD Candidate
>> (212) 854-9230 (lab)
>> (212) 854-8121 (fax)
>>
>> --------------------------------------------------------------------------------
>>
>
>
>
> --
>
>
> -------------------------------------------------------------------------------
> Joy Didier
> Columbia Astrophysics Lab
> Physics PhD Candidate
> (212) 854-9230 (lab)
> (212) 854-8121 (fax)
>
> --------------------------------------------------------------------------------
>
--
-------------------------------------------------------------------------------
Joy Didier
Columbia Astrophysics Lab
Physics PhD Candidate
(212) 854-9230 (lab)
(212) 854-8121 (fax)
--------------------------------------------------------------------------------
|
|
From: Joy D. <did...@gm...> - 2014-02-20 04:01:48
|
Investigating the bug more I've found that the problem is not which channel
I read, but in which order I read them, between the bitfield and it's raw
parent. See the printscreen below.
The first channel read with D.getdata("channel_name") returns the correct
values, but the second channel read with D.getdata returns bogus values, if
the channels read have a bitfield/parent relation.
If I close the dirfile and then reopen it, the first channel I read is
correct and the second channel I read is incorrect.
I've checked and this does not concern other raw channels that don't have
an associated bitfield (I can read these raw channels first, or last, and
they are still read correctly).
This may invalidate the test of dirfile2ascii since I suspect it closes the
dirfile after every call. It can also explain why the bug appeared the
other way around in KST: I must have been loading the bitfield first,
before the raw channel.
===================================================================================
In [9]: D = gd.dirfile("ex_dirfile")
In [10]: D.getdata("raw_flag")
Out[10]:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
In [11]: D.getdata("flag0")
Out[11]:
array([ 140468334942856, 140468334942856, 45150976,
45150976, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 641, 140468334941848,
140468334941848, 140468348018048, 28396456,
13497258184966649490, 35796000, 35728200,
13497258184966649491, 0, 0,
0, 1, 43002700,
0, 1, 0,
0, 0, 0,
2314054222116390903, 140468347760000, 35804624,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
17647712777910934205, 140468348361064, 35804688,
0, 0, 0,
0, 0, 0,
43137904, 257, 45155872,
140468334942088], dtype=uint64)
In [12]: D.close()
In [13]: D = gd.dirfile("ex_dirfile")
In [14]: D.getdata("flag0")
Out[14]:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
In [15]: D.getdata("raw_flag")
Out[15]:
array([ -8, -42, 52, 85, -63, 127, 0, 0, -128, -98, -84,
2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 112, 0, 0,
0], dtype=int8)
In [16]: D.close()
============================================================
On Wed, Feb 19, 2014 at 10:29 PM, Joy Didier <did...@gm...> wrote:
> Hi,
> Here is a similar print screen. I'm using 0.8.2.
>
> I'm running debian jessie/sid:
> Linux version 3.9-1-amd64 (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP
> Debian 3.9.4-1
>
> I just tried dirfile2ascii and it returns the correct values for both
> raw_flag and flag0.
> Joy
>
> ====================================================================
> $ ls -l ex_dirfile/
> total 8
> -rw-r--r-- 1 joy ebex 263 Feb 19 00:27 format
> -rw-r--r-- 1 joy ebex 35 Feb 19 00:27 raw_flag.gz
> $ python
> Python 2.7.5+ (default, Sep 17 2013, 15:31:50)
> [GCC 4.8.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pygetdata as gd
> >>> D = gd.dirfile('ex_dirfile')
> >>> D.getdata('raw_flag')
> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3,
> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
> >>> D.getdata('flag0')
> array([ 140462112111368, 140462112111368, 26196784,
> 26196784, 25650002, 10,
> 140462112113216, 8, 0,
> 4507079348539146979, 140462124693968, 9446016,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 10710030370915244967, 140462124691504, 9446016,
> 0, 0, 0,
> 0, 0, 0,
> 2880483376040711751, 140462125738096, 24267584,
> 16077188030449198219, 20838272, 9446016,
> 0, 0, 0,
> 11123187229275223373, 140462124461968, 19069888,
> 0, 0, 0,
> 16606360166094669967, 140462124569872, 9446016,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 1, 140462124908820,
> 140462124908820, 1, 6009525,
> 0, 1, 0,
> 9805731381512495190, 20838656, 24150544,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 0, 0, 0,
> 784, 784, 0,
> 0], dtype=uint64)
> >>> gd.__version__
> (0, 8, 2, '')
> =======================================================================
>
>
>
>
> On Wed, Feb 19, 2014 at 8:37 PM, D. V. Wiebe <ge...@ke...>wrote:
>
>> On Wed, Feb 19, 2014 at 01:58:44AM -0500, Joy Didier wrote:
>> > Hi,
>> > I think I've found a bug with getdata when reading gzip encoded
>> channels
>> > and using bitfields.
>> > The attached dirfile was written using pygetdata and gzip encoding.
>> > It has a 100 sample long int8 "raw_flag" channel that increments
>> from 0 to
>> > 3, and then back to zero.
>> > It also has a bitfield "flag0" corresponding to bit 0 of "raw_flag".
>> > When trying to read the data, I see the following bugs:
>> > * reading the data using pygetdata: the raw_flag channel reads
>> properly,
>> > but the bitfield channel reads incorrect values. The incorrect
>> values are
>> > different every time I read the data.
>> > * reading the data using KST2: the raw_flag channel reads zeros, but
>> the
>> > bitfield channel reads proper values. If I delete the bitfield from
>> the
>> > format file, then KST2 reads the raw_flag channel properly.
>> > If I gunzip the channel, and get rid of the ENCODING directive in the
>> > format file, all the channels read properly through KST and
>> pygetdata.
>> > Thanks,
>> > Joy
>>
>> Thanks for the report, but I can't seem to reproduce the problem here with
>> 0.8.5 and python (I haven't tried kst yet):
>>
>> $ ls -l test-dirfile
>> total 8
>> -rw------- 1 dvw dvw 263 Feb 19 17:09 format
>> -rw------- 1 dvw dvw 35 Feb 19 17:09 raw_flag.gz
>> $ python
>> Python 2.7.5 (default, May 29 2013, 03:26:28)
>> [GCC 4.8.0] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import pygetdata as gd
>> >>> D = gd.dirfile('test-dirfile')
>> >>> D.getdata('raw_flag')
>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
>> 1, 1,
>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
>> 2, 2,
>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3,
>> 3, 3,
>> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>> >>> D.getdata('flag0')
>> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
>> 1, 1,
>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
>> 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
>> 1, 1,
>> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
>> >>> gd.__version__
>> (0, 8, 5, '')
>> >>>
>>
>> Can you read the data correctly with, say, dirfile2ascii, which uses the
>> C library directly? What architecture are you running on?
>>
>> Cheers,
>> -dvw
>> --
>> D. V. Wiebe
>> ge...@ke...
>> http://getdata.sourceforge.net/
>>
>
>
>
> --
>
>
> -------------------------------------------------------------------------------
> Joy Didier
> Columbia Astrophysics Lab
> Physics PhD Candidate
> (212) 854-9230 (lab)
> (212) 854-8121 (fax)
>
> --------------------------------------------------------------------------------
>
--
-------------------------------------------------------------------------------
Joy Didier
Columbia Astrophysics Lab
Physics PhD Candidate
(212) 854-9230 (lab)
(212) 854-8121 (fax)
--------------------------------------------------------------------------------
|
|
From: Joy D. <did...@gm...> - 2014-02-20 03:30:27
|
Hi,
Here is a similar print screen. I'm using 0.8.2.
I'm running debian jessie/sid:
Linux version 3.9-1-amd64 (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP
Debian 3.9.4-1
I just tried dirfile2ascii and it returns the correct values for both
raw_flag and flag0.
Joy
====================================================================
$ ls -l ex_dirfile/
total 8
-rw-r--r-- 1 joy ebex 263 Feb 19 00:27 format
-rw-r--r-- 1 joy ebex 35 Feb 19 00:27 raw_flag.gz
$ python
Python 2.7.5+ (default, Sep 17 2013, 15:31:50)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygetdata as gd
>>> D = gd.dirfile('ex_dirfile')
>>> D.getdata('raw_flag')
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>>> D.getdata('flag0')
array([ 140462112111368, 140462112111368, 26196784,
26196784, 25650002, 10,
140462112113216, 8, 0,
4507079348539146979, 140462124693968, 9446016,
0, 0, 0,
0, 0, 0,
0, 0, 0,
10710030370915244967, 140462124691504, 9446016,
0, 0, 0,
0, 0, 0,
2880483376040711751, 140462125738096, 24267584,
16077188030449198219, 20838272, 9446016,
0, 0, 0,
11123187229275223373, 140462124461968, 19069888,
0, 0, 0,
16606360166094669967, 140462124569872, 9446016,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 1, 140462124908820,
140462124908820, 1, 6009525,
0, 1, 0,
9805731381512495190, 20838656, 24150544,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
784, 784, 0,
0], dtype=uint64)
>>> gd.__version__
(0, 8, 2, '')
=======================================================================
On Wed, Feb 19, 2014 at 8:37 PM, D. V. Wiebe <ge...@ke...> wrote:
> On Wed, Feb 19, 2014 at 01:58:44AM -0500, Joy Didier wrote:
> > Hi,
> > I think I've found a bug with getdata when reading gzip encoded
> channels
> > and using bitfields.
> > The attached dirfile was written using pygetdata and gzip encoding.
> > It has a 100 sample long int8 "raw_flag" channel that increments from
> 0 to
> > 3, and then back to zero.
> > It also has a bitfield "flag0" corresponding to bit 0 of "raw_flag".
> > When trying to read the data, I see the following bugs:
> > * reading the data using pygetdata: the raw_flag channel reads
> properly,
> > but the bitfield channel reads incorrect values. The incorrect values
> are
> > different every time I read the data.
> > * reading the data using KST2: the raw_flag channel reads zeros, but
> the
> > bitfield channel reads proper values. If I delete the bitfield from
> the
> > format file, then KST2 reads the raw_flag channel properly.
> > If I gunzip the channel, and get rid of the ENCODING directive in the
> > format file, all the channels read properly through KST and pygetdata.
> > Thanks,
> > Joy
>
> Thanks for the report, but I can't seem to reproduce the problem here with
> 0.8.5 and python (I haven't tried kst yet):
>
> $ ls -l test-dirfile
> total 8
> -rw------- 1 dvw dvw 263 Feb 19 17:09 format
> -rw------- 1 dvw dvw 35 Feb 19 17:09 raw_flag.gz
> $ python
> Python 2.7.5 (default, May 29 2013, 03:26:28)
> [GCC 4.8.0] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pygetdata as gd
> >>> D = gd.dirfile('test-dirfile')
> >>> D.getdata('raw_flag')
> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
> 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2,
> 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
> 3,
> 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
> >>> D.getdata('flag0')
> array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
> 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1,
> 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
> >>> gd.__version__
> (0, 8, 5, '')
> >>>
>
> Can you read the data correctly with, say, dirfile2ascii, which uses the
> C library directly? What architecture are you running on?
>
> Cheers,
> -dvw
> --
> D. V. Wiebe
> ge...@ke...
> http://getdata.sourceforge.net/
>
--
-------------------------------------------------------------------------------
Joy Didier
Columbia Astrophysics Lab
Physics PhD Candidate
(212) 854-9230 (lab)
(212) 854-8121 (fax)
--------------------------------------------------------------------------------
|
|
From: D. V. W. <ge...@ke...> - 2014-02-20 01:38:04
|
On Wed, Feb 19, 2014 at 01:58:44AM -0500, Joy Didier wrote:
> Hi,
> I think I've found a bug with getdata when reading gzip encoded channels
> and using bitfields.
> The attached dirfile was written using pygetdata and gzip encoding.
> It has a 100 sample long int8 "raw_flag" channel that increments from 0 to
> 3, and then back to zero.
> It also has a bitfield "flag0" corresponding to bit 0 of "raw_flag".
> When trying to read the data, I see the following bugs:
> * reading the data using pygetdata: the raw_flag channel reads properly,
> but the bitfield channel reads incorrect values. The incorrect values are
> different every time I read the data.
> * reading the data using KST2: the raw_flag channel reads zeros, but the
> bitfield channel reads proper values. If I delete the bitfield from the
> format file, then KST2 reads the raw_flag channel properly.
> If I gunzip the channel, and get rid of the ENCODING directive in the
> format file, all the channels read properly through KST and pygetdata.
> Thanks,
> Joy
Thanks for the report, but I can't seem to reproduce the problem here with
0.8.5 and python (I haven't tried kst yet):
$ ls -l test-dirfile
total 8
-rw------- 1 dvw dvw 263 Feb 19 17:09 format
-rw------- 1 dvw dvw 35 Feb 19 17:09 raw_flag.gz
$ python
Python 2.7.5 (default, May 29 2013, 03:26:28)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygetdata as gd
>>> D = gd.dirfile('test-dirfile')
>>> D.getdata('raw_flag')
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=int8)
>>> D.getdata('flag0')
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0], dtype=uint64)
>>> gd.__version__
(0, 8, 5, '')
>>>
Can you read the data correctly with, say, dirfile2ascii, which uses the
C library directly? What architecture are you running on?
Cheers,
-dvw
--
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/
|
|
From: Steve M. <Ste...@na...> - 2014-01-23 13:16:24
|
Hi Don, Thanks for the thorough explanation. I haven't used multiple dirfiles in kst. Having used it a bit I wouldn't be surprised if it "just works". One thing I'm considering is throwing out (or interpolating) samples from my non-uniform data before writing them to the dirfile to make sure I hit the ratio in the format file. Thanks, Steve On Mon, Jan 20, 2014 at 9:49 PM, D. V. Wiebe <ge...@ke...> wrote: > On Fri, Jan 10, 2014 at 02:57:21PM -0500, Steve Maher wrote: > [...] > > Assume we have a clock that provides "frame numbers". We have > resistance > > and temperature samples with associated frame numbers, but the > sampling > > isn't at a fixed rate. So for our dirfile we have four files: > > 1. temperature // the temperature readings > > 2. temperatureFrameNumbers // frame numbers of temperature samples > > 3. resistance // the resistance readings > > 4. resistanceFrameNumbers // frame numbers of resistance samples > > In general, temperatures have more samples. We have tried format > files > > like > > > > temperature RAW ... >1 [and >> 1] > > temperatureFrameNumbers RAW ... >1 [ and >> 1] > > resistance RAW ... 1 > > resistanceFrameNumbers RAW ... 1 > > /REFERENCE resistanceFrameNumbers > > > > and > > > > temperature RAW ... 1 > > temperatureFrameNumbers RAW ... 1 > > resistance RAW ... 1 > > resistanceFrameNumbers RAW ... 1 > > /REFERENCE resistanceFrameNumbers [also tried > temperatureFrameNumbers] > > > > The last format version seems to work the best. I am able to read the > > temperatures and resistances into kst (using x axes of > > temperatureFrameNumbers and resistanceFrameNumbers respectively) and > then > > align the data using a shared axes. However, kst wants to make the > > vectors the same size (probably what it's supposed to do) and will > pad the > > shorter vector with the first value (causing a line drawn from the > end of > > the plot back to the beginning). > > Is there are better way to store this data with dirfile? > > Thanks, > > Steve > > Hey Steve, > > Short answer: I'd advise storing the temperature data in one dirfile and > the resistance data is a second, separate dirfile. > > Long answer follows. > > Dirfiles don't actually need data to be sampled at a fixed rate (because > they have no conception of time), but they *do* assume that: > > 1. the datastream is broken up into a sequence of frames > 2. a timestream (RAW) is sampled a fixed number of times in a frame > 3. every timestream is sampled in every frame > > (i.e. they assume the data are sampled synchronously). > > Non-synchronously sampled data are especially problematic if you're using > kst to > read them. As you have probably surmised, in "read to end" mode, every kst > update cycle: > > 1. kst determines the number of frames of data in the REFERENCE field, and > assumes every field in the dirfile has (at least) that amount of data in > it. > > 2. for each loaded vector, kst attempts to read enough data to advance > the end of the vector to the frame number retrieved in step one > ie. it's trying to make all the vectors the same number of frames > (not samples) in size. > > If a Dirfile field has fewer samples available than the reference > field, kst will be given less data than it requests when it calls > getdata for a read. I thought in this case, it would just produce a > short vector, and attempt to read more data the next update cycle, > but perhaps that behaviour has changed. > > Similarly, in "count from end" mode, kst uses the REFERENCE field to > figure out where the "end" of the dirfile is; that is, it doesn't > get a separate end frame for each vector. > > The easiest solution might be to store the two different datastreams in > two different dirfiles. Then you get to use two REFERENCE fields and > it won't matter that there are differing amounts of temperature and > resistance data. The fact that the data come from different data sources > should be transparent to the user once read into kst. > > If you need to stick with a single dirfile, unfortunately, I haven't been > able to come up with a clever way of working around this limitation. > > It sounds like you've already done this, but things may improve somewhat if > you play around with the samples-per-frame for the two sets of data (the > last parameter in the RAW field line). Ideally you'd want the ratio of > the these two numbers to be as close to the ratio of the two sample > rates of the actual data. This should reduce the amount of zeroed data > you see in kst. > > With the REFERENCE field tied to the slower of the two data streams, as > you've done, what will happen is that kst won't read to the end of the > temperature data, since it assumes (from the REFERENCE field) that > there's less temperature data than there actually is. So, you might > find that you're missing the end of the temperature data. > > Also: because kst uses changes in the number of frames in the REFERENCE > field to detect updates to the dirfile, when creating a dirfile writer, > make sure it writes to the REFERENCE field last. > > We're running into problems associated with non-synchronously sampled > data with Spider, and I am trying to come up with a scheme to permit it, > but it will likely require some invasive changes to the innards of > GetData, so it may not pan out. > > If you haven't done so yet, you might want to ask about this on the kst > mailing list, too. While I have a good idea of how kst is interfacing > with GetData, there may be higher-level behaviour that I'm missing that > you might be able to exploit. > > Cheers, > -don > -- > D. V. Wiebe > ge...@ke... > http://getdata.sourceforge.net/ > |
|
From: D. V. W. <ge...@ke...> - 2014-01-21 03:18:45
|
On Fri, Jan 10, 2014 at 02:57:21PM -0500, Steve Maher wrote: [...] > Assume we have a clock that provides "frame numbers". We have resistance > and temperature samples with associated frame numbers, but the sampling > isn't at a fixed rate. So for our dirfile we have four files: > 1. temperature // the temperature readings > 2. temperatureFrameNumbers // frame numbers of temperature samples > 3. resistance // the resistance readings > 4. resistanceFrameNumbers // frame numbers of resistance samples > In general, temperatures have more samples. We have tried format files > like > > temperature RAW ... >1 [and >> 1] > temperatureFrameNumbers RAW ... >1 [ and >> 1] > resistance RAW ... 1 > resistanceFrameNumbers RAW ... 1 > /REFERENCE resistanceFrameNumbers > > and > > temperature RAW ... 1 > temperatureFrameNumbers RAW ... 1 > resistance RAW ... 1 > resistanceFrameNumbers RAW ... 1 > /REFERENCE resistanceFrameNumbers [also tried temperatureFrameNumbers] > > The last format version seems to work the best. I am able to read the > temperatures and resistances into kst (using x axes of > temperatureFrameNumbers and resistanceFrameNumbers respectively) and then > align the data using a shared axes. However, kst wants to make the > vectors the same size (probably what it's supposed to do) and will pad the > shorter vector with the first value (causing a line drawn from the end of > the plot back to the beginning). > Is there are better way to store this data with dirfile? > Thanks, > Steve Hey Steve, Short answer: I'd advise storing the temperature data in one dirfile and the resistance data is a second, separate dirfile. Long answer follows. Dirfiles don't actually need data to be sampled at a fixed rate (because they have no conception of time), but they *do* assume that: 1. the datastream is broken up into a sequence of frames 2. a timestream (RAW) is sampled a fixed number of times in a frame 3. every timestream is sampled in every frame (i.e. they assume the data are sampled synchronously). Non-synchronously sampled data are especially problematic if you're using kst to read them. As you have probably surmised, in "read to end" mode, every kst update cycle: 1. kst determines the number of frames of data in the REFERENCE field, and assumes every field in the dirfile has (at least) that amount of data in it. 2. for each loaded vector, kst attempts to read enough data to advance the end of the vector to the frame number retrieved in step one ie. it's trying to make all the vectors the same number of frames (not samples) in size. If a Dirfile field has fewer samples available than the reference field, kst will be given less data than it requests when it calls getdata for a read. I thought in this case, it would just produce a short vector, and attempt to read more data the next update cycle, but perhaps that behaviour has changed. Similarly, in "count from end" mode, kst uses the REFERENCE field to figure out where the "end" of the dirfile is; that is, it doesn't get a separate end frame for each vector. The easiest solution might be to store the two different datastreams in two different dirfiles. Then you get to use two REFERENCE fields and it won't matter that there are differing amounts of temperature and resistance data. The fact that the data come from different data sources should be transparent to the user once read into kst. If you need to stick with a single dirfile, unfortunately, I haven't been able to come up with a clever way of working around this limitation. It sounds like you've already done this, but things may improve somewhat if you play around with the samples-per-frame for the two sets of data (the last parameter in the RAW field line). Ideally you'd want the ratio of the these two numbers to be as close to the ratio of the two sample rates of the actual data. This should reduce the amount of zeroed data you see in kst. With the REFERENCE field tied to the slower of the two data streams, as you've done, what will happen is that kst won't read to the end of the temperature data, since it assumes (from the REFERENCE field) that there's less temperature data than there actually is. So, you might find that you're missing the end of the temperature data. Also: because kst uses changes in the number of frames in the REFERENCE field to detect updates to the dirfile, when creating a dirfile writer, make sure it writes to the REFERENCE field last. We're running into problems associated with non-synchronously sampled data with Spider, and I am trying to come up with a scheme to permit it, but it will likely require some invasive changes to the innards of GetData, so it may not pan out. If you haven't done so yet, you might want to ask about this on the kst mailing list, too. While I have a good idea of how kst is interfacing with GetData, there may be higher-level behaviour that I'm missing that you might be able to exploit. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Steve M. <Ste...@na...> - 2014-01-16 14:59:44
|
Hmm.. anyone out there? Steve On Fri, Jan 10, 2014 at 2:57 PM, Steve Maher <Ste...@na...>wrote: > Greetings, > > I've been having fun with dirfile - good stuff! > > I'm running into a bit of a problem, though, as our application has data > streams that don't have uniform sampling rates and dirfile seems to require > uniformity (given the last field of the RAW field spec). > > I should say that the main issue is that we're having difficulty with kst > being happy with our data (although kst is probably doing the Right Thing). > > I understand I may be trying to fit a square peg in a round hole, but I'm > hoping you guys could provide some insight. > > I imagine I don't need to provide an example for this group, but to be > clear I'll provide one. > > Assume we have a clock that provides "frame numbers". We have resistance > and temperature samples with associated frame numbers, but the sampling > isn't at a fixed rate. So for our dirfile we have four files: > > > 1. temperature // the temperature readings > 2. temperatureFrameNumbers // frame numbers of temperature samples > 3. resistance // the resistance readings > 4. resistanceFrameNumbers // frame numbers of resistance samples > > > In general, temperatures have more samples. We have tried *format* files > like > > temperature RAW ... >1 [and >> 1] > temperatureFrameNumbers RAW ... >1 [ and >> 1] > resistance RAW ... 1 > resistanceFrameNumbers RAW ... 1 > /REFERENCE resistanceFrameNumbers > > > and > > temperature RAW ... 1 > temperatureFrameNumbers RAW ... 1 > resistance RAW ... 1 > resistanceFrameNumbers RAW ... 1 > /REFERENCE resistanceFrameNumbers [also tried temperatureFrameNumbers] > > > > The last format version seems to work the best. I am able to read the > temperatures and resistances into kst (using x axes of > temperatureFrameNumbers and resistanceFrameNumbers respectively) and then > align the data using a shared axes. However, kst wants to make the vectors > the same size (probably what it's supposed to do) and will pad the shorter > vector with the first value (causing a line drawn from the end of the plot > back to the beginning). > > Is there are better way to store this data with dirfile? > > Thanks, > Steve > > > > |
|
From: Steve M. <Ste...@na...> - 2014-01-10 19:57:29
|
Greetings, I've been having fun with dirfile - good stuff! I'm running into a bit of a problem, though, as our application has data streams that don't have uniform sampling rates and dirfile seems to require uniformity (given the last field of the RAW field spec). I should say that the main issue is that we're having difficulty with kst being happy with our data (although kst is probably doing the Right Thing). I understand I may be trying to fit a square peg in a round hole, but I'm hoping you guys could provide some insight. I imagine I don't need to provide an example for this group, but to be clear I'll provide one. Assume we have a clock that provides "frame numbers". We have resistance and temperature samples with associated frame numbers, but the sampling isn't at a fixed rate. So for our dirfile we have four files: 1. temperature // the temperature readings 2. temperatureFrameNumbers // frame numbers of temperature samples 3. resistance // the resistance readings 4. resistanceFrameNumbers // frame numbers of resistance samples In general, temperatures have more samples. We have tried *format* files like temperature RAW ... >1 [and >> 1] temperatureFrameNumbers RAW ... >1 [ and >> 1] resistance RAW ... 1 resistanceFrameNumbers RAW ... 1 /REFERENCE resistanceFrameNumbers and temperature RAW ... 1 temperatureFrameNumbers RAW ... 1 resistance RAW ... 1 resistanceFrameNumbers RAW ... 1 /REFERENCE resistanceFrameNumbers [also tried temperatureFrameNumbers] The last format version seems to work the best. I am able to read the temperatures and resistances into kst (using x axes of temperatureFrameNumbers and resistanceFrameNumbers respectively) and then align the data using a shared axes. However, kst wants to make the vectors the same size (probably what it's supposed to do) and will pad the shorter vector with the first value (causing a line drawn from the end of the plot back to the beginning). Is there are better way to store this data with dirfile? Thanks, Steve |
|
From: Steve B. <sb...@ph...> - 2013-11-13 00:57:58
|
Hi getdata, I was getting a segfault in RawEntry's destructor, because the non-default constructor does not initialize filename. The attached patch resolves the problem for me. Best, -Steve |
|
From: D. V. W. <ge...@ke...> - 2013-10-19 03:15:37
|
On Mon, Apr 22, 2013 at 11:21:58AM -0700, sf-...@sn... wrote: > Hi Don- > > I'd like to propose the attached patch. It moves the responsibility > for temporary file creation to the encoder, instead of passing in an > already opened file descriptor. The justification for the change is > to allow expandability for encodings to use fields that don't > correspond to a specific file in the directory (e.g. the zzip > interface). > > I'd like to submit an additional patch to add HDF encoding to getdata > but the attached patch is a pre-req that touches other encodings as > well, so I'm proposing it separately. I think this will provide for > greater expandability in the future as well. > > Of note: I don't change the oop writes, which also use temporary > files. I think that this could be addressed in the future by the > addition of a GD_FILE_OOP flag. > > Let me know if you have any questions, concerns, suggestions, etc. > > Best- > Seth Hey Seth, I think I forgot to do something with this patch. I've now committed it to trunk (revision 852). Apologies for the delay, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2013-09-10 23:33:28
|
On Tue, Sep 10, 2013 at 10:37:25AM -0400, Kevin MacDermid wrote: > Hey guys, > I've had a minor issue with the python getdata bindings for 0.8.4 and now > 0.8.5. After doing the standard "configure", "make", "make install." I get > the following from the python shell: > > ImportError: libgetdata.so.5: cannot open shared object file: No such > file or directory > > This is easily fixed by creating the following link: > > sudo ln -s /usr/local/lib/libgetdata.so.5 /usr/lib/libgetdata.so.5 > > But is a minor annoyance that seems like it should be taken care of by the > installer. > Thanks! Sounds like /usr/local/lib isn't in the search path of your runtime linker-loader. Try adding /usr/local/lib to /etc/ld.so.conf and then running /sbin/ldconfig. Alternately, you can change where GetData is installed by using the --prefix option in the configure script. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Kevin M. <kev...@ma...> - 2013-09-10 14:38:03
|
Hey guys, I've had a minor issue with the python getdata bindings for 0.8.4 and now 0.8.5. After doing the standard "configure", "make", "make install." I get the following from the python shell: ImportError: libgetdata.so.5: cannot open shared object file: No such file > or directory This is easily fixed by creating the following link: sudo ln -s /usr/local/lib/libgetdata.so.5 /usr/lib/libgetdata.so.5 But is a minor annoyance that seems like it should be taken care of by the installer. Thanks! -- Kevin MacDermid PhD Student Department of Physics McGill University Lab 1-514-398-4838 |
|
From: D. V. W. <ge...@ke...> - 2013-09-05 22:39:25
|
GetData 0.8.5 has been released. It may be downloaded from your local SourceForge mirror: http://sourceforge.net/projects/getdata/files/getdata/0.8.5/ This release fixes two minor bugs introduced in GetData-0.8.4: * A spurious debugging statement has been removed from the Python bindings. * The zzslim encoding framework, indadvertantly broken in 0.8.4, should once again work. Full release notes are provided at the bottom of the release page linked above as well as in the file called NEWS included in the GetData release archives. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Steve M. <Ste...@na...> - 2013-08-30 19:06:52
|
Don, Thanks for the thoughts. I've seen verification of the stat64 issue (e.g., MS forum<http://social.msdn.microsoft.com/Forums/vstudio/en-US/fcc39d92-efbc-4c37-b43e-0d80c802efde/stat64-does-not-get-current-file-size>). I had tried flushing the writes (I'm using custom Java code) with no luck. I may muck around with a different way to get the file size - hopefully platform independent .. Steve On Tue, Aug 27, 2013 at 7:12 PM, D. V. Wiebe <ge...@ke...> wrote: > On Fri, Aug 23, 2013 at 01:43:11PM -0400, Steve Maher wrote: > > Hi, > > I'm using kst2 2.0.7 on Windows with dirfiles for a data source and > kst > > doesn't seem to detect dirfile updates. The funny thing is, if I do > an > > "ls" in a Cygwin shell on the dirfile directory, kst instantly sees > the > > change. The same system updates fine on Linux. > > I swear in early googling I saw a comment about Windows doing > something > > funky and users had to close their dirfiles to trigger the update but > I > > can't find that now! Not a nice solution regardless ... > > After an initial source code swoop it looks like gd_nframes() is used > by > > kst to trigger the update so after I complete my struggle to compile > kst > > with 3rd party libs on Windows I'll be looking into that function. > > In the meantime if anyone has any tips I'd greatly appreciate it. > > Steve > > Hi Steve, > > Unfortunately, I don't have a whole lot of experience with dirfiles on > windows, but... > > kst using gd_nframes() to detect dirfile changes sounds right. > gd_nframes() is a pretty simple thing. When using the Microsoft VC > runtime, for uncompressed raw data, (after drilling down through all the > compatibility layers) it just calls _stat64 on the raw file associated > with the reference field to get the file size and divides that by the > sample rate and data size to get number of frames. > > However, after some investigation, it appears that Win32 doesn't > synchronously update file metadata on open filehandles (so _stat64 > returns the wrong thing). I don't understand what Cygwin is doing to > the filesystem to get it to force it, but it may be that simply using > _stat64 doesn't work as expected, and GetData should be using something > else under Windows. > > As a work-around, in the dirfile writer, it might be worth trying to > flush and/or close the reference field after adding data to it. (If > you're using GetData to write the dirfile, try a call to one of > gd_raw_close, gd_sync, or gd_flush.) This may force Windows to update > the file metadata. You should only ever have to flush/close the > reference field. The downside is the additional latency it will add > to writing. > > Cheers, > -dvw > -- > D. V. Wiebe > ge...@ke... > http://getdata.sourceforge.net/ > |
|
From: D. V. W. <ge...@ke...> - 2013-08-27 23:45:42
|
On Fri, Aug 23, 2013 at 01:43:11PM -0400, Steve Maher wrote: > Hi, > I'm using kst2 2.0.7 on Windows with dirfiles for a data source and kst > doesn't seem to detect dirfile updates. The funny thing is, if I do an > "ls" in a Cygwin shell on the dirfile directory, kst instantly sees the > change. The same system updates fine on Linux. > I swear in early googling I saw a comment about Windows doing something > funky and users had to close their dirfiles to trigger the update but I > can't find that now! Not a nice solution regardless ... > After an initial source code swoop it looks like gd_nframes() is used by > kst to trigger the update so after I complete my struggle to compile kst > with 3rd party libs on Windows I'll be looking into that function. > In the meantime if anyone has any tips I'd greatly appreciate it. > Steve Hi Steve, Unfortunately, I don't have a whole lot of experience with dirfiles on windows, but... kst using gd_nframes() to detect dirfile changes sounds right. gd_nframes() is a pretty simple thing. When using the Microsoft VC runtime, for uncompressed raw data, (after drilling down through all the compatibility layers) it just calls _stat64 on the raw file associated with the reference field to get the file size and divides that by the sample rate and data size to get number of frames. However, after some investigation, it appears that Win32 doesn't synchronously update file metadata on open filehandles (so _stat64 returns the wrong thing). I don't understand what Cygwin is doing to the filesystem to get it to force it, but it may be that simply using _stat64 doesn't work as expected, and GetData should be using something else under Windows. As a work-around, in the dirfile writer, it might be worth trying to flush and/or close the reference field after adding data to it. (If you're using GetData to write the dirfile, try a call to one of gd_raw_close, gd_sync, or gd_flush.) This may force Windows to update the file metadata. You should only ever have to flush/close the reference field. The downside is the additional latency it will add to writing. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Steve M. <Ste...@na...> - 2013-08-23 17:43:23
|
Hi, I'm using kst2 2.0.7 on Windows with dirfiles for a data source and kst doesn't seem to detect dirfile updates. The funny thing is, if I do an "ls" in a Cygwin shell on the dirfile directory, kst instantly sees the change. The same system updates fine on Linux. I swear in early googling I saw a comment about Windows doing something funky and users had to close their dirfiles to trigger the update but I can't find that now! Not a nice solution regardless ... After an initial source code swoop it looks like gd_nframes() is used by kst to trigger the update so after I complete my struggle to compile kst with 3rd party libs on Windows I'll be looking into that function. In the meantime if anyone has any tips I'd greatly appreciate it. Steve |
|
From: D. V. W. <ge...@ke...> - 2013-05-14 02:55:45
|
GetData 0.8.4 has been released. It may be downloaded from your local SourceForge mirror: http://sourceforge.net/projects/getdata/files/getdata/0.8.4/ This release of GetData co-incides with a correction to the Dirfile Standards which hopefully clarifies the meaning of the final, optional parameter of a MPLEX definition, along with fixing the inconsistent behaviour in the library concerning the handling of this parameter. See the release notes for particulars. Other bug fixes in GetData 0.8.4 include: * seeking to a negative sample number with gd_seek now properly fails * reading sample zero of a gzipped RAW field now always works * /REFERENCE directives in subdirfiles are no longer corrupted on write * NumPy arrays returned by the Python dirfile.getdata() method are now the correct length Full release notes are provided at the bottom of the release page linked above as well as in the file called NEWS included in the GetData release archives. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: <sf-...@sn...> - 2013-04-22 18:22:11
|
Hi Don-
I'd like to propose the attached patch. It moves the responsibility
for temporary file creation to the encoder, instead of passing in an
already opened file descriptor. The justification for the change is
to allow expandability for encodings to use fields that don't
correspond to a specific file in the directory (e.g. the zzip
interface).
I'd like to submit an additional patch to add HDF encoding to getdata
but the attached patch is a pre-req that touches other encodings as
well, so I'm proposing it separately. I think this will provide for
greater expandability in the future as well.
Of note: I don't change the oop writes, which also use temporary
files. I think that this could be addressed in the future by the
addition of a GD_FILE_OOP flag.
Let me know if you have any questions, concerns, suggestions, etc.
Best-
Seth
Index: src/ascii.c
===================================================================
diff --git a/trunk/getdata/src/ascii.c b/trunk/getdata/src/ascii.c
--- a/trunk/getdata/src/ascii.c (revision 830)
+++ b/trunk/getdata/src/ascii.c (working copy)
@@ -37,13 +37,14 @@
file->idata = gd_OpenAt(file->D, fd, file->name, ((mode & GD_FILE_WRITE)
? (O_RDWR | O_CREAT) : O_RDONLY) | O_BINARY, 0666);
- if (file->idata < 0) {
- dreturn("%i", -1);
- return -1;
- }
} else
- file->idata = fd;
+ file->idata = _GD_MakeTempFile(file->D, fd, file->name);
+ if (file->idata < 0) {
+ dreturn("%i", -1);
+ return -1;
+ }
+
file->edata = fdopen(file->idata, (mode & GD_FILE_WRITE) ? "rb+" : "rb");
if (file->edata == NULL) {
Index: src/encoding.c
===================================================================
diff --git a/trunk/getdata/src/encoding.c b/trunk/getdata/src/encoding.c
--- a/trunk/getdata/src/encoding.c (revision 830)
+++ b/trunk/getdata/src/encoding.c (working copy)
@@ -544,16 +544,13 @@
E->e->u.raw.file + 1, filebase, 1, 0))
{
; /* error already set */
- } else if ((temp_fd = _GD_MakeTempFile(D, D->fragment[fragment].dirfd,
- E->e->u.raw.file[1].name)) < 0)
- {
- _GD_SetError(D, GD_E_RAW_IO, 0, E->e->u.raw.file[1].name, errno, NULL);
- } else if ((*enc->open)(temp_fd, E->e->u.raw.file + 1, swap,
+ } else if ((*enc->open)(D->fragment[fragment].dirfd,
E->e->u.raw.file + 1, swap,
GD_FILE_WRITE | GD_FILE_TEMP))
{
_GD_SetError(D, GD_E_RAW_IO, 0, E->e->u.raw.file[1].name, errno, NULL);
Index: src/gzip.c
===================================================================
diff --git a/trunk/getdata/src/gzip.c b/trunk/getdata/src/gzip.c
--- a/trunk/getdata/src/gzip.c (revision 830)
+++ b/trunk/getdata/src/gzip.c (working copy)
@@ -51,6 +51,8 @@
return 1;
}
gzmode = "r";
+ } else if (mode & GD_FILE_TEMP) {
+ file->idata = _GD_MakeTempFile(file->D, fd, file->name);
} else
file->idata = fd;
Index: src/raw.c
===================================================================
diff --git a/trunk/getdata/src/raw.c b/trunk/getdata/src/raw.c
--- a/trunk/getdata/src/raw.c (revision 830)
+++ b/trunk/getdata/src/raw.c (working copy)
@@ -34,9 +34,11 @@
file->idata = gd_OpenAt(file->D, fd, file->name, ((mode & GD_FILE_WRITE) ?
(O_RDWR | O_CREAT) : O_RDONLY) | O_BINARY, 0666);
- } else
- file->idata = fd;
+ } else {
+ file->idata = _GD_MakeTempFile(file->D, fd, file->name);
+ }
+
file->pos = 0;
file->mode = mode | GD_FILE_READ;
Index: src/sie.c
===================================================================
diff --git a/trunk/getdata/src/sie.c b/trunk/getdata/src/sie.c
--- a/trunk/getdata/src/sie.c (revision 830)
+++ b/trunk/getdata/src/sie.c (working copy)
@@ -40,14 +40,14 @@
if (!(mode & GD_FILE_TEMP)) {
fd = gd_OpenAt(file->D, fdin, file->name, ((mode & GD_FILE_WRITE) ?
(O_RDWR | O_CREAT) : O_RDONLY) | O_BINARY, 0666);
+ } else {
+ fd = _GD_MakeTempFile(file->D, fdin, file->name);
+ }
- if (fd < 0) {
- dreturn("%i", -1);
- return -1;
- }
- } else
- fd = fdin;
-
+ if (fd < 0) {
+ dreturn("%i", -1);
+ return -1;
+ }
stream = fdopen(fd, (mode & GD_FILE_WRITE) ? "rb+" : "rb");
if (stream == NULL) {
|
|
From: D. V. W. <ge...@ke...> - 2013-04-11 00:30:26
|
On Wed, Apr 10, 2013 at 12:50:20PM -0700, sf-...@sn... wrote:
> Hi Don-
>
> My ExtUtils::CBuilder has 'use strict' and parses include_dirs as an
> array. This breaks when the Perl binding build with the error:
> Can't use string ("../../src") as an ARRAY ref while "strict refs" in
> use at /System/Library/Perl/5.10.0/ExtUtils/CBuilder/Base.pm line 92.
>
> I've attached a patch which fixes the issue and works correctly for
> Debian Sid, Ubuntu 12.04 and MacOS 10.6.8
>
> Thanks,
> Seth
Thanks for the report. I've committed the patch. It should appear in
in the imminent 0.8.4 release.
If you (or any one else) would like to try it, there's a release
candidate here:
http://ketiltrout.net/tmp/
Cheers,
-dvw
--
D. V. Wiebe
ge...@ke...
http://getdata.sourceforge.net/
|
1 message has been excluded from this view by a project administrator.
