getdata-devel Mailing List for GetData (Page 2)
Scientific Database Format
Brought to you by:
ketiltrout
Menu
▾
▴
getdata-devel — General discussion of GetData development.
You can subscribe to this list here.
| 2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(10) |
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2009 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2010 |
Jan
|
Feb
(4) |
Mar
(1) |
Apr
(3) |
May
|
Jun
|
Jul
(21) |
Aug
(1) |
Sep
(16) |
Oct
(2) |
Nov
(12) |
Dec
(11) |
| 2011 |
Jan
(2) |
Feb
(5) |
Mar
(42) |
Apr
(1) |
May
|
Jun
|
Jul
(5) |
Aug
|
Sep
(4) |
Oct
(4) |
Nov
(7) |
Dec
(9) |
| 2012 |
Jan
|
Feb
|
Mar
|
Apr
(3) |
May
(1) |
Jun
|
Jul
(9) |
Aug
(1) |
Sep
|
Oct
(3) |
Nov
|
Dec
(5) |
| 2013 |
Jan
(2) |
Feb
|
Mar
(9) |
Apr
(3) |
May
(1) |
Jun
|
Jul
|
Aug
(3) |
Sep
(3) |
Oct
(1) |
Nov
(1) |
Dec
|
| 2014 |
Jan
(4) |
Feb
(7) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
|
Dec
(1) |
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(7) |
Jun
|
Jul
|
Aug
(6) |
Sep
(6) |
Oct
(1) |
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
(6) |
Mar
(11) |
Apr
(2) |
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
(1) |
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
|
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
| 2020 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
(1) |
Dec
|
|
From: D. V. W. <dv...@ke...> - 2016-02-25 00:06:51
|
On Wed, Feb 24, 2016 at 03:16:34PM -0800, D. V. Wiebe wrote: > Hi Cristian, > > Thanks for the report! > > What platform is this on? Did 0.9.0 work for you? I shall investigate. > > -don Ah, I found the problem. Looks like I added a fencepost error to gd_entry_list in this latest release. It's been fixed in the repo. I guess there'll be a 0.9.2 in the near future, but probably not until next week. There's one more thing that I need to look in to. Cheers, -don -- Don Wiebe dv...@ph... Department of Physics and Astronomy University of British Columbia Hennings 204 6224 Agricultural Road Tele: +1-604-822-2585 University Endowment Lands, BC Canada V6T 1Z1 http://ketiltrout.net/ |
|
From: D. V. W. <ge...@ke...> - 2016-02-24 23:16:44
|
On Sat, Feb 20, 2016 at 09:54:14AM +0100, Christian Trippe wrote: > Hi Don! > > I was going to update the openSUSE packages. However the name_alias test is > always failing. Can you have a look at this. > > Attached is the relevant part of the build log. > > Regards > Christian Hi Cristian, Thanks for the report! What platform is this on? Did 0.9.0 work for you? I shall investigate. -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Christian T. <ct...@op...> - 2016-02-20 08:54:24
|
Hi Don!
I was going to update the openSUSE packages. However the name_alias test is
always failing. Can you have a look at this.
Attached is the relevant part of the build log.
Regards
Christian
*** Error in `/home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/
name_alias': free(): invalid pointer: 0x0000000000cc97c0 ***
[ 898s] ======= Backtrace: =========
[ 898s] /lib64/libc.so.6(+0x73755)[0x7f3e35604755]
[ 898s] /lib64/libc.so.6(+0x79096)[0x7f3e3560a096]
[ 898s] /home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.
7(+0x15164)[0x7f3e3594b164]
[ 898s] /home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.
7(+0x9e61)[0x7f3e3593fe61]
[ 898s] /home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/
name_alias[0x4011de]
[ 898s] /lib64/libc.so.6(__libc_start_main+0xf0)[0x7f3e355b1610]
[ 898s] /home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/
name_alias[0x401529]
[ 898s] ======= Memory map: ========
[ 898s] 00400000-00402000 r-xp 00000000 fd:00 492884
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/name_alias
[ 898s] 00601000-00602000 r--p 00001000 fd:00 492884
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/name_alias
[ 898s] 00602000-00603000 rw-p 00002000 fd:00 492884
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/test/.libs/name_alias
[ 898s] 00cc9000-00ceb000 rw-p 00000000 00:00 0
[heap]
[ 898s] 7f3e34c6e000-7f3e34c84000 r-xp 00000000 fd:00 178959
/lib64/libgcc_s.so.1
[ 898s] 7f3e34c84000-7f3e34e83000 ---p 00016000 fd:00 178959
/lib64/libgcc_s.so.1
[ 898s] 7f3e34e83000-7f3e34e84000 r--p 00015000 fd:00 178959
/lib64/libgcc_s.so.1
[ 898s] 7f3e34e84000-7f3e34e85000 rw-p 00016000 fd:00 178959
/lib64/libgcc_s.so.1
[ 898s] 7f3e34e85000-7f3e34e88000 r-xp 00000000 fd:00 178830
/lib64/libdl-2.22.so
[ 898s] 7f3e34e88000-7f3e35087000 ---p 00003000 fd:00 178830
/lib64/libdl-2.22.so
[ 898s] 7f3e35087000-7f3e35088000 r--p 00002000 fd:00 178830
/lib64/libdl-2.22.so
[ 898s] 7f3e35088000-7f3e35089000 rw-p 00003000 fd:00 178830
/lib64/libdl-2.22.so
[ 898s] 7f3e35089000-7f3e35186000 r-xp 00000000 fd:00 178832
/lib64/libm-2.22.so
[ 898s] 7f3e35186000-7f3e35385000 ---p 000fd000 fd:00 178832
/lib64/libm-2.22.so
[ 898s] 7f3e35385000-7f3e35386000 r--p 000fc000 fd:00 178832
/lib64/libm-2.22.so
[ 898s] 7f3e35386000-7f3e35387000 rw-p 000fd000 fd:00 178832
/lib64/libm-2.22.so
[ 898s] 7f3e35387000-7f3e35390000 r-xp 00000000 fd:00 50565
/usr/lib64/libltdl.so.7.3.1
[ 898s] 7f3e35390000-7f3e3558f000 ---p 00009000 fd:00 50565
/usr/lib64/libltdl.so.7.3.1
[ 898s] 7f3e3558f000-7f3e35590000 r--p 00008000 fd:00 50565
/usr/lib64/libltdl.so.7.3.1
[ 898s] 7f3e35590000-7f3e35591000 rw-p 00009000 fd:00 50565
/usr/lib64/libltdl.so.7.3.1
[ 898s] 7f3e35591000-7f3e3572c000 r-xp 00000000 fd:00 178824
/lib64/libc-2.22.so
[ 898s] 7f3e3572c000-7f3e3592c000 ---p 0019b000 fd:00 178824
/lib64/libc-2.22.so
[ 898s] 7f3e3592c000-7f3e35930000 r--p 0019b000 fd:00 178824
/lib64/libc-2.22.so
[ 898s] 7f3e35930000-7f3e35932000 rw-p 0019f000 fd:00 178824
/lib64/libc-2.22.so
[ 898s] 7f3e35932000-7f3e35936000 rw-p 00000000 00:00 0
[ 898s] 7f3e35936000-7f3e35981000 r-xp 00000000 fd:00 489485
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.7.0.1
[ 898s] 7f3e35981000-7f3e35b81000 ---p 0004b000 fd:00 489485
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.7.0.1
[ 898s] 7f3e35b81000-7f3e35b82000 r--p 0004b000 fd:00 489485
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.7.0.1
[ 898s] 7f3e35b82000-7f3e35b83000 rw-p 0004c000 fd:00 489485
/home/abuild/rpmbuild/BUILD/getdata-0.9.1/src/.libs/libgetdata.so.7.0.1
[ 899s] 7f3e35b83000-7f3e35ba5000 r-xp 00000000 fd:00 178946
/lib64/ld-2.22.so
[ 899s] 7f3e35d9b000-7f3e35da0000 rw-p 00000000 00:00 0
[ 899s] 7f3e35da2000-7f3e35da4000 rw-p 00000000 00:00 0
[ 899s] 7f3e35da4000-7f3e35da5000 r--p 00021000 fd:00 178946
/lib64/ld-2.22.so
[ 899s] 7f3e35da5000-7f3e35da6000 rw-p 00022000 fd:00 178946
/lib64/ld-2.22.so
[ 899s] 7f3e35da6000-7f3e35da7000 rw-p 00000000 00:00 0
[ 899s] 7fff4e719000-7fff4e73a000 rw-p 00000000 00:00 0
[stack]
[ 899s] 7fff4e7c8000-7fff4e7ca000 r--p 00000000 00:00 0
[vvar]
[ 899s] 7fff4e7ca000-7fff4e7cc000 r-xp 00000000 00:00 0
[vdso]
[ 899s] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
[ 899s] /bin/sh: line 5: 2480 Aborted ${dir}$tst
[ 899s] FAIL: name_alias
|
|
From: D. V. W. <ge...@ke...> - 2016-02-19 00:18:59
|
Hi all, A new bug-fix release of GetData, version 0.9.1, is availabe here: https://sourceforge.net/projects/getdata/files/getdata/0.9.1/ GetData-0.9.1 fixes bugs found since the release of GetData-0.9.0. See the release notes at the bottom of the file release page linked above for full details. Downstream packagers should note the following change to the build system: * The build system now uses ExtUtils::MakeMaker instead of Module::Build to build the perl bindings. Module::Build was removed from the Perl 5 core in Perl 5.22. ExtUtils::MakeMaker is still a core module. The bindings are unchanged; the change in build prerequisites is the only difference users should notice due to this switch. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2016-02-16 23:19:51
|
On Sat, Feb 13, 2016 at 05:45:43PM +0000, Matthew Petroff wrote: > When I apply the /HIDDEN directive to a RAW data field that has an > alias, both the original field and the alias disappear from the field > list (I'm using GetData 0.9.0). This seems contradictory to the > documentation, which says that "[the /HIDDEN directive] does not hide > all aliases of the field-name." Is this behavior intended, or is it a > bug? Thanks. > -Matthew Petroff That is indeed a bug. As it turns out, it's already been fixed in the repository. It should be in the 0.9.1 release I hope to put out soon. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Matthew P. <ge...@mp...> - 2016-02-13 18:14:39
|
When I apply the /HIDDEN directive to a RAW data field that has an alias, both the original field and the alias disappear from the field list (I'm using GetData 0.9.0). This seems contradictory to the documentation, which says that "[the /HIDDEN directive] does not hide all aliases of the field-name." Is this behavior intended, or is it a bug? Thanks. -Matthew Petroff |
|
From: D. V. W. <ge...@ke...> - 2015-10-16 23:06:36
|
Hi all, We've just released a new version of GetData, version 0.9.0. You can download it from your local SourceForge mirror: https://sourceforge.net/projects/getdata/files/getdata/0.9.0/ GetData-0.9.0 fixes bugs found in the 0.8-series. It also adds PHP bindings, a FLAC encoding, write support for bzip2 and xz-compressed datafiles. GetData-0.9.0 has been tested on various platforms including Linux, MaxOS X, OpenBSD, Cygwin, and native Microsoft Windows. Full release notes can be found at the above link, and in the file called NEWS in the release archive. Downstream packagers may wish to note especially the the following changes to the build system: * the legacy (pre-0.3) API is no longer built by default. Pass --enable-legacy-api to the configure script to enable it. * building support for the FLAC encoding reqires Xiph.Org's libFLAC. * the Python bindings now require NumPy; previously, it was optional * building the PHP bindings requires the PHP command-line interpreter * to be installed at configure time Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2015-09-17 00:07:39
|
I've fixed the bug. It turns out the problem isn't with the amount of data you write at a time, but the seek that occurs at the start of your second write (specified by the first_frame= parameter in the putdata call.) The bug in GetData-0.8.9 and earlier mean that seeks of more than 1,000,000 bytes (corresponding to 250,000 samples for FLOAT32 data) at a time before a write to a gzipped datafile will fail. While I've been investigating this, I've found a number of bugs related to writing gzipped data at large offsets, which makes a simple workaround difficult. I'm hoping to release an update 0.8.10 soon with fixes for these issues. (If you can't wait you could give the branches/getdata-0.8 SVN version, where the fixes have been committed, a try.) Thanks again for the report, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2015-09-16 22:18:24
|
Hi, Thanks for the report. On Wed, Sep 16, 2015 at 05:35:18PM +0000, Schuster, Jochen wrote: > Hi There, > [...] > and I compiled the code from following example: > > http://sourceforge.net/p/getdata/mailman/message/27172486/ I just checked, and this example does work for me on Linux. To check, I simply copied the program from http://sourceforge.net/p/getdata/mailman/message/27185152/ into a file (test.c) and then compiled it with: gcc -o test test.c -lm -lgetdata If I run it, it returns: $ ./test sin 0.1 = 0.0998334 sin 0.1 + log(100) = 4.705 which seems right. > The code is compiling fine and running successfully, but neither data1 nor > data2 contain any data after running the program. After running the program, I end up with: $ ls -l /tmp/dirfile-test/ total 20 -rw-r--r-- 1 user group 8000 Sep 16 14:42 data1 -rw-r--r-- 1 user group 8000 Sep 16 14:42 data2 -rw-r--r-- 1 user group 312 Sep 16 14:32 format > I added error checking > after each operation on the dirfile descriptor and it revealed that I > always obtain the error "Memory allocation error" when calling the > "gd_putdata" function for adding "data1" to the binary file. I played a > bit around with the number format (changing GD_FLOAT64 to GD_INTx after > changing double data1[] to uintx_t data1[]; changing frame to 1000 and > sample to 0; I assumed this could have an influence), but I never get the > expected result. The "Memory allocation error" message indicates a malloc(3) call failed, which is unusual on Linux, with it's optimistic memory allocator. Even if you'e disabled memory overcommittment, the test program shouldn't be using very much memory. > My system is: > > -Ubuntu 14.04 > > -Getdata0.8.9 (installed from source with: ./configure; make; make check; > sudo make install) Were there any errors reported by the test suite (make check)? > Please, could you point me to a direction where the problem may be > (library, compiler settings, system settings) and how I could solve that? > What other information might you need? > > Cheers, > Jochen. If you are able, I'd be interested in the output of your test program with GetData debugging turned on. You'll have to rebuild the library to do this after enabling the debugging messages via the GetData-0.8.9 configure script: ./configure --enable-debug When running with a debugging-enabled copy of the library, the test program (and anything else linked against the library) should spew a lot of debugging messages. Simply capture the output and send me it. A copy of the actual source code you're trying may also be helpful. Another thing to try is using the precompiled GetData packages for Ubuntu prepared by Steve Benton and Barth Netterfield. You can install them from the Launchpad PPA here: https://launchpad.net/~getdata/+archive/ubuntu/ppa This might help you figure out whether the problem is with your test program or your build of the GetData library. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Schuster, J. <js...@hw...> - 2015-09-16 18:08:45
|
Hi There, I am new to getdata and the dirfile format and I was able to create a directory by myself as described here (link below) without calling the getdata library: http://sourceforge.net/p/getdata/mailman/message/29184811/ Nevertheless, I would like to use getdata to write my data into a dirfile and I compiled the code from following example: http://sourceforge.net/p/getdata/mailman/message/27172486/ The code is compiling fine and running successfully, but neither data1 nor data2 contain any data after running the program. I added error checking after each operation on the dirfile descriptor and it revealed that I always obtain the error "Memory allocation error" when calling the "gd_putdata" function for adding "data1" to the binary file. I played a bit around with the number format (changing GD_FLOAT64 to GD_INTx after changing double data1[] to uintx_t data1[]; changing frame to 1000 and sample to 0; I assumed this could have an influence), but I never get the expected result. My system is: -Ubuntu 14.04 -Getdata0.8.9 (installed from source with: ./configure; make; make check; sudo make install) -C/C++ compiler options: w/o -std=c99 or -std=gnu99 -Linked against getdata (libgetdata.a) and mathmatics -Netbeans 7.4 Please, could you point me to a direction where the problem may be (library, compiler settings, system settings) and how I could solve that? What other information might you need? Cheers, Jochen. ----- We invite research leaders and ambitious early career researchers to join us in leading and driving research in key inter-disciplinary themes. Please see www.hw.ac.uk/researchleaders for further information and how to apply. Heriot-Watt University is a Scottish charity registered under charity number SC000278. |
|
From: Joy D. <did...@gm...> - 2015-09-16 05:29:27
|
Great, thanks! On Tue, Sep 15, 2015 at 5:42 PM, D. V. Wiebe <ge...@ke...> wrote: > On Mon, Aug 31, 2015 at 08:46:00PM -0700, Joy Didier wrote: > > Hi, > > Thanks for the explanation. I upgraded to pygetdata 0.8.9 but > > unfortunately the problem persists. The code I wrote in my first email > > still breaks for num_frames = 300000 and larger. > > Joy > > I've been able to reproduce your problem on the latest SVN revision. > I'll see what I can figure out and let you know. > > -don > -- > D. V. Wiebe > ge...@ke... > http://getdata.sourceforge.net/ > -- ------------------------------------------------------------------------------- Joy Didier Columbia Astrophysics Lab Physics PhD Candidate (212) 854-9230 (lab) (212) 854-8121 (fax) -------------------------------------------------------------------------------- |
|
From: D. V. W. <ge...@ke...> - 2015-09-16 00:42:55
|
On Mon, Aug 31, 2015 at 08:46:00PM -0700, Joy Didier wrote: > Hi, > Thanks for the explanation. I upgraded to pygetdata 0.8.9 but > unfortunately the problem persists. The code I wrote in my first email > still breaks for num_frames = 300000 and larger. > Joy I've been able to reproduce your problem on the latest SVN revision. I'll see what I can figure out and let you know. -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Joy D. <did...@gm...> - 2015-09-01 03:46:27
|
Hi, Thanks for the explanation. I upgraded to pygetdata 0.8.9 but unfortunately the problem persists. The code I wrote in my first email still breaks for num_frames = 300000 and larger. Joy On Thu, Aug 27, 2015 at 3:34 PM, D. V. Wiebe <ge...@ke...> wrote: > On Thu, Aug 27, 2015 at 11:05:52AM -0700, Joy Didier wrote: > > Hi, > > I'm using pygetdata version (0, 8, 5, '') and I'm getting a > segmentation > > fault when trying to iteratively write large arrays to a gzipped > dirfile > > using putdata. I'm not sure if this is a bug or if you just can't > write > > iteratively to a gzipped dirfile. > > > > The following code will give a seg fault for num_frames = 300000 and > > larger, but not for num_frames = 100000 and smaller. > > > > Is there a way around this? > > Thanks, > > Joy > > Yeah, it's a bug. There are known problems with the gzip enging in 0.8.5. > There are also known bugs in the python bindings in that version. > Sincere apologies. > > My advice would be to upgrade to the latest version, 0.8.9, which > (hopefully) fixes your problem. If you still have issues when using > the latest version, please report them and we'll investigate further. > > Cheers, > -don > -- > D. V. Wiebe > ge...@ke... > http://getdata.sourceforge.net/ > -- ------------------------------------------------------------------------------- Joy Didier Columbia Astrophysics Lab Physics PhD Candidate (212) 854-9230 (lab) (212) 854-8121 (fax) -------------------------------------------------------------------------------- |
|
From: D. V. W. <ge...@ke...> - 2015-08-27 22:34:57
|
On Thu, Aug 27, 2015 at 11:05:52AM -0700, Joy Didier wrote: > Hi, > I'm using pygetdata version (0, 8, 5, '') and I'm getting a segmentation > fault when trying to iteratively write large arrays to a gzipped dirfile > using putdata. I'm not sure if this is a bug or if you just can't write > iteratively to a gzipped dirfile. > > The following code will give a seg fault for num_frames = 300000 and > larger, but not for num_frames = 100000 and smaller. > > Is there a way around this? > Thanks, > Joy Yeah, it's a bug. There are known problems with the gzip enging in 0.8.5. There are also known bugs in the python bindings in that version. Sincere apologies. My advice would be to upgrade to the latest version, 0.8.9, which (hopefully) fixes your problem. If you still have issues when using the latest version, please report them and we'll investigate further. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Joy D. <did...@gm...> - 2015-08-27 18:06:17
|
Hi,
I'm using pygetdata version (0, 8, 5, '') and I'm getting a segmentation
fault when trying to iteratively write large arrays to a gzipped dirfile
using putdata. I'm not sure if this is a bug or if you just can't write
iteratively to a gzipped dirfile.
The following code will give a seg fault for num_frames = 300000 and
larger, but not for num_frames = 100000 and smaller.
Is there a way around this?
Thanks,
Joy
====================================================================
#! /usr/bin/python
import pygetdata
import os
import shutil
import pylab as pl
# Make new dirfile
dirfile_name = "dirfile_example"
dirfile_path = os.path.join(".", dirfile_name)
if os.path.isdir(dirfile_path):
shutil.rmtree(dirfile_path)
spf = 1
first_frame = 0
num_frames = 300000
d = pygetdata.dirfile(dirfile_path, pygetdata.CREAT | pygetdata.RDWR |
pygetdata.GZIP_ENCODED)
entry = pygetdata.entry(pygetdata.RAW_ENTRY, 'data', 0, (pygetdata.FLOAT32,
spf))
d.add(entry)
d.putdata('data', pl.arange(0.0, num_frames), first_frame=first_frame)
d.close()
# Read and print dirfile field
print '== 1st pass == '
d = pygetdata.dirfile(dirfile_path)
print 'dirfile nframe', d.nframes
read_data = d.getdata('data')
print 'data read:'
print read_data[-10:]
d.close()
# Update the dirfile to write array starting at the end of previous array
first_frame += num_frames
d = pygetdata.dirfile(dirfile_path, pygetdata.CREAT | pygetdata.RDWR |
pygetdata.GZIP_ENCODED)
d.putdata('data', pl.arange(num_frames, 2*num_frames),
first_frame=first_frame)
d.close()
# Read and plot dirfile
print '== 2nd pass =='
d = pygetdata.dirfile(dirfile_path)
print 'dirfile nframe', d.nframes
read_data = d.getdata('data')
print 'data read:'
print read_data[-10:]
d.close()
--
-------------------------------------------------------------------------------
Joy Didier
Columbia Astrophysics Lab
Physics PhD Candidate
(212) 854-9230 (lab)
(212) 854-8121 (fax)
--------------------------------------------------------------------------------
|
|
From: D. V. W. <ge...@ke...> - 2015-08-11 23:48:46
|
On Tue, Aug 11, 2015 at 01:00:11PM -0400, Matthew Petroff wrote: > While running a monitoring script that opened ~1e6 Dirfiles a week, I > determined that there was a memory leak in the pygetdata bindings, since > the script's memory usage would slowly increase, surpassing a gigabyte > after a few weeks. I traced and fixed said leak. Having already built > appropriate debug versions of Python and pygetdata, I also ran the binding > tests with Valgrind and fixed the remaining leaks that were detected (and > one reference counting bug). See attached patch. > > -Matthew Petroff Thanks for the patch! I've committed it with a few tweaks to the 0.8 branch of the GetData repository (revision 985). Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Matthew P. <ge...@mp...> - 2015-08-11 18:47:35
|
I'm resending this message since SourceForge seems to have decided that the patch was the message body last time: While running a monitoring script that opened ~1e6 Dirfiles a week, I determined that there was a memory leak in the pygetdata bindings, since the script's memory usage would slowly increase, surpassing a gigabyte after a few weeks. I traced and fixed said leak. Having already built appropriate debug versions of Python and pygetdata, I also ran the binding tests with Valgrind and fixed the remaining leaks that were detected (and one reference counting bug). See attached patch. -Matthew Petroff |
|
From: Matthew P. <ge...@mp...> - 2015-08-11 17:31:25
|
Index: getdata/bindings/python/pydirfile.c
===================================================================
--- getdata/bindings/python/pydirfile.c (revision 984)
+++ getdata/bindings/python/pydirfile.c (working copy)
@@ -124,6 +124,8 @@
gd_close(self->D);
free(self->verbose_prefix);
+ Py_XDECREF(self->callback_data);
+ free(self);
dreturnvoid();
}
@@ -1479,7 +1481,7 @@
{
char *keywords[] = { "field_code", NULL };
const char *field_code;
- const char *filename;
+ char *filename;
PyObject *pyobj;
dtrace("%p, %p, %p", self, args, keys);
@@ -1492,6 +1494,7 @@
}
filename = gd_raw_filename(self->D, field_code);
+ free(filename);
PYGD_CHECK_ERROR(self->D, NULL);
Index: getdata/bindings/python/pyentry.c
===================================================================
--- getdata/bindings/python/pyentry.c (revision 984)
+++ getdata/bindings/python/pyentry.c (working copy)
@@ -70,6 +70,7 @@
gd_free_entry_strings(self->E);
free(self->E);
+ free(self);
dreturnvoid();
}
Index: getdata/bindings/python/pyfragment.c
===================================================================
--- getdata/bindings/python/pyfragment.c (revision 984)
+++ getdata/bindings/python/pyfragment.c (working copy)
@@ -27,6 +27,7 @@
dtrace("%p", self);
Py_XDECREF(self->dirfile);
+ free(self);
dreturnvoid();
}
@@ -309,6 +310,7 @@
dtrace("%p, %p", self, closure);
gd_fragment_affixes(self->dirfile->D, self->n, &prefix, &suffix);
+ free(suffix);
PYGD_CHECK_ERROR(self->dirfile->D, NULL);
@@ -319,6 +321,7 @@
}
pyobj = PyString_FromString(prefix);
+ free(prefix);
dreturn("%p", pyobj);
return pyobj;
@@ -355,6 +358,7 @@
dtrace("%p, %p", self, closure);
gd_fragment_affixes(self->dirfile->D, self->n, &prefix, &suffix);
+ free(prefix);
PYGD_CHECK_ERROR(self->dirfile->D, NULL);
@@ -365,6 +369,7 @@
}
pyobj = PyString_FromString(suffix);
+ free(suffix);
dreturn("%p", pyobj);
return pyobj; |
|
From: D. V. W. <ge...@ke...> - 2015-08-05 00:26:39
|
GetData 0.8.9 has been released. You may obtain it from your local sourceforge mirror: https://sourceforge.net/projects/getdata/files/getdata/0.8.9/ This is another bugfix release which fixes issues found after the release of GetData-0.8.7 (and GetData-0.8.8). Full release notes are provided at the bottom of the release page linked above as well as in the file called NEWS included in the GetData release archives. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2015-05-14 00:34:50
|
Well, 0.8.7 didn't go so well; here's hoping 0.8.8 will do better. GetData-0.8.8 has been released. As usual, it may be downloaded from your local SourceForge mirror: https://sourceforge.net/projects/getdata/files/getdata/0.8.8/ This release makes it possible again to compile GetData on 32-bit systems. It is otherwise the same as GetData-0.8.7. Full release notes for both the 0.8.8 and the 0.8.7 releases are provided at the bottom of the page linked above, as well as in the file called NEWS included in the GetData release archives. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Christian T. <ct...@op...> - 2015-05-12 20:02:43
|
Am Dienstag, 12. Mai 2015, 12:27:50 schrieb D. V. Wiebe: > On Tue, May 12, 2015 at 08:49:54PM +0200, Christian Trippe wrote: > > Hi Dan, > > > > I tried to build GetData 0.8.7 for openSUSE, however it always fails to > > build on i586 (x86_64 work fine). This is true for different versions of > > openSUSE (see > > https://build.opensuse.org/package/show/home:christiantrippe:branches:KDE > > :Extra/getdata ) > > Lovely. > > Thanks for the report. It looks like the 32-bit machine I used to test > on got switched to 64-bits when I wasn't looking. I think I know what > went wrong. I'll look into fixing. > > Is there some way I can view the config.log that the configure script > generated during these failed builds? > I enhanced the build script to also put the config.log in the build ouput, please find it attached. Regards Christian |
|
From: D. V. W. <ge...@ke...> - 2015-05-12 19:27:59
|
On Tue, May 12, 2015 at 08:49:54PM +0200, Christian Trippe wrote: > Hi Dan, > > I tried to build GetData 0.8.7 for openSUSE, however it always fails to build > on i586 (x86_64 work fine). This is true for different versions of openSUSE (see > https://build.opensuse.org/package/show/home:christiantrippe:branches:KDE:Extra/getdata > ) Lovely. Thanks for the report. It looks like the 32-bit machine I used to test on got switched to 64-bits when I wasn't looking. I think I know what went wrong. I'll look into fixing. Is there some way I can view the config.log that the configure script generated during these failed builds? Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: Christian T. <ct...@op...> - 2015-05-12 18:52:12
|
Hi Dan, I tried to build GetData 0.8.7 for openSUSE, however it always fails to build on i586 (x86_64 work fine). This is true for different versions of openSUSE (see https://build.opensuse.org/package/show/home:christiantrippe:branches:KDE:Extra/getdata ) Hopefully relevant part is for example: [ 73s] libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -Wall -Wextra - DGETDATA_MODULEDIR=\"/usr/lib/getdata\" -fomit-frame-pointer -fmessage- length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -c flush.c -fPIC -DPIC -o .libs/flush.o [ 73s] libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -Wall -Wextra - DGETDATA_MODULEDIR=\"/usr/lib/getdata\" -fomit-frame-pointer -fmessage- length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -c fpos.c -fPIC -DPIC -o .libs/fpos.o [ 74s] /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 - DHAVE_CONFIG_H -I. -Wall -Wextra - DGETDATA_MODULEDIR="\"/usr/lib/getdata\"" -fomit-frame-pointer -fmessage- length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -c -o fragment.lo fragment.c [ 74s] libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -Wall -Wextra - DGETDATA_MODULEDIR=\"/usr/lib/getdata\" -fomit-frame-pointer -fmessage- length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -c fragment.c -fPIC -DPIC -o .libs/fragment.o [ 74s] /tmp/cc1O1Lx2.s: Assembler messages: [ 74s] /tmp/cc1O1Lx2.s:622: Error: symbol `gd_tell64' is already defined [ 74s] /tmp/cc1O1Lx2.s:1817: Error: symbol `gd_seek64' is already defined [ 74s] /tmp/cc7OIvHX.s: Assembler messages: [ 74s] /tmp/cc7OIvHX.s:1325: Error: symbol `gd_alter_frameoffset64' is already defined [ 74s] /tmp/cc7OIvHX.s:1365: Error: symbol `gd_frameoffset64' is already defined [ 74s] /tmp/cc7OIvHX.s:2235: Makefile:850: recipe for target 'fpos.lo' failed [ 74s] make[2]: *** [fpos.lo] Error 1 [ 74s] make[2]: *** Waiting for unfinished jobs.... [ 74s] Error: symbol `gd_eof64' is already defined [ 74s] /tmp/cc7OIvHX.s:2405: Error: symbol `gd_bof64' is already defined [ 74s] Makefile:850: recipe for target 'flimits.lo' failed [ 74s] make[2]: *** [flimits.lo] Error 1 [ 74s] make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/getdata-0.8.7/src' [ 74s] Makefile:622: recipe for target 'all' failed [ 74s] make[1]: *** [all] Error 2 [ 74s] make[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/getdata-0.8.7/src' [ 74s] Makefile:551: recipe for target 'all-recursive' failed [ 74s] make: *** [all-recursive] Error 1 [ 74s] error: Bad exit status from /var/tmp/rpm-tmp.ELVXIP (%build) The full build log can be found at https://build.opensuse.org/package/live_build_log/home:christiantrippe:branches:KDE:Extra/getdata/openSUSE_13.2/i586 Do you have an idea, what the problem is? Regards Christian |
|
From: D. V. W. <ge...@ke...> - 2015-05-11 22:25:04
|
Greetings, GetData-0.8.7 has been released. As usual, it may be downloaded from your local SourceForge mirror: https://sourceforge.net/projects/getdata/files/getdata/0.8.7/ This release fixes a few issues found after the release of GetData-0.8.6, particularly with the Python bindings (pygetdata) and some substantial fixes to the sample-index encoding (SIE) framework. Full release notes are provided at the bottom of the release page linked above as well as in the file called NEWS included in the GetData release archives. Cheers, -dvw -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
|
From: D. V. W. <ge...@ke...> - 2015-05-07 00:15:32
|
On Wed, May 06, 2015 at 10:12:27AM -0700, Seth Hillbrand wrote: > Hi Don- > > I believe that there are a couple bugs in _GD_MungeCode when metafields are > used. This is crashing in my use case but as it is a memory error, I > haven't gotten the test case to raise an error independently of valgrind. > With the attached patches, the valgrind error goes away, all tests pass and > my use case is also stable. > > First: > The slash pointer points to the metafield after the '/' marker but len_sub > includes the '/' in its length calculation. While correct for allocation, > when nptr is incremented after the slash pointer is concatenated, it moves > the pointer beyond the end of the allocated string. The NULL-termination > then writes past the allocated memory by 1 byte. > > Second: > The line `new_code[len_newpx + len + len_newsx] = '/';` duplicates the > behavior of the line immediately before it without name spaces and appears > to write into the middle of the new_code string when name spaces are in use. > > I'm attaching a proposed patch for name.c as well as a revision to > test/madd_string.c that exposes the first issue when run using valgrind. > Do let me know if I am misunderstanding the intended behavior of the code > in the second issue. > > Regards- > Seth Hi Seth, Thanks for the report, I'll take a look and commit your patch. I appreciate the test-suite changes. At the moment, trunk isn't very stable, and probably buggy as we work towards 0.9. It's likely to get worse before it gets better. Do you see these problems in GetData-0.8 as well? There's a stable branch for work on GetData-0.8 at branches/getdata-0.8. It was forked after GetData-0.8.5, although there has been subsequent cross-polination between the branch and trunk. Cheers, -don -- D. V. Wiebe ge...@ke... http://getdata.sourceforge.net/ |
1 message has been excluded from this view by a project administrator.
