octave-dev

[OctDev] new package ncArray 1.0.0

[Alexander B. <bar...@gm...>]

Dear all,

I have released a new package called ncArray for manipulating netcdf
files. It depends on the functions ncread, ncwrite and ncinfo from
octcdf. Maybe someday, these functions will be implemented in octave
(as they are in matlab) and ncArray will be independent from octcdf.

The package allows to access a single or a collection of NetCDF files
(or OPeNDAP URLs) as a multi-dimensional array.
For a collection of NetCDF files, the variables are concatenated along
a given dimension similar to the command cat. The ncArray object
contains only a reference to the underlying files. The data is loaded
only, if the array is indexed.

For example, if the current directory containts file1.nc, file2.nc and
file3.nc and each file contains the 20 x 30 variable temp, then:

A = ncCatArray(3,'file*.nc','temp')

would create and array A of size 20 x 30 x 3 without actually loading
the files. Only when the array is index e.g., A(:,:,2:3), the file
file2.nc and file3.nc are loaded.

Also, reduction operators are implement. For example, sum(A,3) would
compute the sum over the 3rd dimension and sequentially loading all
files.

ncArray allows thus to write code for dataset which are too big to be
loaded in memory and still to write code that are independent on the
way the data is stored and how a data set is split across files.

More example how to use this package are available here:
http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray

Best rega

Re: [OctDev] new package ncArray 1.0.0

[Alexander B. <bar...@gm...>]

Sorry for poor spelling in my last email. The email was sent
inadvertently before I could check it.
Source packages are here:

http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-1.0.0.tar.gz
http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-html.tar.gz

Can somebody update the server?
Best regards,
Alex


On Wed, Jul 18, 2012 at 1:30 PM, Alexander Barth
<bar...@gm...> wrote:
> Dear all,
>
> I have released a new package called ncArray for manipulating netcdf
> files. It depends on the functions ncread, ncwrite and ncinfo from
> octcdf. Maybe someday, these functions will be implemented in octave
> (as they are in matlab) and ncArray will be independent from octcdf.
>
> The package allows to access a single or a collection of NetCDF files
> (or OPeNDAP URLs) as a multi-dimensional array.
> For a collection of NetCDF files, the variables are concatenated along
> a given dimension similar to the command cat. The ncArray object
> contains only a reference to the underlying files. The data is loaded
> only, if the array is indexed.
>
> For example, if the current directory containts file1.nc, file2.nc and
> file3.nc and each file contains the 20 x 30 variable temp, then:
>
> A = ncCatArray(3,'file*.nc','temp')
>
> would create and array A of size 20 x 30 x 3 without actually loading
> the files. Only when the array is index e.g., A(:,:,2:3), the file
> file2.nc and file3.nc are loaded.
>
> Also, reduction operators are implement. For example, sum(A,3) would
> compute the sum over the 3rd dimension and sequentially loading all
> files.
>
> ncArray allows thus to write code for dataset which are too big to be
> loaded in memory and still to write code that are independent on the
> way the data is stored and how a data set is split across files.
>
> More example how to use this package are available here:
> http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray
>
> Best rega

Re: [OctDev] new package ncArray 1.0.0

[Michael G. <mic...@gm...>]

On Wed, Jul 18, 2012 at 12:44 PM, Alexander Barth <bar...@gm...
> wrote:

> Sorry for poor spelling in my last email. The email was sent
> inadvertently before I could check it.
> Source packages are here:
>
>
> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-1.0.0.tar.gz
> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-html.tar.gz
>
> Can somebody update the server?
> Best regards,
> Alex
>
>
> On Wed, Jul 18, 2012 at 1:30 PM, Alexander Barth
> <bar...@gm...> wrote:
> > Dear all,
> >
> > I have released a new package called ncArray for manipulating netcdf
> > files. It depends on the functions ncread, ncwrite and ncinfo from
> > octcdf. Maybe someday, these functions will be implemented in octave
> > (as they are in matlab) and ncArray will be independent from octcdf.
> >
> > The package allows to access a single or a collection of NetCDF files
> > (or OPeNDAP URLs) as a multi-dimensional array.
> > For a collection of NetCDF files, the variables are concatenated along
> > a given dimension similar to the command cat. The ncArray object
> > contains only a reference to the underlying files. The data is loaded
> > only, if the array is indexed.
> >
> > For example, if the current directory containts file1.nc, file2.nc and
> > file3.nc and each file contains the 20 x 30 variable temp, then:
> >
> > A = ncCatArray(3,'file*.nc','temp')
> >
> > would create and array A of size 20 x 30 x 3 without actually loading
> > the files. Only when the array is index e.g., A(:,:,2:3), the file
> > file2.nc and file3.nc are loaded.
> >
> > Also, reduction operators are implement. For example, sum(A,3) would
> > compute the sum over the 3rd dimension and sequentially loading all
> > files.
> >
> > ncArray allows thus to write code for dataset which are too big to be
> > loaded in memory and still to write code that are independent on the
> > way the data is stored and how a data set is split across files.
> >
> > More example how to use this package are available here:
> > http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray
> >
> > Best rega
>
>
 Is there a reason not to integrate these functions into the octcdf package
directly?

Michael.

Re: [OctDev] new package ncArray 1.0.0

[Alexander B. <bar...@gm...>]

On Wed, Jul 18, 2012 at 3:14 PM, Michael Goffioul
<mic...@gm...> wrote:
> On Wed, Jul 18, 2012 at 12:44 PM, Alexander Barth
> <bar...@gm...> wrote:
>>
>> Sorry for poor spelling in my last email. The email was sent
>> inadvertently before I could check it.
>> Source packages are here:
>>
>>
>> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-1.0.0.tar.gz
>>
>> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-html.tar.gz
>>
>> Can somebody update the server?
>> Best regards,
>> Alex
>>
>>
>> On Wed, Jul 18, 2012 at 1:30 PM, Alexander Barth
>> <bar...@gm...> wrote:
>> > Dear all,
>> >
>> > I have released a new package called ncArray for manipulating netcdf
>> > files. It depends on the functions ncread, ncwrite and ncinfo from
>> > octcdf. Maybe someday, these functions will be implemented in octave
>> > (as they are in matlab) and ncArray will be independent from octcdf.
>> >
>> > The package allows to access a single or a collection of NetCDF files
>> > (or OPeNDAP URLs) as a multi-dimensional array.
>> > For a collection of NetCDF files, the variables are concatenated along
>> > a given dimension similar to the command cat. The ncArray object
>> > contains only a reference to the underlying files. The data is loaded
>> > only, if the array is indexed.
>> >
>> > For example, if the current directory containts file1.nc, file2.nc and
>> > file3.nc and each file contains the 20 x 30 variable temp, then:
>> >
>> > A = ncCatArray(3,'file*.nc','temp')
>> >
>> > would create and array A of size 20 x 30 x 3 without actually loading
>> > the files. Only when the array is index e.g., A(:,:,2:3), the file
>> > file2.nc and file3.nc are loaded.
>> >
>> > Also, reduction operators are implement. For example, sum(A,3) would
>> > compute the sum over the 3rd dimension and sequentially loading all
>> > files.
>> >
>> > ncArray allows thus to write code for dataset which are too big to be
>> > loaded in memory and still to write code that are independent on the
>> > way the data is stored and how a data set is split across files.
>> >
>> > More example how to use this package are available here:
>> > http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray
>> >
>> > Best rega
>>
>
>  Is there a reason not to integrate these functions into the octcdf package
> directly?
>
> Michael.
>

Yes, ncArray contains pure m-scripts that work also with matlab.
octcdf is mostly written in C++ code that are specific to octave. A
matlab user thus only needs the package ncArray.

Best regards,
Alex

Re: [OctDev] new package ncArray 1.0.0

[Michael G. <mic...@gm...>]

On Wed, Jul 18, 2012 at 9:46 PM, Alexander Barth
<bar...@gm...>wrote:

> On Wed, Jul 18, 2012 at 3:14 PM, Michael Goffioul
> <mic...@gm...> wrote:
> > On Wed, Jul 18, 2012 at 12:44 PM, Alexander Barth
> > <bar...@gm...> wrote:
> >>
> >> Sorry for poor spelling in my last email. The email was sent
> >> inadvertently before I could check it.
> >> Source packages are here:
> >>
> >>
> >>
> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-1.0.0.tar.gz
> >>
> >>
> http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-html.tar.gz
> >>
> >> Can somebody update the server?
> >> Best regards,
> >> Alex
> >>
> >>
> >> On Wed, Jul 18, 2012 at 1:30 PM, Alexander Barth
> >> <bar...@gm...> wrote:
> >> > Dear all,
> >> >
> >> > I have released a new package called ncArray for manipulating netcdf
> >> > files. It depends on the functions ncread, ncwrite and ncinfo from
> >> > octcdf. Maybe someday, these functions will be implemented in octave
> >> > (as they are in matlab) and ncArray will be independent from octcdf.
> >> >
> >> > The package allows to access a single or a collection of NetCDF files
> >> > (or OPeNDAP URLs) as a multi-dimensional array.
> >> > For a collection of NetCDF files, the variables are concatenated along
> >> > a given dimension similar to the command cat. The ncArray object
> >> > contains only a reference to the underlying files. The data is loaded
> >> > only, if the array is indexed.
> >> >
> >> > For example, if the current directory containts file1.nc, file2.ncand
> >> > file3.nc and each file contains the 20 x 30 variable temp, then:
> >> >
> >> > A = ncCatArray(3,'file*.nc','temp')
> >> >
> >> > would create and array A of size 20 x 30 x 3 without actually loading
> >> > the files. Only when the array is index e.g., A(:,:,2:3), the file
> >> > file2.nc and file3.nc are loaded.
> >> >
> >> > Also, reduction operators are implement. For example, sum(A,3) would
> >> > compute the sum over the 3rd dimension and sequentially loading all
> >> > files.
> >> >
> >> > ncArray allows thus to write code for dataset which are too big to be
> >> > loaded in memory and still to write code that are independent on the
> >> > way the data is stored and how a data set is split across files.
> >> >
> >> > More example how to use this package are available here:
> >> > http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray
> >> >
> >> > Best rega
> >>
> >
> >  Is there a reason not to integrate these functions into the octcdf
> package
> > directly?
> >
> > Michael.
> >
>
> Yes, ncArray contains pure m-scripts that work also with matlab.
> octcdf is mostly written in C++ code that are specific to octave. A
> matlab user thus only needs the package ncArray.
>

OK, thanks. I was just asking, because from a packager point of view, the
proliferation of single niche-specific packages is a pain and increases the
workload. For instance, I would also see ncArray to fit into the io package.


>
> Best regards,
> Alex
>

Re: [OctDev] new package ncArray 1.0.0

[Philip N. <pr....@hc...>]

Michael Goffioul wrote:
> On Wed, Jul 18, 2012 at 9:46 PM, Alexander Barth
> <bar...@gm... <mailto:bar...@gm...>> wrote:
>
>     On Wed, Jul 18, 2012 at 3:14 PM, Michael Goffioul
>     <mic...@gm... <mailto:mic...@gm...>> wrote:
>      > On Wed, Jul 18, 2012 at 12:44 PM, Alexander Barth
>      > <bar...@gm... <mailto:bar...@gm...>> wrote:
>      >>
>      >> Sorry for poor spelling in my last email. The email was sent
>      >> inadvertently before I could check it.
>      >> Source packages are here:
>      >>
>      >>
>      >>
>     http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-1.0.0.tar.gz
>      >>
>      >>
>     http://modb.oce.ulg.ac.be/mediawiki/upload/Alex/ncArray/ncarray-html.tar.gz
>      >>
>      >> Can somebody update the server?
>      >> Best regards,
>      >> Alex
>      >>
>      >>
>      >> On Wed, Jul 18, 2012 at 1:30 PM, Alexander Barth
>      >> <bar...@gm... <mailto:bar...@gm...>>
>     wrote:
>      >> > Dear all,
>      >> >
>      >> > I have released a new package called ncArray for manipulating
>     netcdf
>      >> > files. It depends on the functions ncread, ncwrite and ncinfo from
>      >> > octcdf. Maybe someday, these functions will be implemented in
>     octave
>      >> > (as they are in matlab) and ncArray will be independent from
>     octcdf.
>      >> >
>      >> > The package allows to access a single or a collection of
>     NetCDF files
>      >> > (or OPeNDAP URLs) as a multi-dimensional array.
>      >> > For a collection of NetCDF files, the variables are
>     concatenated along
>      >> > a given dimension similar to the command cat. The ncArray object
>      >> > contains only a reference to the underlying files. The data is
>     loaded
>      >> > only, if the array is indexed.
>      >> >
>      >> > For example, if the current directory containts file1.nc
>     <http://file1.nc>, file2.nc <http://file2.nc> and
>      >> > file3.nc <http://file3.nc> and each file contains the 20 x 30
>     variable temp, then:
>      >> >
>      >> > A = ncCatArray(3,'file*.nc','temp')
>      >> >
>      >> > would create and array A of size 20 x 30 x 3 without actually
>     loading
>      >> > the files. Only when the array is index e.g., A(:,:,2:3), the file
>      >> > file2.nc <http://file2.nc> and file3.nc <http://file3.nc> are
>     loaded.
>      >> >
>      >> > Also, reduction operators are implement. For example, sum(A,3)
>     would
>      >> > compute the sum over the 3rd dimension and sequentially
>     loading all
>      >> > files.
>      >> >
>      >> > ncArray allows thus to write code for dataset which are too
>     big to be
>      >> > loaded in memory and still to write code that are independent
>     on the
>      >> > way the data is stored and how a data set is split across files.
>      >> >
>      >> > More example how to use this package are available here:
>      >> > http://modb.oce.ulg.ac.be/mediawiki/index.php/NcArray
>      >> >
>      >> > Best rega
>      >>
>      >
>      >  Is there a reason not to integrate these functions into the
>     octcdf package
>      > directly?
>      >
>      > Michael.
>      >
>
>     Yes, ncArray contains pure m-scripts that work also with matlab.
>     octcdf is mostly written in C++ code that are specific to octave. A
>     matlab user thus only needs the package ncArray.
>
>
> OK, thanks. I was just asking, because from a packager point of view,
> the proliferation of single niche-specific packages is a pain and
> increases the workload. For instance, I would also see ncArray to fit
> into the io package.

Michael,

You mean (Linux) distro packagers? Binary Octave distribution packagers?

If so, I see your point, but packagers and OF package maintainers may 
have a few opposite interests.

The io package may not be a good example here as by its very nature it 
constitutes a sort of dumping ground for all kinds of I/O code. OTOH 
because of that, it does have some fine examples for the issues below:


(A) Right now in the io package there are already several functions that 
I cannot maintain as I'm more or less clueless about:
(1) What purpose they serve (Nastran PCH?);
(2) Because of (1), how they relate to other OF or core Octave functions;
(3) How they actually work. Several contributed function files lack even 
the most basic code documentation (xmlread/-write), let alone formal 
tests (everything save the spreadsheet stuff which has test scripts). 
Just don't mention documentation for users.

Adding in functions maintained by someone else might work but I have 
trouble imagining how that would help reduce total workload (for who?).
In contrast, I think smaller packages would help distribute workload for 
maintainers in a more natural way. Plus, if a package maintainer 
disappears and the package becomes orphaned, it would be isolated and 
not silently pollute better maintained packages with unmaintained and 
bit-rotting code.

What may help packagers here is some development setup that allows 
automated building all OF packages with just "one or two" commands.
I'm afraid most of such a setup would need to be at the packagers' side.


(B) Another issue is formed by the growing dependency chains of OF 
packages. As to io, I'd frown on having it also depend on the octcdf 
package. It would imply that the statistics package, which since a few 
months depends on io (because of just one function), depends on the 
octcdf package in turn. Etc.
I suppose these dependency chains increase complexity, viz. workload, 
for packagers. Am I wrong here?


So, contrary to your packagers' POV, as package maintainer I'd rather 
have a wider collection of smaller OF packages, perhaps even with 
overlapping functionality, but with clear distinctions between 
(sub-)purpose, dependencies and between maintained and unmaintained code.

Philip

Re: [OctDev] new package ncArray 1.0.0

[Carnë D. <car...@gm...>]

On 18 July 2012 07:30, Alexander Barth <bar...@gm...> wrote:
> I have released a new package called ncArray for manipulating netcdf
> files. It depends on the functions ncread, ncwrite and ncinfo from
> octcdf.

Now uploaded to the octave-forge servers. Have fun

Carnë

[OctDev] valgrind for oct files?

[dirk <dir...@co...>]

Hello all - 
I'm trying to learn valgrind for oct files... can anyone share a working
example I could study?
Hope so, thanks
- dirk

Re: [OctDev] valgrind for oct files?

[Jordi G. H. <jo...@oc...>]

On 22 August 2012 07:11, dirk <dir...@co...> wrote:
> I'm trying to learn valgrind for oct files... can anyone share a working
> example I could study?

Off the top of my head...

Compile your oct file with debug symols:

    CXXFLAGS=-g mkoctfile foo.cc

Write some m-script that executes code from your octfile

    echo "foo();" > myscript.m

Run Octave with valgrind:

    valgrind octave -q myscript.m

HTH,
- Jordi G. H.

Re: [OctDev] valgrind for oct files?

[Andy B. <and...@gm...>]

> On Wed, 2012-08-22 at 12:21 +0100, Andy Buckle wrote:
>> On 22 August 2012 12:11, dirk <dir...@co...> wrote:
>> > Hello all -
>> > I'm trying to learn valgrind for oct files... can anyone share a working
>> > example I could study?
>> > Hope so, thanks
>> > - dirk
>>
>> Invoke octave from the command line of your OS, in a way that calls
>> your oct-file function. Run the whole of octave through valgrind. You
>> will probably want to generate suppressions by doing this and not
>> calling your oct-file.
>>
>> In the past i have also used pre-processor to change around the code,
>> so i can call parts of my code from the OS, instead of from Octave.
>> This is much faster than running the whole of Octave through valgrind.
>>
>> Which bit do you want examples of?

On 22 August 2012 20:58, dirk <dir...@co...> wrote:
> thanks for the tips Andy.
> I suspect my issues are just newbie problems, so any & all advice is
> helpful.  Here's an example point of confusion:
>
> $ valgrind octave --eval exit &> octave_grind.txt
>
> gets me this in the .txt:
>
> ==17994== LEAK SUMMARY:
> ==17994==    definitely lost: 3,108 bytes in 116 blocks
> ==17994==    indirectly lost: 27,994 bytes in 201 blocks
> ==17994==      possibly lost: 659,439 bytes in 3,101 blocks
> ==17994==    still reachable: 34,585,196 bytes in 6,407 blocks
> ==17994==         suppressed: 0 bytes in 0 blocks
>
> which seems to be telling that I lose 3k by just starting and stopping
> octave... hard to believe.
>
> If it matters, I'm running octave 3.4.2 under linux, installed via
> apt-get.
>
> - dirk

(ooops, I forgot to reply-all)

It's common to see that. I get the same kind of thing. It is probably
Octave (or the libraries it depends on).

Run that and make a suppression file to exclude the things that are
not your fault. Then run your code through valgrind in a similar way,
and you should only see things that are your fault. Write a bit of bad
code on purpose to see what you get.

Andy



-- 
/* andy buckle */

Re: [OctDev] valgrind for oct files?

[Daniel J S. <dan...@ie...>]

On 08/22/2012 03:30 PM, Andy Buckle wrote:
>> On Wed, 2012-08-22 at 12:21 +0100, Andy Buckle wrote:
>>> On 22 August 2012 12:11, dirk<dir...@co...>  wrote:
>>>> Hello all -
>>>> I'm trying to learn valgrind for oct files... can anyone share a working
>>>> example I could study?
>>>> Hope so, thanks
>>>> - dirk
>>>
>>> Invoke octave from the command line of your OS, in a way that calls
>>> your oct-file function. Run the whole of octave through valgrind. You
>>> will probably want to generate suppressions by doing this and not
>>> calling your oct-file.
>>>
>>> In the past i have also used pre-processor to change around the code,
>>> so i can call parts of my code from the OS, instead of from Octave.
>>> This is much faster than running the whole of Octave through valgrind.
>>>
>>> Which bit do you want examples of?
>
> On 22 August 2012 20:58, dirk<dir...@co...>  wrote:
>> thanks for the tips Andy.
>> I suspect my issues are just newbie problems, so any&  all advice is
>> helpful.  Here's an example point of confusion:
>>
>> $ valgrind octave --eval exit&>  octave_grind.txt
>>
>> gets me this in the .txt:
>>
>> ==17994== LEAK SUMMARY:
>> ==17994==    definitely lost: 3,108 bytes in 116 blocks
>> ==17994==    indirectly lost: 27,994 bytes in 201 blocks
>> ==17994==      possibly lost: 659,439 bytes in 3,101 blocks
>> ==17994==    still reachable: 34,585,196 bytes in 6,407 blocks
>> ==17994==         suppressed: 0 bytes in 0 blocks
>>
>> which seems to be telling that I lose 3k by just starting and stopping
>> octave... hard to believe.
>>
>> If it matters, I'm running octave 3.4.2 under linux, installed via
>> apt-get.
>>
>> - dirk
>
> (ooops, I forgot to reply-all)
>
> It's common to see that. I get the same kind of thing. It is probably
> Octave (or the libraries it depends on).

The version with the bundle I'm using (Fedora 14...they are past that 
now) is also 3.4.2.  I had problems with it crashing after just a few 
operations.  I ran the valgrind command you list above and get:

==5329== LEAK SUMMARY:
==5329==    definitely lost: 268 bytes in 19 blocks
==5329==    indirectly lost: 0 bytes in 0 blocks
==5329==      possibly lost: 102,948 bytes in 2,490 blocks
==5329==    still reachable: 1,848,332 bytes in 5,439 blocks
==5329==         suppressed: 0 bytes in 0 blocks

Could be Octave.  Could be maintainers using out-of-date library.  Hard 
to tell.  In any case, the latest Octave in the repository comes out 
with 0 bytes lost for the same test.

Dan