ojalgo-user

[ojAlgo-user] Populating matrices

[Nigel S. <nig...@gm...>]

Hi, I wonder if someone could please comment on the proper way to
populate blank matricies, with memory efficiency in mind.

The example I have is loading a CSV file into the matrix. Previously I
used JAMA:
1) Created the matrix (new Matrix(rowCount, colCount))
2) Iterate over the rows:
2.1)   Iterate over the columns
2.1.1) Write to the array (currentMatrix.set(row, j,
Double.parseDouble(cols[remainingCols.get(j)]));)

I adapted this to ojalgo, so now I use
1) PrimitiveMatrix.FACTORY.buildZero(rowCount, colCount)
...
2.1.1) currentMatrix = currentMatrix.set(row, j,
Double.parseDouble(cols[colsOfInterest[j]]));

This is very slow, in that the JAMA version takes 0.3 s to run, and
the ojalgo version takes 30+ seconds to run. Essentially because the
matrix set method seems to make a copy of the matrix, which it
returns. This does seem at odds with the java doc which indicates that
the physical stores are mutable, so I would assume the matrix should
also be mutable.

The other option appears to be to create a double array and copy it
into the matrix with the factory, but this will mean, for a short
time, there will be two copies of a potentially large array in memory.

Any advice would be most helpful.

Cheers
Nigel

Re: [ojAlgo-user] Populating matrices

[Anders P. <an...@op...>]

On 13 nov 2008, at 11.32, Nigel Sim wrote:

> Hi, I wonder if someone could please comment on the proper way to
> populate blank matricies, with memory efficiency in mind.
>
> The example I have is loading a CSV file into the matrix. Previously I
> used JAMA:
> 1) Created the matrix (new Matrix(rowCount, colCount))
> 2) Iterate over the rows:
> 2.1)   Iterate over the columns
> 2.1.1) Write to the array (currentMatrix.set(row, j,
> Double.parseDouble(cols[remainingCols.get(j)]));)
>
> I adapted this to ojalgo, so now I use
> 1) PrimitiveMatrix.FACTORY.buildZero(rowCount, colCount)
> ....
> 2.1.1) currentMatrix = currentMatrix.set(row, j,
> Double.parseDouble(cols[colsOfInterest[j]]));
>
> This is very slow, in that the JAMA version takes 0.3 s to run, and
> the ojalgo version takes 30+ seconds to run. Essentially because the
> matrix set method seems to make a copy of the matrix, which it
> returns. This does seem at odds with the java doc which indicates that
> the physical stores are mutable, so I would assume the matrix should
> also be mutable.


Didn't know it was that slow... I rarely use the set-method in  
BasicMatrix, and when I do it's only to set a few (a small number) of  
elements.

The BasicMatrix interface doesn't force implementations to be  
mutable, and the implementations BigMatrix, PrimitiveMatrix and  
ComplexMatrix are immutable. When you call set(...) nothing is  
copied, but a couple of new instances are created - a new BasicMatrix  
and a new MatrixStore - and they contain the old/previous MatrixStore.

I'd do it like this:

PrimitiveDenseStore.FACTORY.makeEmpty(rowCount, colCount);

and then loop to set the elements the way you did before.

You do not have to create a BasicMatrix instance, and if you worry  
very much about memory consumption then you probably shouldn't.


> The other option appears to be to create a double array and copy it
> into the matrix with the factory, but this will mean, for a short
> time, there will be two copies of a potentially large array in memory.
>
> Any advice would be most helpful.


Currently the only way to create a new BasicMatrix instance without  
using the set-method and without any "unnecessary" copying is to  
create a MatrixStore and then instantiate a BasicMatrix  
implementation by calling a constructor directly. The constructors in  
BigMatrix, PrimitiveMatrix and ComplexMatrix do not copy anything.


/Anders

Re: [ojAlgo-user] Populating matrices

[Nigel S. <nig...@gm...>]

Thanks Anders, works like a charm. We're back down to 0.3s.

Cheers
Nigel

2008/11/13 Anders Peterson <an...@op...>:
> On 13 nov 2008, at 11.32, Nigel Sim wrote:
>
>> Hi, I wonder if someone could please comment on the proper way to
>> populate blank matricies, with memory efficiency in mind.
>>
>> The example I have is loading a CSV file into the matrix. Previously I
>> used JAMA:
>> 1) Created the matrix (new Matrix(rowCount, colCount))
>> 2) Iterate over the rows:
>> 2.1)   Iterate over the columns
>> 2.1.1) Write to the array (currentMatrix.set(row, j,
>> Double.parseDouble(cols[remainingCols.get(j)]));)
>>
>> I adapted this to ojalgo, so now I use
>> 1) PrimitiveMatrix.FACTORY.buildZero(rowCount, colCount)
>> ....
>> 2.1.1) currentMatrix = currentMatrix.set(row, j,
>> Double.parseDouble(cols[colsOfInterest[j]]));
>>
>> This is very slow, in that the JAMA version takes 0.3 s to run, and
>> the ojalgo version takes 30+ seconds to run. Essentially because the
>> matrix set method seems to make a copy of the matrix, which it
>> returns. This does seem at odds with the java doc which indicates that
>> the physical stores are mutable, so I would assume the matrix should
>> also be mutable.
>
>
> Didn't know it was that slow... I rarely use the set-method in
> BasicMatrix, and when I do it's only to set a few (a small number) of
> elements.
>
> The BasicMatrix interface doesn't force implementations to be
> mutable, and the implementations BigMatrix, PrimitiveMatrix and
> ComplexMatrix are immutable. When you call set(...) nothing is
> copied, but a couple of new instances are created - a new BasicMatrix
> and a new MatrixStore - and they contain the old/previous MatrixStore.
>
> I'd do it like this:
>
> PrimitiveDenseStore.FACTORY.makeEmpty(rowCount, colCount);
>
> and then loop to set the elements the way you did before.
>
> You do not have to create a BasicMatrix instance, and if you worry
> very much about memory consumption then you probably shouldn't.
>
>
>> The other option appears to be to create a double array and copy it
>> into the matrix with the factory, but this will mean, for a short
>> time, there will be two copies of a potentially large array in memory.
>>
>> Any advice would be most helpful.
>
>
> Currently the only way to create a new BasicMatrix instance without
> using the set-method and without any "unnecessary" copying is to
> create a MatrixStore and then instantiate a BasicMatrix
> implementation by calling a constructor directly. The constructors in
> BigMatrix, PrimitiveMatrix and ComplexMatrix do not copy anything.
>
>
> /Anders
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> ojAlgo-user mailing list
> ojA...@li...
> https://lists.sourceforge.net/lists/listinfo/ojalgo-user
>