Re: [ojAlgo-user] Does the 16GB array size limitation affect you?

[Anders P. <an...@op...>]

Some time ago I had support for "huge arrays" implemented and done in ojAlgo, but decided NOT to use/include it - just didn't like some of the necessary code changes.

Generally ojAlgo supports (2^31 - 1) rows, columns, matrices, cubes.... (standard Java limitation). For some specific use cases the practical limit is data structures with (2^31 - 1) elements regardless of how many dimensions you use to access that structure.

As far as I can see now it will stay this way.

/Anders


On 25 apr 2012, at 13:01, Anders Peterson <an...@op...> wrote:

> Currently ojAlgo has an array size limit of 2^31 - 1 which is the limit java has for arrays and collections. Even when you work with 2-dimensional arrays/matrixes or any-dimensional arrays ojAlgo has a limit of 2^31 - 1 elements, instead of having that many rows, columns, planes, cubes...
> 
> Compared to other libraries ojAlgo performs best when the arrays/matrices are a bit larger. I always felt that because of that it's a bit ironic that ojAlgo has a size limitation that some other libraries don't have.
> 
> An array of double with length 2^31 - 1 uses 16GB memory. For a 2-dimensional square matrix the size limit is 46340x46340. Personally I've never been close to this limit with real world ojAlgo usage.
> 
> Currently I'm thinking about adding support for huge "arrays" in ojAlgo, but I hesitate, and it would only be supported with Array1D, Array2D and ArrayAnyD - not with the various MatrixStore implementations. That would be too much of a change.
> 
> The implementation would of course require that some sort of array abstraction is introduced, and that will decrease performance (only when you actually use those huge arrays).
> 
> Anyone have an opinion? If I add this; will you, or anyone you know, be able to do things you cannot do now?
> 
> /Anders