Re: [ojAlgo-user] Interface analogous to Numpy?

[Stefan R. <Ste...@gm...>]

Hello together,
for simplicity, I will just comment in my opinion about your statements, regardless to the author...

> I thought learning about what would be possible was "the project". That
> SVN module is there if can be useful to you.

Agree ;-)


> Can't answer any of your questions regarding the importance of fully
> supporting various MumPy-capabilities. Why not just try to port some small
> application to Java/Jython?

Agree again. I think, Applications, which suffer from those java restrictions are so special-purpouse, that they should not be done via java anyway (and in my opinion also not in python) or should be outsourced into specialized native code.


> Don't know about such a benchmark. For simple operations like scaling, or
> adding to, all the aliments of an array I would expect little or no
> difference between Java and C/Fortran. For more complex operations like matrix
> decompositions I'd guess that C/Fortran is several times faster. Then again
> there can be be quite large differences between C/Fortran implementations...

I once thought about building kind of jumpy on top of JMT http://code.google.com/p/matrix-toolkits-java/ because it can use native lapack and blas bakcends for those "complex" operations. Problem with it is, that it does not support this any-dim-array-stuff as far as I know.

Maybe it would be a nice plan for future ojalgo (absolutely apart of this jumpy/numpy/jython/python thing) to support netlib-java (http://code.google.com/p/netlib-java/) or even JMT as a backend. So it could benefit from any native, high-efficiency blas or lapack implementation.


> > - unsigned long ist not supported. BigInteger as a replacement would be
> much
> >  too slow because of the many boxed heap object allocations (mentioned,
> >  for example, in Bloch, Effective Java, Item 49: Prefer primitive types
> to
> >  boxed primitives). How often are unsigned longs used in Numpy?

I think, this is a general jython-issue anyway, which we can't aim to cure here. I could imagine, that pure integer processing is a rather seldom used feature anyway.
If we could set up a jumpy capable of efficient floating point processing and a consistent and compatible but slow (and slow only when using longs) integer processing with relatively small effort, I would be completely satisfied for now.
Regarding to a project like jumpy, there is a very significant distance between "works with some restrictions" and "is not available at all".


> > - Java arrays (and, sadly, Collections as well) have a maximum length of
> >  Integer.MAX_VALUE which is specified as 2^31 - 1. Numpy arrays, on the
> other
> >  hand, can be much larger. If we use a 1-dimensional array with strides
> in
> >  Jumpy, this would limit our quadratic matrices to only 46340 rows in
> Java.
> >  But the design of Numpy is seriously tied to this underlying
> 1-dimensional
> >  array implementation, and applications might allocate larger ndarrays
> than we
> >  can provide.

Same as for unsigned long/BigInteger-issue. This could only resolved by my entire idea to adapt original numpy to jython by JNI, making the complete nd-array handling native. But this idea is not ojalgo-related anyway, so this is the wrong place to discuss it.



> > - We cannot support the Array interface (__array_interface__,
> >  __array_struct__) for interfacing C libraries and the ctypes foreign
> >  function interface of Python. How often are they used in Numpy
> applications,
> >  in particular Machine Learning applications?

I think, bringing numpy to jython would already be a huge benefit. Accessing other native libraries can't be the scope of this project in any case.



> > - I'm not sure whether Numpy applications sometimes need to perform
> casts of the
> >  contained data array to a different kind of array without copying, e.g.
> from
> >  int[] to byte[] with 4 times the size or vice versa. This could not be
> >  supported in Java.

It would be easy, to write some native methods for this. JNI allows to access java-arrays from c directly without copying (JNI can give you a memory pointer, which points to the actual array data inside the jvm in many cases - however this functionality is not guaranteed, but promised to work quite often).
I'm not sure, whether it would work, because I didn't try yet. But I think, there is a fair chance, that one could cast that c pointer and construct a new java-array from it via JNI. One would have to compile this for a sufficient set of platforms, of course.

Another possibility might be to use stuff from javax.nio. I'm not familiar with it, but I think, it was designed to do things like that. However, the ojalgo-class providing nd-array features would probably need to be reimplemented to use javax.nio, which would burst the effort needed for jumpy. 


> >> I created a module ExtNumPy in the already existing SVN repository.
> (Actually tried switching to Git but couldn't get it working.) You'll have to
> give me your Google user names if you want commit rights to that
> repository.

I have no google-account yet - will see to get one during the next days.



> >>>> I'm not sure, how jython deals with it. Can it map field-access in
> python-code to getter/setter methods on java-side? If not, numpy-python code
> would run only with an additional preprocessing.
> >>>> If you don't know, I would offer to check this out.
> >>> 
> >>> You can always use properties in Python. It's explained here, for
> example:
> >>>
> http://bytebaker.com/2009/03/31/python-properties-vs-java-access-modifiers/

I agree, it will probably work. Only thing I worried about is that java does not have this feature and whether this will go all through with jython.


> >>> 
> >>>> The rest of your post sounds good to me after a first view, although
> I must admit, that I only used rather basic numpy-stuff yet. For instance I
> very rarely used reshaping and never used striding, so I might lack enough
> experience to see problems in your approach.
> >>> 
> >>> Reshaping and striding are seldom used explicitly by the users, but
> very often
> >>> implicitly.

Never doubted that. Just wanted to say, that I might lack the experience needed to judge your ideas with a sufficient competence.



All in all I think, the project is still worth the effort and we should not worry about the arising restrictions in first place. We should aim to adopt as much numpy-functionality as is possible with a sensible large effort. Everything else could be done in future versions or on feature-request basis once a proof of concept is done.



Cheers

Stefan





> >>> Stefan Richthofer wrote:
> >>>>> There are many fields which are all public. We could replace them by
> set-
> >>>>> and
> >>>>> get methods in the adapter.
> >>>> 
> >>>> I'm not sure, how jython deals with it. Can it map field-access in
> python-code to getter/setter methods on java-side? If not, numpy-python code
> would run only with an additional preprocessing.
> >>>> If you don't know, I would offer to check this out.
> >>>> 
> >>>> The rest of your post sounds good to me after a first view, although
> I must admit, that I only used rather basic numpy-stuff yet. For instance I
> very rarely used reshaping and never used striding, so I might lack enough
> experience to see problems in your approach.
> >>>> 
> >>>> cheers
> >>>> 
> >>>> Stefan
> >>>> -- 
> >>>> Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
> >>>> belohnen Sie mit bis zu 50,- Euro!
> https://freundschaftswerbung.gmx.de
> >>>> 
> >>>>
> ------------------------------------------------------------------------------
> >>>> For Developers, A Lot Can Happen In A Second.
> >>>> Boundary is the first to Know...and Tell You.
> >>>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> >>>> http://p.sf.net/sfu/Boundary-d2dvs2
> >>>> _______________________________________________
> >>>> ojAlgo-user mailing list
> >>>> ojA...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/ojalgo-user
> >>> 
> >>>
> ------------------------------------------------------------------------------
> >>> For Developers, A Lot Can Happen In A Second.
> >>> Boundary is the first to Know...and Tell You.
> >>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> >>> http://p.sf.net/sfu/Boundary-d2dvs2
> >>> _______________________________________________
> >>> ojAlgo-user mailing list
> >>> ojA...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/ojalgo-user
> >>> 
> >>> 
> >> 
> >> 
> >>
> ------------------------------------------------------------------------------
> >> For Developers, A Lot Can Happen In A Second.
> >> Boundary is the first to Know...and Tell You.
> >> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> >> http://p.sf.net/sfu/Boundary-d2dvs2
> >> _______________________________________________
> >> ojAlgo-user mailing list
> >> ojA...@li...
> >> https://lists.sourceforge.net/lists/listinfo/ojalgo-user
> > 
> >
> ------------------------------------------------------------------------------
> > For Developers, A Lot Can Happen In A Second.
> > Boundary is the first to Know...and Tell You.
> > Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> > http://p.sf.net/sfu/Boundary-d2dvs2
> > _______________________________________________
> > ojAlgo-user mailing list
> > ojA...@li...
> > https://lists.sourceforge.net/lists/listinfo/ojalgo-user
> > 
> > 
> 
> 
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> ojAlgo-user mailing list
> ojA...@li...
> https://lists.sourceforge.net/lists/listinfo/ojalgo-user

-- 
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!                                  
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a