pysparse-users

Re: [Pysparse-users] Trouble finding the correct ATLAS LAPACK libraries with pysparse 1.2a1

[Dominique O. <dom...@gm...>]

On Tue, May 11, 2010 at 12:46:43PM -0500, Eddie Schwalbach wrote:
> Hi guys, thanks for the response.  Our machine went down for maintenance
> this morning, as soon as its back up I'll send the requested outputs.
> 
> I haven't tested the ATLAS or LAPACK libraries, but I have been able to
> compile other software against them (specifically Trilinos 9.0.3 including
> the PyTrilinos module).  Not sure if it matters, but there are other
> versions of ATLAS (3.9.16 and 3.9.24) at /software/ATLAS/#.#.##  where #'s
> are the version numbers.  These newer versions don't have full LAPACK
> support or shared versions of the libraries.
> 
> To answer your question Daniel, there was no previous install of PySparse
> for the Python in my home directory, but I believe on the first attempt at
> installing I used the default setup.py, perhaps this left something around
> that's interfering with more recent attempts to install?  I did run
> 
> python setup.py clean
> 
> before modifying setup.py and rerunning the build/install steps.  Is there a
> better way of removing/cleaning up before I try installing again?

It is much safer to 'rm -rf build'. I find the Distutils aren't as
reliable as Make.
 
> Thanks again, and more info to follow when I can get access to the machine.
> 
> Eddie
> 
> 
> On Tue, May 11, 2010 at 9:01 AM, Daniel Wheeler
> <dan...@gm...>wrote:
> Was there a previous pysparse installation?
> 
> 
> On Tue, May 11, 2010 at 2:01 AM, Dominique Orban
> <dom...@gm...>wrote:
> 
> > On Mon, May 10, 2010 at 5:59 PM, Eddie Schwalbach
> > <e-s...@no...> wrote:
> > > I'm trying to install pysparse on a Linux cluster with ATLAS 3.8.3 (which
> > > was built with full LAPACK support).  ATLAS is located at
> > > /software/ATLAS/3.8.3
> > > and I'm using Python2.6.4 which is located in the nonstandard location
> > > /home/ejs698/software (which is at the beginning of my PATH and
> > PYTHONPATH).
> > >
> > > I've modified the setup.py file to include the appropriate (I think)
> > library
> > > directories:
> > >
> > > library_dirs_list=
> > >
> > ['/software/ATLAS/3.8.3/lib/','/home/ejs698/software/lib/','/usr/bin/lib/']
> > > libraries_list = ['f77blas', 'cblas', 'atlas', 'lapack', 'gfortran']
> > >
> > > I also have the environment variables:
> > > LD_LIBRARY_PATH
> > >
> > /opt/moab/lib:/opt/torque/lib::/software/ATLAS/3.8.3/lib:/home/ejs698/software/lib
> > > ATLAS  /software/ATLAS/3.8.3  (this was necessary for getting numpy built
> > > against the correct ATLAS libraries)
> > >
> > > I don't see errors when I run python setup.py install, or when I try
> > >      from pysparse import *
> > > However, if I attempt to run any of the tests in the Examples directory,
> > I
> > > get:
> > >
> > > [ejs698@quser04 Examples]$ python jdsym_test.py
> > > Traceback (most recent call last):
> > >   File "jdsym_test.py", line 1, in <module>
> > >     from pysparse import spmatrix, jdsym, itsolvers
> > > ImportError: /software/ATLAS/lib/liblapack.so: cannot open shared object
> > > file: No such file or directory
> > >
> > > This seems to indicate that pysparse is looking in the wrong place for
> > the
> > > lapack libraries (the correct location is
> > > /software/ATLAS/3.8.3/lib/liblapack.so).  Have I missed some environment
> > > variable or setting in the setup.py to properly specify these library
> > > locations?
> > >
> > > Any suggestions are appreciated,  and thanks in advance,
> >
> > Hello Eddie,
> >
> > There are no environment variables to set in Pysparse, so no, you
> > haven't missed any.
> >
> > Could you please post the complete output of 'python setup.py build'
> > as well as the result of 'ls -l
> > /software/ATLAS/3.8.3/lib/liblapack.so'? I don't doubt your word that
> > the library is there but am just trying to get the obvious things out
> > of the way first. Also I presume you can run the ATLAS and LAPACK
> > tests with your libraries?
> >
> > Thanks,
> >
> > --
> > Dominique
> >

Re: [Pysparse-users] Trouble finding the correct ATLAS LAPACK libraries with pysparse 1.2a1

[Eddie S. <e-s...@no...>]

Hi guys, thanks for the response.  Our machine went down for maintenance
this morning, as soon as its back up I'll send the requested outputs.

I haven't tested the ATLAS or LAPACK libraries, but I have been able to
compile other software against them (specifically Trilinos 9.0.3 including
the PyTrilinos module).  Not sure if it matters, but there are other
versions of ATLAS (3.9.16 and 3.9.24) at /software/ATLAS/#.#.##  where #'s
are the version numbers.  These newer versions don't have full LAPACK
support or shared versions of the libraries.

To answer your question Daniel, there was no previous install of PySparse
for the Python in my home directory, but I believe on the first attempt at
installing I used the default setup.py, perhaps this left something around
that's interfering with more recent attempts to install?  I did run

python setup.py clean

before modifying setup.py and rerunning the build/install steps.  Is there a
better way of removing/cleaning up before I try installing again?

Thanks again, and more info to follow when I can get access to the machine.

Eddie


On Tue, May 11, 2010 at 9:01 AM, Daniel Wheeler
<dan...@gm...>wrote:
Was there a previous pysparse installation?


On Tue, May 11, 2010 at 2:01 AM, Dominique Orban
<dom...@gm...>wrote:

> On Mon, May 10, 2010 at 5:59 PM, Eddie Schwalbach
> <e-s...@no...> wrote:
> > I'm trying to install pysparse on a Linux cluster with ATLAS 3.8.3 (which
> > was built with full LAPACK support).  ATLAS is located at
> > /software/ATLAS/3.8.3
> > and I'm using Python2.6.4 which is located in the nonstandard location
> > /home/ejs698/software (which is at the beginning of my PATH and
> PYTHONPATH).
> >
> > I've modified the setup.py file to include the appropriate (I think)
> library
> > directories:
> >
> > library_dirs_list=
> >
> ['/software/ATLAS/3.8.3/lib/','/home/ejs698/software/lib/','/usr/bin/lib/']
> > libraries_list = ['f77blas', 'cblas', 'atlas', 'lapack', 'gfortran']
> >
> > I also have the environment variables:
> > LD_LIBRARY_PATH
> >
> /opt/moab/lib:/opt/torque/lib::/software/ATLAS/3.8.3/lib:/home/ejs698/software/lib
> > ATLAS  /software/ATLAS/3.8.3  (this was necessary for getting numpy built
> > against the correct ATLAS libraries)
> >
> > I don't see errors when I run python setup.py install, or when I try
> >      from pysparse import *
> > However, if I attempt to run any of the tests in the Examples directory,
> I
> > get:
> >
> > [ejs698@quser04 Examples]$ python jdsym_test.py
> > Traceback (most recent call last):
> >   File "jdsym_test.py", line 1, in <module>
> >     from pysparse import spmatrix, jdsym, itsolvers
> > ImportError: /software/ATLAS/lib/liblapack.so: cannot open shared object
> > file: No such file or directory
> >
> > This seems to indicate that pysparse is looking in the wrong place for
> the
> > lapack libraries (the correct location is
> > /software/ATLAS/3.8.3/lib/liblapack.so).  Have I missed some environment
> > variable or setting in the setup.py to properly specify these library
> > locations?
> >
> > Any suggestions are appreciated,  and thanks in advance,
>
> Hello Eddie,
>
> There are no environment variables to set in Pysparse, so no, you
> haven't missed any.
>
> Could you please post the complete output of 'python setup.py build'
> as well as the result of 'ls -l
> /software/ATLAS/3.8.3/lib/liblapack.so'? I don't doubt your word that
> the library is there but am just trying to get the obvious things out
> of the way first. Also I presume you can run the ATLAS and LAPACK
> tests with your libraries?
>
> Thanks,
>
> --
> Dominique
>

Re: [Pysparse-users] Trouble finding the correct ATLAS LAPACK libraries with pysparse 1.2a1

[Daniel W. <dan...@gm...>]

Was there a previous pysparse installation?

On Mon, May 10, 2010 at 12:59 PM, Eddie Schwalbach
<e-s...@no...> wrote:
> I'm trying to install pysparse on a Linux cluster with ATLAS 3.8.3 (which
> was built with full LAPACK support).  ATLAS is located at
> /software/ATLAS/3.8.3
> and I'm using Python2.6.4 which is located in the nonstandard location
> /home/ejs698/software (which is at the beginning of my PATH and PYTHONPATH).
>
> I've modified the setup.py file to include the appropriate (I think) library
> directories:
>
> library_dirs_list=
> ['/software/ATLAS/3.8.3/lib/','/home/ejs698/software/lib/','/usr/bin/lib/']
> libraries_list = ['f77blas', 'cblas', 'atlas', 'lapack', 'gfortran']
>
> I also have the environment variables:
> LD_LIBRARY_PATH
> /opt/moab/lib:/opt/torque/lib::/software/ATLAS/3.8.3/lib:/home/ejs698/software/lib
> ATLAS  /software/ATLAS/3.8.3  (this was necessary for getting numpy built
> against the correct ATLAS libraries)
>
> I don't see errors when I run python setup.py install, or when I try
>      from pysparse import *
> However, if I attempt to run any of the tests in the Examples directory, I
> get:
>
> [ejs698@quser04 Examples]$ python jdsym_test.py
> Traceback (most recent call last):
>   File "jdsym_test.py", line 1, in <module>
>     from pysparse import spmatrix, jdsym, itsolvers
> ImportError: /software/ATLAS/lib/liblapack.so: cannot open shared object
> file: No such file or directory
>
> This seems to indicate that pysparse is looking in the wrong place for the
> lapack libraries (the correct location is
> /software/ATLAS/3.8.3/lib/liblapack.so).  Have I missed some environment
> variable or setting in the setup.py to properly specify these library
> locations?
>
> Any suggestions are appreciated,  and thanks in advance,
>
> Eddie Schwalbach
>
> ------------------------------------------------------------------------------
>
>
> _______________________________________________
> Pysparse-users mailing list
> Pys...@li...
> https://lists.sourceforge.net/lists/listinfo/pysparse-users
>
>



-- 
Daniel Wheeler

Re: [Pysparse-users] Trouble finding the correct ATLAS LAPACK libraries with pysparse 1.2a1

[Dominique O. <dom...@gm...>]

On Mon, May 10, 2010 at 5:59 PM, Eddie Schwalbach
<e-s...@no...> wrote:
> I'm trying to install pysparse on a Linux cluster with ATLAS 3.8.3 (which
> was built with full LAPACK support).  ATLAS is located at
> /software/ATLAS/3.8.3
> and I'm using Python2.6.4 which is located in the nonstandard location
> /home/ejs698/software (which is at the beginning of my PATH and PYTHONPATH).
>
> I've modified the setup.py file to include the appropriate (I think) library
> directories:
>
> library_dirs_list=
> ['/software/ATLAS/3.8.3/lib/','/home/ejs698/software/lib/','/usr/bin/lib/']
> libraries_list = ['f77blas', 'cblas', 'atlas', 'lapack', 'gfortran']
>
> I also have the environment variables:
> LD_LIBRARY_PATH
> /opt/moab/lib:/opt/torque/lib::/software/ATLAS/3.8.3/lib:/home/ejs698/software/lib
> ATLAS  /software/ATLAS/3.8.3  (this was necessary for getting numpy built
> against the correct ATLAS libraries)
>
> I don't see errors when I run python setup.py install, or when I try
>      from pysparse import *
> However, if I attempt to run any of the tests in the Examples directory, I
> get:
>
> [ejs698@quser04 Examples]$ python jdsym_test.py
> Traceback (most recent call last):
>   File "jdsym_test.py", line 1, in <module>
>     from pysparse import spmatrix, jdsym, itsolvers
> ImportError: /software/ATLAS/lib/liblapack.so: cannot open shared object
> file: No such file or directory
>
> This seems to indicate that pysparse is looking in the wrong place for the
> lapack libraries (the correct location is
> /software/ATLAS/3.8.3/lib/liblapack.so).  Have I missed some environment
> variable or setting in the setup.py to properly specify these library
> locations?
>
> Any suggestions are appreciated,  and thanks in advance,

Hello Eddie,

There are no environment variables to set in Pysparse, so no, you
haven't missed any.

Could you please post the complete output of 'python setup.py build'
as well as the result of 'ls -l
/software/ATLAS/3.8.3/lib/liblapack.so'? I don't doubt your word that
the library is there but am just trying to get the obvious things out
of the way first. Also I presume you can run the ATLAS and LAPACK
tests with your libraries?

Thanks,

-- 
Dominique

[Pysparse-users] Trouble finding the correct ATLAS LAPACK libraries with pysparse 1.2a1

[Eddie S. <e-s...@no...>]

I'm trying to install pysparse on a Linux cluster with ATLAS 3.8.3 (which
was built with full LAPACK support).  ATLAS is located at
/software/ATLAS/3.8.3
and I'm using Python2.6.4 which is located in the nonstandard location
/home/ejs698/software (which is at the beginning of my PATH and PYTHONPATH).

I've modified the setup.py file to include the appropriate (I think) library
directories:

library_dirs_list=
['/software/ATLAS/3.8.3/lib/','/home/ejs698/software/lib/','/usr/bin/lib/']
libraries_list = ['f77blas', 'cblas', 'atlas', 'lapack', 'gfortran']

I also have the environment variables:
LD_LIBRARY_PATH
/opt/moab/lib:/opt/torque/lib::/software/ATLAS/3.8.3/lib:/home/ejs698/software/lib
ATLAS  /software/ATLAS/3.8.3  (this was necessary for getting numpy built
against the correct ATLAS libraries)

I don't see errors when I run python setup.py install, or when I try
     from pysparse import *
However, if I attempt to run any of the tests in the Examples directory, I
get:

[ejs698@quser04 Examples]$ python jdsym_test.py
Traceback (most recent call last):
  File "jdsym_test.py", line 1, in <module>
    from pysparse import spmatrix, jdsym, itsolvers
ImportError: /software/ATLAS/lib/liblapack.so: cannot open shared object
file: No such file or directory

This seems to indicate that pysparse is looking in the wrong place for the
lapack libraries (the correct location is
/software/ATLAS/3.8.3/lib/liblapack.so).  Have I missed some environment
variable or setting in the setup.py to properly specify these library
locations?

Any suggestions are appreciated,  and thanks in advance,

Eddie Schwalbach

Re: [Pysparse-users] DeprecationWarning

[Dominique O. <dom...@gm...>]

On Fri, May 7, 2010 at 12:03 PM,  <kei...@bt...> wrote:
> I've got the latest pysparse and numpy, but I get this:
>
> test.py:102: DeprecationWarning: PyArray_FromDimsAndDataAndDescr: use PyArray_NewFromDescr.
>  info,iter,relres=itsolvers.cgs(A,b,z1,1e-9,10000,K)
>
> How can I get rid of this?  Does it matter?

Hi Keith,

This warning is harmless and can be safely ignored. Did you get the
source from svn?

-- 
Dominique

[Pysparse-users] DeprecationWarning

[<kei...@bt...>]

I've got the latest pysparse and numpy, but I get this:

test.py:102: DeprecationWarning: PyArray_FromDimsAndDataAndDescr: use PyArray_NewFromDescr.
  info,iter,relres=itsolvers.cgs(A,b,z1,1e-9,10000,K)

How can I get rid of this?  Does it matter?

Keith

Re: [Pysparse-users] ll_mat.__getitem__ still leaks lots of memory

[Dominique O. <dom...@gm...>]

Thanks Rob! Good catch. Patch applied and source committed.

Dominique


On Tue, Apr 27, 2010 at 9:09 PM, Rob Speer <rs...@mi...> wrote:
> I found the memory leak and fixed it. The index lists, irow and jcol,
> were never being freed.
>
> I've attached the patch, and I've also updated my version on PyPI
> (csc-pysparse 1.1.1.3).
>
> -- Rob
>
> On Mon, Apr 26, 2010 at 3:09 PM, Rob Speer <rs...@mi...> wrote:
>> I saw that Łukasz Pankowski submitted a fix for a memory leak in
>> ll_mat. I updated my copy of pysparse (from the new-setup-py branch),
>> but I'm still getting a memory leak.
>>
>> Here's an easy way to reproduce it:
>>
>> from pysparse.sparse.spmatrix import ll_mat
>> mat = ll_mat(1000,1000)
>> mat[0,0] = 1
>> for i in xrange(100000): m = mat[0,:]
>> for i in xrange(100000): m = mat[0,:]
>> ...
>>
>> If I repeat that last line indefinitely, I lose about 400 MB of memory
>> each time.
>>
>> -- Rob
>>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Pysparse-users mailing list
> Pys...@li...
> https://lists.sourceforge.net/lists/listinfo/pysparse-users
>
>



-- 
Dominique

Re: [Pysparse-users] ll_mat.__getitem__ still leaks lots of memory

[Rob S. <rs...@MI...>]

I found the memory leak and fixed it. The index lists, irow and jcol,
were never being freed.

I've attached the patch, and I've also updated my version on PyPI
(csc-pysparse 1.1.1.3).

-- Rob

On Mon, Apr 26, 2010 at 3:09 PM, Rob Speer <rs...@mi...> wrote:
> I saw that Łukasz Pankowski submitted a fix for a memory leak in
> ll_mat. I updated my copy of pysparse (from the new-setup-py branch),
> but I'm still getting a memory leak.
>
> Here's an easy way to reproduce it:
>
> from pysparse.sparse.spmatrix import ll_mat
> mat = ll_mat(1000,1000)
> mat[0,0] = 1
> for i in xrange(100000): m = mat[0,:]
> for i in xrange(100000): m = mat[0,:]
> ...
>
> If I repeat that last line indefinitely, I lose about 400 MB of memory
> each time.
>
> -- Rob
>

[Pysparse-users] ll_mat.__getitem__ still leaks lots of memory

[Rob S. <rs...@MI...>]

I saw that Łukasz Pankowski submitted a fix for a memory leak in
ll_mat. I updated my copy of pysparse (from the new-setup-py branch),
but I'm still getting a memory leak.

Here's an easy way to reproduce it:

from pysparse.sparse.spmatrix import ll_mat
mat = ll_mat(1000,1000)
mat[0,0] = 1
for i in xrange(100000): m = mat[0,:]
for i in xrange(100000): m = mat[0,:]
...

If I repeat that last line indefinitely, I lose about 400 MB of memory
each time.

-- Rob

[Pysparse-users] A downstream package: csc-pysparse 1.1.1.1

[Rob S. <rs...@MI...>]

I've got a package of my own, called divisi2, that depends on
pysparse. The ability to install that package depends on the ability
to install pysparse without user intervention.

So, I've made a downstream package of PySparse called "csc-pysparse".
It's in a git repository (http://github.com/rspeer/csc-pysparse/) that
pulls in updates from the real PySparse's new-setup-py branch, and
it's released on PyPI as csc-pysparse version 1.1.1.1.

My downstream changes are:
* changing setup.py to reflect the fact that it's a different package, of course
* adding a MANIFEST.in, so that one can produce source packages of
csc-pysparse without using distutils's Subversion hack
* my small patch that makes sure SpMatrix_API is defined in exactly
one .c file, making it compile on Apple's gcc

As a result, you can now "pip install csc-pysparse".

I intend to keep this package up to date with future changes in
pysparse. I hope it's useful to other pysparse users as well.

-- Rob

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

> You can set compiler options by setting the CFLAGS variable, e.g. :
>
> % CFLAGS='-O0' python setup.py build
>
> See http://docs.python.org/install/#building-extensions-tips-and-tricks
>
> I'm using the gcc from http://hpc.sf.net which seems to be setting
> those options automatically.

Well, I couldn't fix it with the compiler options. I don't know why
hpc's and Apple's compilers are so different here, but they apparently
are. But I could fix it by googling for recommendations on how to
avoid duplicate symbol errors, which basically involves defining the
thing shared between a bunch of modules (in this case,
"SPMATRIX_UNIQUE_SYMBOL") as extern in the .h file, and defining it
for real in one of the .c files.

I might have broken something else by doing this, but it now installs
out of the box and passes the tests I need it for.

I'm attaching this as a very short patch file.

-- Rob

Re: [Pysparse-users] pysparse and multiprocessing

[Dominique O. <dom...@gm...>]

On Thu, Apr 8, 2010 at 3:15 PM, Eduardo Lenz Cardoso
<le...@jo...> wrote:
> Hi,
>
> I am trying to use the pool method of
> multiprocessing. Each process has
> as input a number and a class. This class has
> a sparse matrix (pysparse ll_mat) inside it.
>
> The problem is that the class must be pickled
> and depickled when moving around processes and
> I am having the following error:
>
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
>  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
>    self.run()
>  File "/usr/lib/python2.6/threading.py", line 484, in run
>    self.__target(*self.__args, **self.__kwargs)
>  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
>    put(task)
> UnpickleableError: Cannot pickle <type 'll_mat'> objects
>
> I am searching all around and could not find any tip to
> solve this problem. Any clue ?

Hi Eduardo,

Unfortunately, Python objects implemented in C can't be pickled or
marshalled. A PysparseMatrix class, being a pure Python class, could
be pickled if you removed its 'matrix' member, but that would render
it useless. So: can't pickle in PySparse...

As an alternative I can suggest writing the matrix to file in mtx
format, and then reading it in again. Hopefully, that can help you. If
A is an ll_mat object, use A.export_mtx().

Good luck,

-- 
Dominique

[Pysparse-users] pysparse and multiprocessing

[Eduardo L. C. <le...@jo...>]

Hi,

I am trying to use the pool method of
multiprocessing. Each process has
as input a number and a class. This class has
a sparse matrix (pysparse ll_mat) inside it.

The problem is that the class must be pickled
and depickled when moving around processes and
I am having the following error:


Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
    put(task)
UnpickleableError: Cannot pickle <type 'll_mat'> objects

I am searching all around and could not find any tip to
solve this problem. Any clue ?

Thanks,
Eduardo.




-- 
Esta mensagem foi verificada pelo sistema de antivírus e
 acredita-se estar livre de perigo.

Re: [Pysparse-users] Pysparse installation problems

[Dominique O. <dom...@gm...>]

On Thu, Apr 8, 2010 at 9:50 AM, Rob Speer <rs...@mi...> wrote:
> On Wed, Apr 7, 2010 at 3:00 AM, Dominique Orban
> <dom...@gm...> wrote:
>> On Tue, Apr 6, 2010 at 12:23 AM, Rob Speer <rs...@mi...> wrote:
>>> It seems that it has something to do with the symbol hackery that
>>> defines SPMATRIX_UNIQUE_SYMBOL in a whole bunch of different places.
>>> Apparently numpy.distutils isn't handling this the way you expect.
>>>
>>> If I define NO_IMPORT_SPMATRIX, it compiles.
>>
>> This is odd. It compiles without problems for me on Mac Intel but
>> again I notice that my gcc flags are quite different from yours:
>>
>> C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
>> -Wstrict-prototypes
>>
>> Are you using the default /usr/bin/gcc?
>
> The one that comes from Xcode, yes.
>
> The -fno-strict-aliasing option sounds relevant. Where are these
> options set, anyway?

You can set compiler options by setting the CFLAGS variable, e.g. :

% CFLAGS='-O0' python setup.py build

See http://docs.python.org/install/#building-extensions-tips-and-tricks

I'm using the gcc from http://hpc.sf.net which seems to be setting
those options automatically.

-- 
Dominique

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

On Wed, Apr 7, 2010 at 3:00 AM, Dominique Orban
<dom...@gm...> wrote:
> On Tue, Apr 6, 2010 at 12:23 AM, Rob Speer <rs...@mi...> wrote:
>> It seems that it has something to do with the symbol hackery that
>> defines SPMATRIX_UNIQUE_SYMBOL in a whole bunch of different places.
>> Apparently numpy.distutils isn't handling this the way you expect.
>>
>> If I define NO_IMPORT_SPMATRIX, it compiles.
>
> This is odd. It compiles without problems for me on Mac Intel but
> again I notice that my gcc flags are quite different from yours:
>
> C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
> -Wstrict-prototypes
>
> Are you using the default /usr/bin/gcc?

The one that comes from Xcode, yes.

The -fno-strict-aliasing option sounds relevant. Where are these
options set, anyway?
-- Rob

Re: [Pysparse-users] Pysparse installation problems

[Dominique O. <dom...@gm...>]

On Tue, Apr 6, 2010 at 12:23 AM, Rob Speer <rs...@mi...> wrote:
> It seems that it has something to do with the symbol hackery that
> defines SPMATRIX_UNIQUE_SYMBOL in a whole bunch of different places.
> Apparently numpy.distutils isn't handling this the way you expect.
>
> If I define NO_IMPORT_SPMATRIX, it compiles.

This is odd. It compiles without problems for me on Mac Intel but
again I notice that my gcc flags are quite different from yours:

C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes

Are you using the default /usr/bin/gcc?

-- 
Dominique

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

It seems that it has something to do with the symbol hackery that
defines SPMATRIX_UNIQUE_SYMBOL in a whole bunch of different places.
Apparently numpy.distutils isn't handling this the way you expect.

If I define NO_IMPORT_SPMATRIX, it compiles.

-- Rob


On Mon, Apr 5, 2010 at 7:07 PM, Rob Speer <rs...@mi...> wrote:
> Do you know why itsolvers won't compile in new-setup-py? Here's the
> output again:
>
>
> building 'pysparse.itsolvers.itsolvers' extension
>
> compiling C sources
>
> C compiler: gcc -arch ppc -arch i386 -isysroot
> /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common
> -dynamic -DNDEBUG -g -O3
>
>
>
> creating build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers
>
> creating build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src
>
> compile options: '-DNO_ATLAS_INFO=3 -Ipysparse/itsolvers/src
> -I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include
> -I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include
> -Iinclude -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
> -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
> -c'
>
> extra options: '-faltivec -I/System/Library/Frameworks/vecLib.framework/Headers'
>
> gcc: pysparse/itsolvers/src/qmrs.c
>
> gcc: pysparse/itsolvers/src/bicgstab.c
>
> gcc: pysparse/itsolvers/src/gmres.c
>
> gcc: pysparse/itsolvers/src/pcg.c
>
> gcc: pysparse/itsolvers/src/cgs.c
>
> gcc: pysparse/itsolvers/src/minres.c
>
> gcc: pysparse/itsolvers/src/itsolversmodule.c
>
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g
> -bundle -undefined dynamic_lookup
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/cgs.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/gmres.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/minres.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/pcg.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/qmrs.o -o
> pysparse/itsolvers/itsolvers.so -Wl,-framework -Wl,Accelerate
>
> ld: duplicate symbol _itsolvers_spmatrix in
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.old
> :and  duplicatebuild /symboltemp.macosx -_itsolvers_spmatrix10.3-
> fatin- build2.6//temp.macosxpysparse-/10.3itsolvers-/fatsrc-2.6//itsolversmodule.o
> pysparse/foritsolvers architecture /ppc
>
> src/bicgstab.o and
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
> for architecture i386
>
> collect2: ld returned 1 exit status
>
> collect2: ld returned 1 exit status
>
> lipo: can't open input file:
> /var/folders/SZ/SZiYTR1fGoeIwworxM8sgU+++TI/-Tmp-//ccXGMs4h.out (No
> such file or directory)
>
> ld: duplicate symbol _itsolvers_spmatrix in
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.old
> :and  duplicatebuild /symboltemp.macosx -_itsolvers_spmatrix10.3-
> fatin- build2.6//temp.macosxpysparse-/10.3itsolvers-/fatsrc-2.6//itsolversmodule.o
> pysparse/foritsolvers architecture /ppc
>
> src/bicgstab.o and
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
> for architecture i386
>
> collect2: ld returned 1 exit status
>
> collect2: ld returned 1 exit status
>
> lipo: can't open input file:
> /var/folders/SZ/SZiYTR1fGoeIwworxM8sgU+++TI/-Tmp-//ccXGMs4h.out (No
> such file or directory)
>
> error: Command "gcc -arch ppc -arch i386 -isysroot
> /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/cgs.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/gmres.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/minres.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/pcg.o
> build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/qmrs.o -o
> pysparse/itsolvers/itsolvers.so -Wl,-framework -Wl,Accelerate" failed
> with exit status 1
>
> -- Rob
>

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

Do you know why itsolvers won't compile in new-setup-py? Here's the
output again:


building 'pysparse.itsolvers.itsolvers' extension

compiling C sources

C compiler: gcc -arch ppc -arch i386 -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common
-dynamic -DNDEBUG -g -O3



creating build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers

creating build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src

compile options: '-DNO_ATLAS_INFO=3 -Ipysparse/itsolvers/src
-I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include
-Iinclude -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-c'

extra options: '-faltivec -I/System/Library/Frameworks/vecLib.framework/Headers'

gcc: pysparse/itsolvers/src/qmrs.c

gcc: pysparse/itsolvers/src/bicgstab.c

gcc: pysparse/itsolvers/src/gmres.c

gcc: pysparse/itsolvers/src/pcg.c

gcc: pysparse/itsolvers/src/cgs.c

gcc: pysparse/itsolvers/src/minres.c

gcc: pysparse/itsolvers/src/itsolversmodule.c

gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g
-bundle -undefined dynamic_lookup
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/cgs.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/gmres.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/minres.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/pcg.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/qmrs.o -o
pysparse/itsolvers/itsolvers.so -Wl,-framework -Wl,Accelerate

ld: duplicate symbol _itsolvers_spmatrix in
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.old
:and  duplicatebuild /symboltemp.macosx -_itsolvers_spmatrix10.3-
fatin- build2.6//temp.macosxpysparse-/10.3itsolvers-/fatsrc-2.6//itsolversmodule.o
pysparse/foritsolvers architecture /ppc

src/bicgstab.o and
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
for architecture i386

collect2: ld returned 1 exit status

collect2: ld returned 1 exit status

lipo: can't open input file:
/var/folders/SZ/SZiYTR1fGoeIwworxM8sgU+++TI/-Tmp-//ccXGMs4h.out (No
such file or directory)

ld: duplicate symbol _itsolvers_spmatrix in
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.old
:and  duplicatebuild /symboltemp.macosx -_itsolvers_spmatrix10.3-
fatin- build2.6//temp.macosxpysparse-/10.3itsolvers-/fatsrc-2.6//itsolversmodule.o
pysparse/foritsolvers architecture /ppc

src/bicgstab.o and
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
for architecture i386

collect2: ld returned 1 exit status

collect2: ld returned 1 exit status

lipo: can't open input file:
/var/folders/SZ/SZiYTR1fGoeIwworxM8sgU+++TI/-Tmp-//ccXGMs4h.out (No
such file or directory)

error: Command "gcc -arch ppc -arch i386 -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/itsolversmodule.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/bicgstab.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/cgs.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/gmres.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/minres.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/pcg.o
build/temp.macosx-10.3-fat-2.6/pysparse/itsolvers/src/qmrs.o -o
pysparse/itsolvers/itsolvers.so -Wl,-framework -Wl,Accelerate" failed
with exit status 1

-- Rob

Re: [Pysparse-users] Pysparse installation problems

[Dominique O. <dom...@gm...>]

On Fri, Mar 26, 2010 at 9:50 PM, Rob Speer <rs...@mi...> wrote:
> On Fri, Mar 26, 2010 at 4:01 AM, Dominique Orban
> <dom...@gm...> wrote:
>> I did try to wrap 'import setuptools' in a try/except but that had
>> consequences when I did simple 'python setup.py install'. It created
>> new directories and eggs. I'd rather leave it up to the user unless
>> there is a strong consensus otherwise.
>
> The egg-info directory does have a purpose: it allows the package to
> be upgraded or uninstalled without manually hacking the site-packages
> directory. What harm does it do?

Rob,

It's probably just me. I added the try/except in svn. Enjoy!

Dominique




-- 
Dominique

Re: [Pysparse-users] Computing largest eigenvalues?

[Roman G. <rom...@gm...>]

Hi Rob

I have made good experiences with ARPACK in the past.

There seems to be a Python wrapper for it as well (though I have never used it).

Regards,
Roman

On Thu, Mar 25, 2010 at 6:21 PM, Rob Speer <rs...@mi...> wrote:
> Got any pointers? Up until now, I've been working with an old, clunky
> C library called SVDLIBC that implements Lanczos. Pysparse's jdsym is
> the first thing I've seen that can find eigenvectors and presents an
> interface that can actually work with Python objects.
>
> There's the stuff in scipy.sparse, of course, but that's been stalled
> in the development for years now, and it doesn't often compile from
> SVN.
>
> -- Rob
>
> On Thu, Mar 25, 2010 at 3:17 AM, Roman Geus <rom...@gm...> wrote:
>> Hi Rob
>>
>> If you set tau to some very large number, but still far away from the
>> actual largest eigenvalue, you might experience very slow convergence.
>> There are certainly better and simpler algorithms than JDSYM for
>> computing a few of the largest eigenvalues.
>>
>> -- Roman
>>
>> On Wed, Mar 24, 2010 at 8:05 PM, Rob Speer <rs...@mi...> wrote:
>>> It looks like it should be possible to compute the truncated spectral
>>> decomposition of a sparse, symmetric matrix using pysparse.jdsym. This
>>> is the key step in computing a truncated SVD, which is the next thing
>>> to do, and it would be great to be able to do it entirely within
>>> Pysparse.
>>>
>>> There's just one thing I'm unsure about: how do I ask for the
>>> *largest* eigenvalues? jdsym is set up to return eigenvalues around
>>> some value tau, defaulting to 0, so it seems this is set up for
>>> finding the smallest eigenvalues. Do I just set tau to some very large
>>> number, or would that cause numerical stability issues? Is this the
>>> wrong problem for jdsym to solve?
>>>
>>> -- Rob
>>>
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Pysparse-users mailing list
>>> Pys...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pysparse-users
>>>
>>
>

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

On Fri, Mar 26, 2010 at 4:01 AM, Dominique Orban
<dom...@gm...> wrote:
> I did try to wrap 'import setuptools' in a try/except but that had
> consequences when I did simple 'python setup.py install'. It created
> new directories and eggs. I'd rather leave it up to the user unless
> there is a strong consensus otherwise.

The egg-info directory does have a purpose: it allows the package to
be upgraded or uninstalled without manually hacking the site-packages
directory. What harm does it do?
-- Rob

Re: [Pysparse-users] Pysparse installation problems

[Dominique O. <dom...@gm...>]

On Fri, Mar 26, 2010 at 4:32 PM, Rob Speer <rs...@mi...> wrote:
>> The matvec behavior is the intended behavior. The whole idea of the
>> PysparseMatrix class is that it is "easy" and "natural" to use.
>> Amongst other things, that means you can say y=A*x instead of
>> A.matvec(x,y). The savvy user will notice (as you did) that ll_mat
>> objects are more efficient and will use those. The typical user will
>> use the PysparseMatrix class.
>
> That creates a rather broken API. Having an inplace version of matvec
> is fine, but please at least call it "matvec_inplace" or something, in
> both ll_mat and jdsym, to make it clear that it's not supposed to be
> matvec. The current situation is very confusing.

I understand your confusion now. You're not supposed to use the matvec
method of a PysparseMatrix object. You're just supposed to say "*", as
in y=A*x. One of us did add a matvec method in there just because that
is what Scipy solvers expect.

I should really let the others chime in about renaming ll_mat.matvec.
That would surely break some existing code. Perhaps the surest thing
is to subclass PysparseMatrix and add the matvec method there, just
for Scipy solvers.

Thanks for the raising this point.

Dominique




-- 
Dominique

Re: [Pysparse-users] Pysparse installation problems

[Rob S. <rs...@MI...>]

> The matvec behavior is the intended behavior. The whole idea of the
> PysparseMatrix class is that it is "easy" and "natural" to use.
> Amongst other things, that means you can say y=A*x instead of
> A.matvec(x,y). The savvy user will notice (as you did) that ll_mat
> objects are more efficient and will use those. The typical user will
> use the PysparseMatrix class.

That creates a rather broken API. Having an inplace version of matvec
is fine, but please at least call it "matvec_inplace" or something, in
both ll_mat and jdsym, to make it clear that it's not supposed to be
matvec. The current situation is very confusing.

-- Rob

Re: [Pysparse-users] Pysparse installation problems

[Dominique O. <dom...@gm...>]

On Thu, Mar 25, 2010 at 4:11 PM, Rob Speer <rs...@mi...> wrote:
> Thanks for figuring out the include thing!
>
> You should put setuptools in a try/except block. Remember, I want
> users to automatically pull in pysparse as a dependency, so they
> wouldn't have an option to stop the process between downloading and
> installing pysparse to edit a file. And when pip installs something
> from SVN instead of from PyPI, it insists on 'setup.py develop'.
>
> It's fortunate that pip can force setuptools to be imported, but we
> shouldn't rely on that behavior.
>
> Unrelatedly, I've just noticed a bug in the PysparseMatrix class. It
> has a .matvec method that doesn't meet NumPy's requirements (because
> it returns its result, instead of writing it into another array).
> Currently I work around it by using .matrix.matvec, which does the
> right thing, but it would be nice if this worked better. Would it make
> sense for me to have SVN access so I can fix minor things like this?

I did try to wrap 'import setuptools' in a try/except but that had
consequences when I did simple 'python setup.py install'. It created
new directories and eggs. I'd rather leave it up to the user unless
there is a strong consensus otherwise.

The matvec behavior is the intended behavior. The whole idea of the
PysparseMatrix class is that it is "easy" and "natural" to use.
Amongst other things, that means you can say y=A*x instead of
A.matvec(x,y). The savvy user will notice (as you did) that ll_mat
objects are more efficient and will use those. The typical user will
use the PysparseMatrix class.

-- 
Dominique