Menu
▾
▴
numpy-discussion
|
From: Lisandro D. <da...@gm...> - 2006-10-13 20:09:36
|
This post is surely OT, but I cannot imagine a better place to contact people about this subject. Please, don't blame me. Any people here interested in NumPy/SciPy + MPI? From some time ago, I've been developing mpi4py (first release at SF) and I am really near to release a new version. This package exposes an API almost identical to MPI-2 C++ bindings. Almost all MPI-1 and MPI-2 features (even one-sided communications and parallel I/O) are fully supported for any object exposing single-segment buffer interface, an only some of them for communication of general Python objects (with the help of pickle/marshal). The posibility of constructing any user-defined MPI datatypes, as well as virtual topologies (specially cartesian), can be really nice for anyone interested in parallel multidimensional array procesing. Before the next release, I would like to wait for any comment, You can contact me via private mail to get a tarbal with latest developments, or we can have some discussion here, if many of you consider this a good idea. In the long term, I would like to see mpi4py integrated as a subpackage of SciPy. Regards, --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Brian G. <ell...@gm...> - 2006-10-13 22:49:44
|
Just as a data point. I have used mpi4py before and have built it on many systems ranging from my macbook to NERSC supercomputers. In my opinion it is currently the best python mpi bindings available. Lisandro has done a fantastic job with this. Also Fernando and I have worked hard to make sure that mpi4py works with the new parallel capabilities of IPython. I would love to see mpi4py hosted in a public repository for others to contribute. I think this would really solidify mpi4py as a top notch mpi interface. But, my only concern is that there might be many folks who want to use mpi4py who don't need scipy. I am one of those folks - I don't necessarily need scipy on the NERSC supercomputers, but I do need mpi4py. Because of this, I would probably still recommend keeping mpi4py as a separate project. Is there any chance it could be hosted at mip4py.scipy.org? I strongly encourage others to try it out. Installation is is easy. Brian Granger On 10/13/06, Lisandro Dalcin <da...@gm...> wrote: > This post is surely OT, but I cannot imagine a better place to contact > people about this subject. Please, don't blame me. > > Any people here interested in NumPy/SciPy + MPI? From some time ago, > I've been developing mpi4py (first release at SF) and I am really near > to release a new version. > > This package exposes an API almost identical to MPI-2 C++ bindings. > Almost all MPI-1 and MPI-2 features (even one-sided communications and > parallel I/O) are fully supported for any object exposing > single-segment buffer interface, an only some of them for > communication of general Python objects (with the help of > pickle/marshal). > > The posibility of constructing any user-defined MPI datatypes, as well > as virtual topologies (specially cartesian), can be really nice for > anyone interested in parallel multidimensional array procesing. > > Before the next release, I would like to wait for any comment, You can > contact me via private mail to get a tarbal with latest developments, > or we can have some discussion here, if many of you consider this a > good idea. In the long term, I would like to see mpi4py integrated as > a subpackage of SciPy. > > Regards, > > -- > Lisandro Dalc=EDn > --------------- > Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) > Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) > Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) > PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Scipy-dev mailing list > Sci...@sc... > http://projects.scipy.org/mailman/listinfo/scipy-dev > |
|
From: eric <er...@en...> - 2006-10-13 23:03:01
|
Brian Granger wrote: > Just as a data point. > > I have used mpi4py before and have built it on many systems ranging > from my macbook to NERSC supercomputers. In my opinion it is > currently the best python mpi bindings available. Lisandro has done a > fantastic job with this. Also Fernando and I have worked hard to make > sure that mpi4py works with the new parallel capabilities of IPython. > > I would love to see mpi4py hosted in a public repository for others to > contribute. I think this would really solidify mpi4py as a top notch > mpi interface. But, my only concern is that there might be many folks > who want to use mpi4py who don't need scipy. I am one of those folks > - I don't necessarily need scipy on the NERSC supercomputers, but I do > need mpi4py. Because of this, I would probably still recommend > keeping mpi4py as a separate project. Is there any chance it could be > hosted at mip4py.scipy.org? > =20 Fine from our side... eric > I strongly encourage others to try it out. Installation is is easy. > > Brian Granger > > On 10/13/06, Lisandro Dalcin <da...@gm...> wrote: > =20 >> This post is surely OT, but I cannot imagine a better place to contact >> people about this subject. Please, don't blame me. >> >> Any people here interested in NumPy/SciPy + MPI? From some time ago, >> I've been developing mpi4py (first release at SF) and I am really near >> to release a new version. >> >> This package exposes an API almost identical to MPI-2 C++ bindings. >> Almost all MPI-1 and MPI-2 features (even one-sided communications and >> parallel I/O) are fully supported for any object exposing >> single-segment buffer interface, an only some of them for >> communication of general Python objects (with the help of >> pickle/marshal). >> >> The posibility of constructing any user-defined MPI datatypes, as well >> as virtual topologies (specially cartesian), can be really nice for >> anyone interested in parallel multidimensional array procesing. >> >> Before the next release, I would like to wait for any comment, You can >> contact me via private mail to get a tarbal with latest developments, >> or we can have some discussion here, if many of you consider this a >> good idea. In the long term, I would like to see mpi4py integrated as >> a subpackage of SciPy. >> >> Regards, >> >> -- >> Lisandro Dalc=EDn >> --------------- >> Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIM= EC) >> Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INT= EC) >> Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICE= T) >> PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Scipy-dev mailing list >> Sci...@sc... >> http://projects.scipy.org/mailman/listinfo/scipy-dev >> >> =20 > > -----------------------------------------------------------------------= -- > Using Tomcat but need to do more? Need to support web services, securit= y? > Get stuff done quickly with pre-integrated technology to make your job = easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geron= imo > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat= =3D121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > =20 |
|
From: Lisandro D. <da...@gm...> - 2006-10-15 00:17:09
|
On 10/13/06, eric <er...@en...> wrote: > Brian Granger wrote: > > keeping mpi4py as a separate project. > > Is there any chance it could be > > hosted at mip4py.scipy.org? > > > Fine from our side... > > eric > Can anybody help setting up mip4py.scipy.org? I really do not have experience with SVN. What should I do? --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: <tha...@bi...> - 2006-10-14 08:58:46
|
CgpPbiAxMC8xMy8wNiwgTGlzYW5kcm8gRGFsY2luIDxkYWxjaW5sQGdtYWlsLmNvbT4gd3JvdGU6 Cj4gVGhpcyBwb3N0IGlzIHN1cmVseSBPVCwgYnV0IEkgY2Fubm90IGltYWdpbmUgYSBiZXR0ZXIg cGxhY2UgdG8gY29udGFjdAo+IHBlb3BsZSBhYm91dCB0aGlzIHN1YmplY3QuIFBsZWFzZSwgZG9u J3QgYmxhbWUgbWUuCj4gCj4gQW55IHBlb3BsZSBoZXJlIGludGVyZXN0ZWQgaW4gTnVtUHkvU2Np UHkgKyBNUEk/IAoKSSd2ZSBiZWVuIHdvcmtpbmcgb24gYSBEeW5hbWljIEJheWVzaWFuIE5ldHdv cmsgKERCTikgdG9vbGtpdCBmb3Igc29tZSB0aW1lCihjYWxsZWQgTW9jYXB5LCBmcmVlbHkgYXZh aWxhYmxlIGZyb20gc291cmNlZm9yZ2UgaHR0cHM6Ly9zb3VyY2Vmb3JnZS5uZXQvcHJvamVjdHMv bW9jYXB5LykuIFRoZSB0aGluZyBpcyBhbG1vc3QgZW50aXJlbHkgaW1wbGVtZW50ZWQgaW4gUHl0 aG9uLCBhbmQgY3VycmVudGx5IHVzZXMgTnVtZXJpYyBhbmQgcHlNUEkuIEkgcm91dGluZWx5IHRy YWluIERCTnMgZnJvbSAxMDAuMDAwcyBvZiBvYnNlcnZhdGlvbnMgb24gb3VyIDI0MCBDUFUgY2x1 c3Rlci4gCgpJJ20gaW4gdGhlIHByb2NlcyBvZiBwb3J0aW5nIE1vY2FweSB0byBudW1weS4gSSBh c3N1bWUgcHlNUEkgd2lsbCBhbHNvIHdvcmsgd2l0aCBudW1weSwgYnV0IEkgaGF2ZW4ndCB0cmll ZCBpdCBvdXQgeWV0LiBXb3VsZCBiZSBncmVhdCBpZiBzY2lweSBjYW1lIHdpdGggZGVmYXVsdCBN UEkgc3VwcG9ydCwgZXNwZWNpYWxseSBzaW5jZSBweU1QSSBkb2VzIG5vdCBzZWVtIHRvIGJlIGFj dGl2ZWx5IGRldmVsb3BlZCBhbnltb3JlLgoKQ2hlZXJzLAoKLVRob21hcwoKLS0tLQpUaG9tYXMg SGFtZWxyeWNrLCBQb3N0LWRvY3RvcmFsIHJlc2VhcmNoZXIKQmlvaW5mb3JtYXRpY3MgY2VudGVy Ckluc3RpdHV0ZSBvZiBNb2xlY3VsYXIgQmlvbG9neSBhbmQgUGh5c2lvbG9neQpVbml2ZXJzaXR5 IG9mIENvcGVuaGFnZW4KVW5pdmVyc2l0ZXRzcGFya2VuIDE1IC0gQnlnbmluZyAxMApESy0yMTAw IENvcGVuaGFnZW4gw5gKRGVubWFyawpIb21lcGFnZTogaHR0cDovL3d3dy5iaW5mLmt1LmRrL1By b3RlaW5fc3RydWN0dXJlCg== |
|
From: Bill S. <wf...@sa...> - 2006-10-14 12:59:24
|
On Oct 14, 2006, at 2:58 AM, tha...@bi... wrote: > On 10/13/06, Lisandro Dalcin <da...@gm...> wrote: >> This post is surely OT, but I cannot imagine a better place to >> contact >> people about this subject. Please, don't blame me. >> >> Any people here interested in NumPy/SciPy + MPI? > > I've been working on a Dynamic Bayesian Network (DBN) toolkit for > some time > (called Mocapy, freely available from sourceforge https:// > sourceforge.net/projects/mocapy/). The thing is almost entirely > implemented in Python, and currently uses Numeric and pyMPI. I > routinely train DBNs from 100.000s of observations on our 240 CPU > cluster. > > I'm in the proces of porting Mocapy to numpy. I assume pyMPI will > also work with numpy, but I haven't tried it out yet. Would be > great if scipy came with default MPI support, especially since > pyMPI does not seem to be actively developed anymore. I would like to second the notion of converging on a single MPI interface. My parallel project encapsulates most of the inter- processor communication within higher-level objects because the lower- level communication patterns can usually be determined from higher- level data structures. But still, there are times when a user would like access to the lower-level MPI interface. I haven't tried to make my project compatible with any of the several MPI python interfaces out there, because it isn't clear to me which one is most widely used. If one were to emerge -- and even better, if the various independent projects were to then combine their efforts in an open source environment (the way Numeric and numarray are converging to NumPy) -- then this choice would be easy. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wf...@sa... ** |
|
From: Lisandro D. <da...@gm...> - 2006-10-15 00:24:59
|
On 10/14/06, Bill Spotz <wf...@sa...> wrote: > I would like to second the notion of converging on a single MPI > interface. My parallel project encapsulates most of the inter- > processor communication within higher-level objects because the lower- > level communication patterns can usually be determined from higher- > level data structures. But still, there are times when a user would > like access to the lower-level MPI interface. Using mpi4py, you have access to almost all MPI internals directly from the Python side, with an API really similar to MPI-2 C++ bindigs. This is a feature I've not seen in other Python bindings for MPI. I think this is really important for developers and people learning MPI; you do not need to learn a new API. --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Gael V. <gae...@no...> - 2006-10-14 13:34:25
|
On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: > I would like to second the notion of converging on a single MPI =20 > interface. My parallel project encapsulates most of the inter-=20 > processor communication within higher-level objects because the lower-=20 > level communication patterns can usually be determined from higher-=20 > level data structures. But still, there are times when a user would =20 > like access to the lower-level MPI interface. I think we are running in the same old problem of scipy/a super-package of scientific tools. It is important to keep numpy/scipy modular. People may want to install either one without an MPI interface. However it is nice that if somebody just wants "everything" without having to choose he can download that "everything" from scpy.org and that it comes well bundled together. So all we need to do is find a name for that super-package, put it together with good integration and add it to scipy.org. My 2 cents, Ga=EBl |
|
From: A. M. A. <per...@gm...> - 2006-10-14 16:48:16
|
On 14/10/06, Gael Varoquaux <gae...@no...> wrote: > On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: > > I would like to second the notion of converging on a single MPI > > interface. My parallel project encapsulates most of the inter- > > processor communication within higher-level objects because the lower- > > level communication patterns can usually be determined from higher- > > level data structures. But still, there are times when a user would > > like access to the lower-level MPI interface. > > I think we are running in the same old problem of scipy/a super-package > of scientific tools. It is important to keep numpy/scipy modular. People > may want to install either one without an MPI interface. However it is > nice that if somebody just wants "everything" without having to choose > he can download that "everything" from scpy.org and that it comes well > bundled together. So all we need to do is find a name for that > super-package, put it together with good integration and add it to > scipy.org. > > My 2 cents, > > Ga=EBl I agree. Moreover, being picked for such integration work would help encourage people to converge on one MPI interface (for example). There's some discussion of such a package at NumpyProConPage on the wiki (and in particular at http://www.scipy.org/PyLabAwaits ). The "scipy superpack" (which is only for Windows, as I understand it) is a sort of beginning. So, well, any suggestion for a name? pylab is already in use by matplotlib (for some reason), as is Scientific Python (and numpy, Numeric and numarray are obviously already confusing). A. M. Archibald |
|
From: Bill B. <wb...@gm...> - 2006-10-14 16:59:52
|
On 10/15/06, A. M. Archibald <per...@gm...> wrote: > So, well, any suggestion for a name? pylab is already in use by > matplotlib (for some reason), as is Scientific Python (and numpy, > Numeric and numarray are obviously already confusing). I always thought 'pylab' was an odd name, so I always do from matplotlib import pylab as plot I think the matplotlib folks would be willing to change the name of pylab if it were seen as something that would be good for the community. I kind of wonder why they didn't call it 'matplot', personally. --bb |
|
From: Charles R H. <cha...@gm...> - 2006-10-14 16:50:38
|
On 10/14/06, A. M. Archibald <per...@gm...> wrote: > > On 14/10/06, Gael Varoquaux <gae...@no...> wrote: > > On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: <snip> I agree. Moreover, being picked for such integration work would help > encourage people to converge on one MPI interface (for example). > There's some discussion of such a package at NumpyProConPage on the > wiki (and in particular at http://www.scipy.org/PyLabAwaits ). The > "scipy superpack" (which is only for Windows, as I understand it) is a > sort of beginning. > > So, well, any suggestion for a name? pylab is already in use by > matplotlib (for some reason), as is Scientific Python (and numpy, > Numeric and numarray are obviously already confusing). supernumpy? Chuck |
|
From: Gael V. <gae...@no...> - 2006-10-14 16:54:50
|
On Sat, Oct 14, 2006 at 10:50:36AM -0600, Charles R Harris wrote:
> So, well, any suggestion for a name? pylab is already in use by
> matplotlib (for some reason), as is Scientific Python (and numpy,
> Numeric and numarray are obviously already confusing).
> supernumpy?
I think we should rather build upon scipy, rather than numpy, if we
want to build upon an existing name. Numpy is low-level array
manipulation, scipy is all the fancy math that goes above, so it make
more sens to derive a name on scipy. Besides scipy.org is already a good
rallying point for scientific computing with python
How about scipylab ! :->
--=20
Ga=EBl
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X