Re: [Gpiv-user] OpenMP question

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Mattias,
I also reply to the list, so other subscribers know what is going on.
Gerber

On Mon, 2011-05-16 at 00:28 +0200, Matthias Fricke wrote:
> Hi Wernher, hi Gerber!
> 
> to give you a short note on this problem: I'm currently working on this, since I also need the speed of additional cores. As I told Gerber already I think the problem is in the thread-management overhead resulting from the OpenMP '#pragma omp parallel'-lines with which new threads are generated including local copies of all variables . This overhead slows down the calculation process. (Well, this is just what makes most sense to me.)
> 
> So, this is the current status: I removed all current OpenMP-lines and focus on the gpiv_piv_interrogate_img()-function to just parallelise the nested for-loops calling the gpiv_piv_interrogate_ia()-function. With this approach I want to spread equal portions of the grid over the different cores.
> To get a better understanding of the OpenMP-programming I rebuild just the structure of the gpiv_piv_interrogate_img()-function in a simple sample program. Currently I try how to get variables in and out of the parallel region as this may lead to memory access errors under certain conditions. I have already a - hopefully working - idea for this variable handling but first I have to realise it in my sample programs. Gerber, I sent you a more detailed email the next days.
> 
> Unfortunately this solution handles only the case of a grid of interrogation areas. In the rare case of a single point with a very large interrogation area (maybe covering the whole image) there is no multi-core use and therefore no speed up. (You find this distinction in the gpiv_piv_interrogate_img()-function, I think it is the while-loop, that encloses the nested for-loops.) For this case of a single interrogation point/area we have to find another solution.
Personally, I don't have problems with that. However, I wonder if this
would make much sense, as in that case parallelization would be identic
as has been implemented using MPI. So you could use MPI instead. (A
single point doesn't need much speedup, so no need to parallelize here).
The initial idea was to perform a coarse paralellization using MPI for
memory distributed systems, while at each (multi-core) node OMP would
perform the task on lower scale.

I also would like to comment that the gpiv_series program, when MPI is
enabled during configuration (or the -mpi debian package have been
installed), the entire 'process', is scattered to the different nodes.
So, if the 'process' represents gpiv_rr, different images are
interrogated on different nodes. This means a more coarse
parallelization with less overhead. 

Eventually, the MPI protocol can also be used on memory shared systems
(ie multicore). As OMP seems to perform poorly during image
interrogation, I wonder if it makes sense to keep it in the code. Maybe
for PIV-data validation and for image deformation it still is
advatageous.

> 
> 
> Greetings!
> 
> Matthias
> 
> 
> 
> 
> -------- Original-Nachricht --------
> > Datum: Fri, 13 May 2011 12:10:15 +0200
> > Von: Gerber van der Graaf <ger...@gm...>
> > An: gpi...@li...
> > Betreff: Re: [Gpiv-user] OpenMP question
> 
> > Hi Werner
> > Matthias Fricke did report the same on a 4-core system. I checked it out
> > on my dual core and came to the same conclusions that, unfortunately,
> > OMP slows down slightly the image interrogation process.
> > 
> > You can disable the OMP calls by invoking ./configure without the
> > --enable-omp option or by disabling the #pragma OMP statements in the
> > source code of libgpiv.
> > 
> > When you want to use the debian packages for modifying:
> > * apt-get source -b libgpiv3 will download the source and its build
> > dependancies
> > * In libgpiv/debian/rules remove the --enable-omp
> > option for configure and build the debian package with:
> > debuild -us -uc (or dpkg-buildpackage)
> > 
> > On my own system I disabled individual #pragma OMP calls that are used
> > by gpiv_interrogate_img() (in piv.c). Other calls I kept enabled, like
> > for validation and image deformation. While monitoring CPU activity I
> > perfectly could see when CPU-2 was working. This, at least, did not slow
> > down the analyses, but might eventually (I am very carefull here) speed
> > up the process a bit.
> > 
> > It's a pity that OMP thread slows down. I do not how why and how to
> > optimize the image interrogation process for this OMP technology.
> > Something I am probably doing wrong....
> > 
> > When disabled OMP, you might try the MPI technology to try out if this
> > will speed up for interrogation. It already has been implemented and is
> > available in the libgpiv-mpi and gpivtools-mpi packages for
> > debian/ubuntu.
> > 
> > Concerning Pygpiv, there is an example within the source code. In case
> > you installed the pyhton-gpiv package, this can be found
> > in /usr/share/doc/python-gpiv/example/. The API documentation can be
> > used for other libgpiv calls from pygpiv. These API docs are found in
> > the libgpiv3-doc package and online at
> > http://libgpiv.sourceforge.net/doc/index.html
> > 
> > 
> > Greetings, Gerber
> > 
> > 
> > and I have met the same experiences
> > On Fri, 2011-05-13 at 09:31 +0100, Wernher Brevis wrote:
> > > Folks,
> > > 
> > > Hope you are well,
> > > 
> > > Time again I had the following issue but I forgot how to solve it. I
> > > hope you can help me again.
> > > 
> > > I am using a 6-cores AMD computer with Ubuntu 10.10. When I run GPIV 0.6
> > > the 6 CPus are used but the analysis takes longer compared with same
> > > analysis using only one CPU with GPIV 0.5. Due to this issue I am
> > > running 6 simultaneous process of GPIV 0.5, each one using a single
> > > processor. How may I disconnect OpenMP to check if it is having an issue
> > > with my computer Arquitecture?. Do you know anybody else having the same
> > > problem?
> > > 
> > > Thank you in advance,
> > > 
> > > Wernher
> > > 
> > > PS: Does anybody have an example of how to use PyGPIV?
> > > 
> > > 
> > >
> > ------------------------------------------------------------------------------
> > > Achieve unprecedented app performance and reliability
> > > What every C/C++ and Fortran developer should know.
> > > Learn how Intel has extended the reach of its next-generation tools
> > > to help boost performance applications - inlcuding clusters.
> > > http://p.sf.net/sfu/intel-dev2devmay
> > > _______________________________________________
> > > Gpiv-user mailing list
> > > Gpi...@li...
> > > https://lists.sourceforge.net/lists/listinfo/gpiv-user
> > 
> > 
> > 
> > ------------------------------------------------------------------------------
> > Achieve unprecedented app performance and reliability
> > What every C/C++ and Fortran developer should know.
> > Learn how Intel has extended the reach of its next-generation tools
> > to help boost performance applications - inlcuding clusters.
> > http://p.sf.net/sfu/intel-dev2devmay
> > _______________________________________________
> > Gpiv-user mailing list
> > Gpi...@li...
> > https://lists.sourceforge.net/lists/listinfo/gpiv-user
> 