From: Tim K. <tim...@ce...> - 2010-11-04 14:12:26
|
On Thu, 4 Nov 2010, Roy Stogner wrote: > Could we add a libmesh_assert to the destructor, to make sure the > request has been set to MPI_REQUEST_NULL (which I believe both the > wait and free commands do)? No, we can't. (-: Because the current usage (that started this discussion) would be caught by that assert then. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Tim K. <tim...@ce...> - 2010-11-04 14:07:19
|
On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: >>>> Long-term, there are two policy options that I think make sense: >>>> >>>> 1. Don't ever do MPI_Request_free implicitly from a Request object. >>>> >>>> 2. Do reference counting in Request objects; do an MPI_Request_free >>>> when the last reference is destroyed. >>>> >>>> I can see pros and cons to each; I'd appreciate others' opinions. >>> >>> I like #2, but the question is, how much overhead that will produce. >>> My feeling is that it is negligible, but I might be wrong. >> >> Since requests are associated with interprocessor communication I am sure >> the overhead associated with reference counting is trivial in comparison. > > Tim - what compiler and MPI are you using? I'd like to make sure I can > repeat this before declaring we know how to fix it. I don't know at the moment, because the cluster I'm working on is temporarilly out of service, so that I can't login. > What are the downsides of 1.)? Intuitively I would not think that waiting for a request automatically frees this request. However, if you guys are sure that it is guaranteed to do so, this is of course no problem. The other point is that somebody might create a request and then not wait for it. On the other hand, I can't think of any situation in which this should be a sensible thing to do. > When I implemented the request object I clearly misinterpreted the purpose > of MPI_Request_free(). We should add a member Request::free() to do what > the current destructor does, Yes, sure. > although I don't think there is anywhere we need to actually use it. Right. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Tim K. <tim...@ce...> - 2010-11-04 15:28:38
|
On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: > Tim - what compiler and MPI are you using? I'd like to make sure I can > repeat this before declaring we know how to fix it. The cluster is alive again. I'm using: g++ (GCC) 4.1.2 mpich2-1.2 -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-11-04 13:52:37
|
On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: >>> Long-term, there are two policy options that I think make sense: >>> >>> 1. Don't ever do MPI_Request_free implicitly from a Request object. >>> >>> 2. Do reference counting in Request objects; do an MPI_Request_free >>> when the last reference is destroyed. >>> >>> I can see pros and cons to each; I'd appreciate others' opinions. >> >> I like #2, but the question is, how much overhead that will produce. >> My feeling is that it is negligible, but I might be wrong. > > Since requests are associated with interprocessor communication I am sure > the overhead associated with reference counting is trivial in comparison. Ben's exactly right. My only concern with #2 is the implicit shared reference behavior between Request objects and MPI_Request objects. I could imagine someone using a raw MPI_Request to initiate an operation, creating a Request from it, letting the Request go out of scope (freeing the underlying request), then expecting the MPI_Request to still be useful afterwards. Kind of a stretch, but if there are unlikely failure cases that I can quickly come up with I worry that there might be subtle failure cases I just haven't spotted yet. --- Roy |
From: John P. <pet...@ta...> - 2010-11-04 14:27:27
|
On Thu, Nov 4, 2010 at 8:52 AM, Roy Stogner <roy...@ic...> wrote: > > On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: > >>>> Long-term, there are two policy options that I think make sense: >>>> >>>> 1. Don't ever do MPI_Request_free implicitly from a Request object. >>>> >>>> 2. Do reference counting in Request objects; do an MPI_Request_free >>>> when the last reference is destroyed. >>>> >>>> I can see pros and cons to each; I'd appreciate others' opinions. >>> >>> I like #2, but the question is, how much overhead that will produce. >>> My feeling is that it is negligible, but I might be wrong. >> >> Since requests are associated with interprocessor communication I am sure >> the overhead associated with reference counting is trivial in comparison. > > Ben's exactly right. > > My only concern with #2 is the implicit shared reference behavior > between Request objects and MPI_Request objects. I could imagine > someone using a raw MPI_Request to initiate an operation, creating a > Request from it, letting the Request go out of scope (freeing the > underlying request), then expecting the MPI_Request to still be useful > afterwards. Kind of a stretch, but if there are unlikely failure > cases that I can quickly come up with I worry that there might be > subtle failure cases I just haven't spotted yet. I don't think we should worry about reference counting. It's tricky and error prone (as you have shown). In normal MPI code there is almost never a need to cancel an Isend/Irecv, so I think our Request object should just be a wrapper around the MPI_Request and not attempt to manage lifetime in any way. It's true that it's just as easy to leak resources as it was with a naked MPI_Request object, but I'd argue that just doesn't happen too often. Plus, if we provide the Request::free() interface, at least we're giving the user no extra excuses to leak the requests. The documentation should make clear that Request::free() should only be used to cancel an Isend/Irecv, and that all Requests should be waited on before being allowed to go out of scope, just like they are in normal MPI code. -- John |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2010-11-04 14:35:58
|
>> My only concern with #2 is the implicit shared reference behavior >> between Request objects and MPI_Request objects. I could imagine >> someone using a raw MPI_Request to initiate an operation, creating a >> Request from it, letting the Request go out of scope (freeing the >> underlying request), then expecting the MPI_Request to still be useful >> afterwards. Kind of a stretch, but if there are unlikely failure >> cases that I can quickly come up with I worry that there might be >> subtle failure cases I just haven't spotted yet. > > I don't think we should worry about reference counting. It's tricky > and error prone (as you have shown). > > In normal MPI code there is almost never a need to cancel an > Isend/Irecv, so I think our Request object should just be a wrapper > around the MPI_Request and not attempt to manage lifetime in any way. > It's true that it's just as easy to leak resources as it was with a > naked MPI_Request object, but I'd argue that just doesn't happen too > often. Plus, if we provide the Request::free() interface, at least > we're giving the user no extra excuses to leak the requests. The > documentation should make clear that Request::free() should only be > used to cancel an Isend/Irecv, and that all Requests should be waited > on before being allowed to go out of scope, just like they are in > normal MPI code. I'm all for (1). Note thought that MPI_Request_free does not cancel communication - MPI_Cancel does. MPI_Request_free says you have no need for this particular request - possibly because you will be able to infer when it has completed due to some knowledge of program structure. MPI_Cancel is definitely not used in our code and according to the docs 'can be an expensive operation that should be used only exceptionally.' So what about Request::free() and a clean destructor? -Ben |
From: Tim K. <tim...@ce...> - 2010-11-04 14:40:52
Attachments:
patch
|
On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: > I'm all for (1). > > Note thought that MPI_Request_free does not cancel communication - > MPI_Cancel does. MPI_Request_free says you have no need for this particular > request - possibly because you will be able to infer when it has completed > due to some knowledge of program structure. > > MPI_Cancel is definitely not used in our code and according to the docs 'can > be an expensive operation that should be used only exceptionally.' > > So what about Request::free() and a clean destructor? Okay, I'm convinced. I guess the attached patch does everything we need, doesn't it? Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-11-04 15:41:13
|
On Thu, 4 Nov 2010, Tim Kroeger wrote: > On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: > >> I'm all for (1). >> >> Note thought that MPI_Request_free does not cancel communication - >> MPI_Cancel does. MPI_Request_free says you have no need for this >> particular >> request - possibly because you will be able to infer when it has completed >> due to some knowledge of program structure. >> >> MPI_Cancel is definitely not used in our code and according to the docs >> 'can >> be an expensive operation that should be used only exceptionally.' >> >> So what about Request::free() and a clean destructor? > > Okay, I'm convinced. I guess the attached patch does everything we need, > doesn't it? I think that should do it. You want to commit to trunk; I'll commit to (and grumblingly tarball up) 0.7.0.2? Thanks, --- Roy |
From: Tim K. <tim...@ce...> - 2010-11-04 15:44:29
|
On Thu, 4 Nov 2010, Roy Stogner wrote: > > On Thu, 4 Nov 2010, Tim Kroeger wrote: > >> On Thu, 4 Nov 2010, Kirk, Benjamin (JSC-EG311) wrote: >> >>> I'm all for (1). >>> >>> Note thought that MPI_Request_free does not cancel communication - >>> MPI_Cancel does. MPI_Request_free says you have no need for this >>> particular >>> request - possibly because you will be able to infer when it has completed >>> due to some knowledge of program structure. >>> >>> MPI_Cancel is definitely not used in our code and according to the docs >>> 'can >>> be an expensive operation that should be used only exceptionally.' >>> >>> So what about Request::free() and a clean destructor? >> >> Okay, I'm convinced. I guess the attached patch does everything we need, >> doesn't it? > > I think that should do it. You want to commit to trunk; I'll commit > to (and grumblingly tarball up) 0.7.0.2? Okay, it's in now. Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Tim K. <tim...@ce...> - 2010-11-08 15:07:47
|
The next difficulty with my solve-on-part-of-domain stuff has now shown up. It is the constrained-dof issue now, which Roy already predicted and I didn't believe. Luckily enough, my application seems to be sufficiently complicated for any potential problem to actually occur. Let's assume the following situation: a - - - b - - - c - - - - - - - d | | | | | A | B | | | | | | e - - - f - - - g E | | | | | | C | D | | | | | | h - - - i - - - j - - - - - - - k | | | | | | | | | | F | G | | | | | | | | | | l - - - - - - - m - - - - - - - n Elements A, B, and C are inside the active subdomain, the others are outside. My application hence imposes Dirichlet boundary conditions for dofs h, i, f, g, and c, where the values equal the current_local_solution values at the respective dofs. However, using the constrainig procedure, the equation for dof i is replaced with the condition that dof i be the mean value of dofs h and j (likewise for dof g). On first sight, this is no problem because these two dofs are fixed as well, and their mean value coincides with the value of dof i. But then, as PetscLinearSolver creates the submatrix, the matrix row and column corresponding to dof j are removed from the matrix, resulting in the equation that dof i be half the value of dof h. This of course results in a completely wrong solution. To resolve this problem, I thought of the following possibilities: Idea #1: Add constraining dofs (in this case, dof j) to the set of active dofs. The problem with this is that I currently don't assemble anything on element D, so that dof j doesn't have an equation. Idea #2: Ensure that refined elements whose neighbors are not refined are either completely contained in the active subdomain or not at all. I don't like this because it restricts the user's choice of the subdomain. Idea #3: Remove row j from the matrix, but not column j. This results in a non-quadratic matrix, namely an underdetermined system. I wonder what PETSc would do then. I don't think it will do what I would like it to do in this situation. (Jed?) Idea #4: Upon removing the columns of the matrix, automatically adjust the right hand side. I think this would actually be quite easy to do by the following procedure: 1. Remove the inactive rows of the matrix. 2. Let B denote the active columns of the matrix and C all the remaining (inactive) columns. 3. Subtract C*x from b (where b is the right hand side and x the solution vector). 4. Then solve with B as before. Idea #5: Follow Jed's prior objection that just leaving the inactive components of the solution vector unchanged makes the whole thing become a weird beast. For instance, this should be able to be achieved by subtracting an appropriate vector before solving and adding it again afterwards. Any thoughts? Well, the more I think about this, the more I find that #4 is the best choice. It can be done in PetscLinearSolver and will work quite generally (if I don't overlook something), so that the user wouldn't have to care about. Best Regars, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Tim K. <tim...@ce...> - 2010-11-08 15:55:04
|
On Mon, 8 Nov 2010, Tim Kroeger wrote: > Idea #4: Upon removing the columns of the matrix, automatically adjust > the right hand side. I think this would actually be quite easy to do > by the following procedure: > > 1. Remove the inactive rows of the matrix. > > 2. Let B denote the active columns of the matrix and C all the remaining (inactive) columns. > > 3. Subtract C*x from b (where b is the right hand side and x the solution vector). > > 4. Then solve with B as before. After thinking about this some more time, I am now unsure at which point in the code the complement (PETSc) index set should be determined. The most native point would be in PetscLinearSolver::restrict_solve_to(), since this is the point where to other (non-complement) index set is created. But this is not possible because this method (with the current API) has no chance to determine the overall index range, because this method does not get a vector. My idea would be add ab extra method, e.g. PetscLinearSolver::create_complement_is(), to create the complement index set. This method would be called from any solve() method, and if the complement index set has already been computed before, it will do nothing. Any thoughts? -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Derek G. <fri...@gm...> - 2010-11-08 15:18:01
|
On Nov 8, 2010, at 8:07 AM, Tim Kroeger wrote: > The next difficulty with my solve-on-part-of-domain stuff has now > shown up. It is the constrained-dof issue now, which Roy already > predicted and I didn't believe. Luckily enough, my application seems > to be sufficiently complicated for any potential problem to actually > occur. > > Let's assume the following situation: > > a - - - b - - - c - - - - - - - d > | | | | > | A | B | | > | | | | > e - - - f - - - g E | > | | | | > | C | D | | > | | | | > h - - - i - - - j - - - - - - - k > | | | > | | | > | | | > | F | G | > | | | > | | | > | | | > l - - - - - - - m - - - - - - - n I haven't followed this whole thread so forgive me if this has been explained... but how do you end up with a refined element (D) being in a different subdomain from it's parent (that made up ABCD)? Is your subdomain evolving throughout the simulation? Because without that happening this wouldn't be a problem. Derek |
From: Tim K. <tim...@ce...> - 2010-11-08 15:30:51
|
On Mon, 8 Nov 2010, Derek Gaston wrote: > I haven't followed this whole thread so forgive me if this has been > explained... but how do you end up with a refined element (D) being > in a different subdomain from it's parent (that made up ABCD)? Is > your subdomain evolving throughout the simulation? Because without > that happening this wouldn't be a problem. Yes, the subdomain_id values are changing throughout the simulation. Actually, what happens is that in a certain situation I collect the elements that will belong to the subset using certain criteria, and I then just use (one bit of) subdomain_id as a flag. Each time I do this, the subset can be different. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-11-08 16:26:57
|
On Mon, 8 Nov 2010, Tim Kroeger wrote: > Idea #4: Upon removing the columns of the matrix, automatically adjust > the right hand side. I think this would actually be quite easy to do > by the following procedure: > > 1. Remove the inactive rows of the matrix. > > 2. Let B denote the active columns of the matrix and C all the remaining (inactive) columns. > > 3. Subtract C*x from b (where b is the right hand side and x the solution vector). > > 4. Then solve with B as before. > > Idea #5: Follow Jed's prior objection that just leaving the inactive > components of the solution vector unchanged makes the whole thing > become a weird beast. For instance, this should be able to be > achieved by subtracting an appropriate vector before solving and > adding it again afterwards. > > Any thoughts? I dislike ideas #1-3; no opinion on #4 vs #5, but want to add: Idea #6: Add an example 24 (or 25, whatever's free when this is all finally finished) demonstrating a use of SystemSubset stuff. We added constrain_element_matrix_and_vector calls to even the non-adaptive examples a while back after the third time I helped someone fix their "why is my modified-from-ex4 application giving wrong results on adaptive meshes" bug, and this code is less intuitive than that. I'd like to be sure it doesn't get left out of regression testing too, in part for the same reason. --- Roy |
From: Tim K. <tim...@ce...> - 2010-11-09 08:33:50
Attachments:
Makefile
|
On Mon, 8 Nov 2010, Roy Stogner wrote: > I dislike ideas #1-3; no opinion on #4 vs #5, but want to add: > > Idea #6: Add an example 24 (or 25, whatever's free when this is all > finally finished) demonstrating a use of SystemSubset stuff. Good point. I created an example now, see attachment. I will check it in (renamed to the appropriate number) when everything has been finished. > I'd like to be sure it doesn't get left out of regression testing > too, in part for the same reason. Are all the examples automatically regression-tested, or would there be something addtional to do? Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-11-09 14:18:48
|
On Tue, 9 Nov 2010, Tim Kroeger wrote: > On Mon, 8 Nov 2010, Roy Stogner wrote: > >> I dislike ideas #1-3; no opinion on #4 vs #5, but want to add: >> >> Idea #6: Add an example 24 (or 25, whatever's free when this is all >> finally finished) demonstrating a use of SystemSubset stuff. > > Good point. I created an example now, see attachment. I will check it in > (renamed to the appropriate number) when everything has been finished. Looks good; thanks! >> I'd like to be sure it doesn't get left out of regression testing too, in >> part for the same reason. > > Are all the examples automatically regression-tested, or would there be > something addtional to do? They're automatically tested for assertion failures; there's currently no automatic testing for output changes. --- Roy |
From: Tim K. <tim...@ce...> - 2010-11-18 13:16:25
Attachments:
patch
|
The subset-solve stuff now seems to work properly. At least, my application does not crash, and the output looks reasonable. I'd like to check-in this now, see attached patch, together with the example that I posted last week. If somebody has objections, please let me know. Otherwise I will check-in everything tomorrow or on Monday. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Tim K. <tim...@ce...> - 2010-11-22 08:58:04
|
On Thu, 18 Nov 2010, Tim Kroeger wrote: > The subset-solve stuff now seems to work properly. At least, my application > does not crash, and the output looks reasonable. I'd like to check-in this > now, see attached patch, together with the example that I posted last week. > If somebody has objections, please let me know. Otherwise I will check-in > everything tomorrow or on Monday. Checked in everything now. I also updated the class docs, but there seems to be something strange with this. That is, the inherticance diagrams now look different than they did before. Did I do something wrong? I did make doc LIBMESH_SVN_USER=sheep_tk make upload Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-11-22 17:30:37
|
On Mon, 22 Nov 2010, Tim Kroeger wrote: > Checked in everything now. I also updated the class docs, but there seems to > be something strange with this. That is, the inherticance diagrams now look > different than they did before. Did I do something wrong? I did > > make doc > LIBMESH_SVN_USER=sheep_tk make upload My best guess: you're running a different Doxygen version than we had uploaded previously? Not sure if it's an upgrade or a downgrade. --- Roy |
From: Tim K. <tim...@ce...> - 2010-11-23 07:24:20
|
On Mon, 22 Nov 2010, Roy Stogner wrote: > On Mon, 22 Nov 2010, Tim Kroeger wrote: > >> Checked in everything now. I also updated the class docs, but there seems >> to be something strange with this. That is, the inherticance diagrams now >> look different than they did before. Did I do something wrong? I did >> >> make doc >> LIBMESH_SVN_USER=sheep_tk make upload > > My best guess: you're running a different Doxygen version than we had > uploaded previously? Not sure if it's an upgrade or a downgrade. tim@tkroeger-pc:~$ doxygen --version 1.6.1 Although not quite up-to-date, it isn't ancient either. Which version are you using? I liked the previous style better, but it seems I can't restore it, and if anybody can and wants to do so, they just should do so. It's not very probable that I will update the documentation too often in the future. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |
From: Roy S. <roy...@ic...> - 2010-12-02 03:17:35
|
On Tue, 23 Nov 2010, Tim Kroeger wrote: > tim@tkroeger-pc:~$ doxygen --version > 1.6.1 > > Although not quite up-to-date, it isn't ancient either. Which version are > you using? 1.7.1 at the moment, but I think I've used 1.5.something earlier with no problems. But I guess the important program for the graphs is "dot"? Mine claims roystgnr@mycroft:~/libmesh/svn$ dot -V dot - graphviz version 2.26.3 (20100126.1600) But a lot of these graphics programs have more configure-time options than we do; even if your version is newer it's possible that your build just had something or other disabled. > I liked the previous style better, but it seems I can't restore it, and if > anybody can and wants to do so, they just should do so. It's not very > probable that I will update the documentation too often in the future. I'm hoping to do an update soon to rectify our grossly lapsed online example docs; since my dot settings seem to be generating nicer fonts and graph arrows those will be cleaned up too. --- Roy |
From: Tim K. <tim...@ce...> - 2010-12-02 07:21:49
|
On Wed, 1 Dec 2010, Roy Stogner wrote: > On Tue, 23 Nov 2010, Tim Kroeger wrote: > >> tim@tkroeger-pc:~$ doxygen --version >> 1.6.1 >> >> Although not quite up-to-date, it isn't ancient either. Which version are >> you using? > > 1.7.1 at the moment, but I think I've used 1.5.something earlier with > no problems. > > But I guess the important program for the graphs is "dot"? Mine > claims > > roystgnr@mycroft:~/libmesh/svn$ dot -V > dot - graphviz version 2.26.3 (20100126.1600) Good point. I actually hadn't installed "dot" at all. Installing it, reconfiguring libMesh, rebuilding and re-uploading the documentation now made the graphs look nice as before. Thank you very much. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim...@ce... Universitaetsallee 29 tim...@me... D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 |