libmesh-devel — General information for libmesh developers

Re: [Libmesh-devel] Still Really Dislike Automake

[Kirk, B. (JSC-EG311) <ben...@na...>]

Any chance your modifications could create a build system that can coexist?

What I dislike more than automake is having to spend a good chunk of time building the build system on a machine that’s not particularly modern…

-Ben



> On Aug 30, 2018, at 9:47 AM, Derek Gaston <fri...@gm...> wrote:
> 
> After all of these years: I still dislike the Automake build system in libMesh.  I still can't believe that we threw out a perfectly decent build system and saddled ourselves with this thing.
> 
> In hindsight: do you guys TRULY think it was worth it?
> 
> I still maintain a set of scripts that allow me to do in-tree builds.  In addition - I often prototype things in MOOSE / a new MOOSE application before moving it down to libMesh (if I ever do - I am also just increasingly NOT moving stuff to libMesh!).  All of this because of the build system.
> 
> All I'm trying to do right now is add ONE damn header file - and I can't seem to find the magic script / sauce to get everything updated.  I ran include/rebuild_include_HEADERS.sh ... nope.  I ran bootstrap... still nope.  Oh!  Now I remember... I need to run include/libmesh/rebuild_makefile.sh... NOPE!
> 
> Ok - 100% serious here.  If I redo the libMesh build system so that it's autoconf and make based again - would you guys consider switching to it?  I am willing to even code up a "make install" (which I believe was the original reason for wanting to use Automake).
> 
> Derek
> 
> PS - if anyone could tell me what I need to do to get the symlink built in include/libmesh for a new header file I would greatly appreciate it!
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
> Libmesh-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-devel

[Libmesh-devel] Still Really Dislike Automake

[Derek G. <fri...@gm...>]

After all of these years: I still dislike the Automake build system in
libMesh.  I still can't believe that we threw out a perfectly decent build
system and saddled ourselves with this thing.

In hindsight: do you guys TRULY think it was worth it?

I still maintain a set of scripts that allow me to do in-tree builds.  In
addition - I often prototype things in MOOSE / a new MOOSE application
before moving it down to libMesh (if I ever do - I am also just
increasingly NOT moving stuff to libMesh!).  All of this because of the
build system.

All I'm trying to do right now is add ONE damn header file - and I can't
seem to find the magic script / sauce to get everything updated.  I
ran include/rebuild_include_HEADERS.sh ... nope.  I ran bootstrap... still
nope.  Oh!  Now I remember... I need to run
include/libmesh/rebuild_makefile.sh... NOPE!

Ok - 100% serious here.  If I redo the libMesh build system so that it's
autoconf and make based again - would you guys consider switching to it?  I
am willing to even code up a "make install" (which I believe was the
original reason for wanting to use Automake).

Derek

PS - if anyone could tell me what I need to do to get the symlink built in
include/libmesh for a new header file I would greatly appreciate it!

Re: [Libmesh-devel] Refinement flag assert trip when using GMG + AMR

[Boris B. <bor...@bu...>]

For posterity's sake, I'd just like to note that this issue has been
migrated to https://github.com/libMesh/libmesh/issues/1832 under some
slightly simpler mesh configurations.

On Wed, Aug 22, 2018 at 11:50 AM, Boris Boutkov <bor...@bu...>
wrote:

> Hello all,
>
> Ive been looking into using GMG with AMR on an L-shaped domain problem and
> am having issues uniformly_coarsening during the GMG setup phase after I've
> refined some elements on the first AMR step in both serial and parallel
> situations. I am using elem_fraction AMR flagging and am setting a small
> refinement_percentage and have set coarsening_percentage to 0 to keep
> things simple.
>
> Specifically in this 2 level GMG scenario, I  end up tripping the assert
> on elem_refinement.C:157 i.e.
> libmesh_assert_equal_to (mychild.refinement_flag(), Elem::COARSEN) while
> trying to coarsen an element, at this point mychild.refinement_flag() ==
> coarsen_inactive.
>
> When I hit the assert, the element in question that's being coarsened is
> actually already inactive, level 0 and having a coarsen_inactive refinement
> flag, and has children with refinement flags: c[0]: inactive, c[1]:
> coarsen_inactive, (both of which are inactive) while c[2], c[3]: are active
> with a flag == coarsen.
>
> While the c[0,2,3] children have no children themselves, c[1] does, and
> has all its children as active, but with a refinement flag == coarsen.
>
> In short, I've been trying to understand how the refinement flags got into
> such a state but haven't had much luck so far and wondering if there were
> any clues about how I could have ended up in such a situation while trying
> to do the uniform coarsening.
>
> Thanks in advance for any info provided,
> - Boris
>
>
>

[Libmesh-devel] Refinement flag assert trip when using GMG + AMR

[Boris B. <bor...@bu...>]

Hello all,

Ive been looking into using GMG with AMR on an L-shaped domain problem and
am having issues uniformly_coarsening during the GMG setup phase after I've
refined some elements on the first AMR step in both serial and parallel
situations. I am using elem_fraction AMR flagging and am setting a small
refinement_percentage and have set coarsening_percentage to 0 to keep
things simple.

Specifically in this 2 level GMG scenario, I  end up tripping the assert on
elem_refinement.C:157 i.e.
libmesh_assert_equal_to (mychild.refinement_flag(), Elem::COARSEN) while
trying to coarsen an element, at this point mychild.refinement_flag() ==
coarsen_inactive.

When I hit the assert, the element in question that's being coarsened is
actually already inactive, level 0 and having a coarsen_inactive refinement
flag, and has children with refinement flags: c[0]: inactive, c[1]:
coarsen_inactive, (both of which are inactive) while c[2], c[3]: are active
with a flag == coarsen.

While the c[0,2,3] children have no children themselves, c[1] does, and has
all its children as active, but with a refinement flag == coarsen.

In short, I've been trying to understand how the refinement flags got into
such a state but haven't had much luck so far and wondering if there were
any clues about how I could have ended up in such a situation while trying
to do the uniform coarsening.

Thanks in advance for any info provided,
- Boris

Re: [Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Roy S. <roy...@ic...>]

On Wed, 20 Jun 2018, Derek Gaston wrote:

> On Wed, Jun 20, 2018 at 1:56 PM Roy Stogner <roy...@ic...> wrote:
>       Hmm... would it be sane to write a shim iterator class, just something
>       to prevent conversion to std::data_type_now_locked_in_stone::iterator,
>       and then return a pair of those?
> 
> Can we just assume that everyone will capture the return value using
> "auto" so it's not a problem?  :-)

Well, we definitely *can*, and I'm lazy enough that I'm tempted...

> From some snooping around it looks like MultiMap returns
> BidirectionalIterators... could we just make a std::pair of those
> and return them?

I wish.  Those aren't a real type, though, they're what the C++
standards people call "concepts" or "named requirements".  But
language support for concepts has been getting deferred for years, and
there's no underlying class you can attach them to.
---
Roy

Re: [Libmesh-devel] DofObject::var_to_vg()

[Derek G. <fri...@gm...>]

On Wed, Jun 20, 2018 at 2:06 PM Roy Stogner <roy...@ic...>
wrote:

> No kidding.  How many variables in the group?
>

For my current problem: 8.  But the problems I'll actually be running tend
to have more like 64.


> You mean using dof_number()?  DofMap::dof_indices() will at minimum
> return you all the dofs for a single variable.
>

Yep - sorry about that


> No, but (if you're using dof_number()) it sounds like what you want is
> a dof_number_group() call instead.  Then you can determine vg and vig
> *once* rather than in each dof_number() call.
>

Yep - that would be good!


> Wait, I'm being stupid.  We don't actually have to make a pointer to
> it a member variable!  Every time we call dof_number() we're doing so
> with a DofMap in hand, so it would be easy to cache lookups there and
> then pass a DofMap * to the DofObject as an argument with which to
> access that cache.
>

Sounds good to me!

Derek

Re: [Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Derek G. <fri...@gm...>]

On Wed, Jun 20, 2018 at 1:56 PM Roy Stogner <roy...@ic...>
wrote:

> Hmm... would it be sane to write a shim iterator class, just something
> to prevent conversion to std::data_type_now_locked_in_stone::iterator,
> and then return a pair of those?
>

Can we just assume that everyone will capture the return value using "auto"
so it's not a problem?  :-)

>From some snooping around it looks like MultiMap returns
BidirectionalIterators... could we just make a std::pair of those and
return them?

Derek

Re: [Libmesh-devel] DofObject::var_to_vg()

[Roy S. <roy...@ic...>]

On Wed, 20 Jun 2018, Derek Gaston wrote:

> Right now I have exactly _one_ var group (which makes this even more
> puzzling)...

No kidding.  How many variables in the group?

> and I'm trying to just pull out the first dof from that group
> using dof_indices().  I then do all the rest of my indexing off of
> that first dof for all the other dofs in the group.

You mean using dof_number()?  DofMap::dof_indices() will at minimum
return you all the dofs for a single variable.

> Maybe there's a better way to be doing that?  Is there a more
> straightforward way to get the first dof in a var_group?

No, but (if you're using dof_number()) it sounds like what you want is
a dof_number_group() call instead.  Then you can determine vg and vig
*once* rather than in each dof_number() call.

We could use the same thing in DofMap::_dof_indices and
DofMap::old_dof_indices if it works well.

> I think I wouldn't mind just storing a var_num->var_group vector
> somewhere that every DofObject has access to.  The variable numbers
> are always contiguous starting from 0 so we should just be able to
> index into it.

The trouble is "has access to".  I'd rather avoid adding another 8
bytes per node+elem if we can.

Wait, I'm being stupid.  We don't actually have to make a pointer to
it a member variable!  Every time we call dof_number() we're doing so
with a DofMap in hand, so it would be easy to cache lookups there and
then pass a DofMap * to the DofObject as an argument with which to
access that cache.
---
Roy

Re: [Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Roy S. <roy...@ic...>]

On Wed, 20 Jun 2018, Derek Gaston wrote:

> It turns out that I wasn't reusing the vector as well as I thought -
> which was causing some of the re-allocation.  That is solved now so
> the re-allocation is at a minimal level.  I'm doing a reserve()
> myself to put plenty of space in that vector to keep down
> re-allocation.

Good to hear!

> That said: I would still like to think about an interface that
> doesn't involve any allocation.  Even just one that returns the
> results of the equal_range() would be nice.

Hmm... would it be sane to write a shim iterator class, just something
to prevent conversion to std::data_type_now_locked_in_stone::iterator,
and then return a pair of those?
---
Roy

Re: [Libmesh-devel] DofObject::var_to_vg()

[Derek G. <fri...@gm...>]

Interesting thoughts.

Right now I have exactly _one_ var group (which makes this even more
puzzling)... and I'm trying to just pull out the first dof from that group
using dof_indices().  I then do all the rest of my indexing off of that
first dof for all the other dofs in the group.

Maybe there's a better way to be doing that?  Is there a more
straightforward way to get the first dof in a var_group?

I think I wouldn't mind just storing a var_num->var_group vector somewhere
that every DofObject has access to.  The variable numbers are always
contiguous starting from 0 so we should just be able to index into it.

Derek

On Wed, Jun 20, 2018 at 8:11 AM Roy Stogner <roy...@ic...>
wrote:

>
> On Tue, 19 Jun 2018, Derek Gaston wrote:
>
> > (Note: this falls into the category of "microptimization" but with
> > my current app that's important).
>
> I've seen this method show up in other timings too... maybe it was
> RELAP-7?  Anyway, if we can speed it up your current app won't be the
> only beneficiary.
>
> > Something that consistently shows up in my timing with MOOSE and my
> > new app is DofObject::var_to_vg().  It is gets called in a number of
> > places throughout DofObject... and I'm wondering if it's currently
> > implemented in the most optimal way.
> >
> > It's doing a small loop in there and I'm wondering if it's
> > necessary.  Is it not possible to directly compute the var group?
> >
> > vg_dof_base() seems to do some direct calculation - can we use that
> > instead?
>
> vg_dof_base already has the variable group index vg, which is what we
> can use for a direct lookup of data related to all variables in that
> group; buried in that index buffer are subbuffers indexed by vg.
>
> var_to_vg has the variable index v, and is trying to figure out vg.
> We can only do that implicitly right now, since there may be an
> arbitrary number of variables in each group.
>
> I can think of ways to make that lookup faster in some cases, but
> always at the cost of more space or more time in other cases or much
> less code modularity, so I guess my first question is: what's your
> case look like?
>
> How many variables, and how are they grouped?
>
> What is the heavy user of var_to_vg() for you?
>
>
> A couple random ideas that might be worth trying:
>
> We could cache a var_to_vg lookup table, and give every DofObject a
> pointer to that table.  Less modular and an extra 8 bytes per object
> but it'd be faster in at least some cases.
>
> We could store n_vars in a cumulative sense, which would then let us
> do a binary search instead of a linear search.  Slightly slower for
> most use cases, but much faster for cases with lots of variable
> groups.
>
> We could try to avoid using var_to_vg-reliant methods in more places?
> Or maybe that's already been done... we're consistently using
> set_n_comp_group instead of set_n_comp, we're not using set_dof_number
> internally anymore... but n_comp and dof_number both rely on
> var_to_vg, and we don't use n_comp_group everywhere we might, and we
> don't even *have* a dof_number_group, even though we use both n_comp
> and dof_number in lots of loops that could be optimized in such a way.
> ---
> Roy

Re: [Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Derek G. <fri...@gm...>]

It turns out that I wasn't reusing the vector as well as I thought - which
was causing some of the re-allocation.  That is solved now so the
re-allocation is at a minimal level.  I'm doing a reserve() myself to put
plenty of space in that vector to keep down re-allocation.

That said: I would still like to think about an interface that doesn't
involve any allocation.  Even just one that returns the results of the
equal_range() would be nice.

Derek

On Wed, Jun 20, 2018 at 8:25 AM Roy Stogner <roy...@ic...>
wrote:

>
> On Tue, 19 Jun 2018, Derek Gaston wrote:
>
> > Sorry to send two messages in a row
>
> It's upsetting to me when people put lots of thought into how to
> optimize libMesh?  Grr?  Argh?
>
> > but I'm wondering if it would be ok to expose
> > BoundaryInfo::_boundary_side_id ?
>
> My first inclination is "hell no" but that may just mean I haven't
> thought about it enough.
>
> I don't like exposing members since that basically freezes our data
> structures.  It'd be sad if e.g. we discovered we could shave
> O(log(N)) off of some heavy code by switching from multimap to
> unordered_multimap, but were then precluded from doing so by backwards
> compatibility concerns.
>
> I think I'd be willing to compromise with a getter tagged
> libmesh_experimental().
>
> > The issue I'm having is that calling boundary_ids() is "slow" (and
> > causes memory allocations which is actually the thing I'm trying to
> > remove) because it passes in a std::vector that it fills...
>
> > I am reusing the vec_to_fill... but it's still not enough to keep
> > down the memory allocation/deallocation associated with that
> > routine.
>
> Stupid question - why not?  If you're reusing the vector then there
> should only be allocation O(log2(N)) times at most, where N is the
> size of your loop.  You could get that down to O(1) by calling
> reserve() on the vector yourself before starting.
>
> > Barring that - would be it ok if I make boundary_ids() reserve()
> > enough size in the vec_to_fill to potentially hold all of the IDs
> > from the equal_range()?  That should help out with the reallocs.
>
> Should it?  vector::clear() isn't allowed by the C++ standard to
> modify vector capacity, so if your input vector was large enough to
> hold the whole range then there shouldn't be any reallocs.
>
> > Another option is to template boundary_ids() so that I can pass in
> > my own container that is statically sized (over-sized to hold the
> > max number of IDs) and won't do any allocation.
>
> If we actually need this, I don't have any objections to it, but I
> don't yet understand why we need it.
> ---
> Roy

Re: [Libmesh-devel] DofObject::var_to_vg()

[Roy S. <roy...@ic...>]

On Tue, 19 Jun 2018, Derek Gaston wrote:

> (Note: this falls into the category of "microptimization" but with
> my current app that's important).

I've seen this method show up in other timings too... maybe it was
RELAP-7?  Anyway, if we can speed it up your current app won't be the
only beneficiary.

> Something that consistently shows up in my timing with MOOSE and my
> new app is DofObject::var_to_vg().  It is gets called in a number of
> places throughout DofObject... and I'm wondering if it's currently
> implemented in the most optimal way.
> 
> It's doing a small loop in there and I'm wondering if it's
> necessary.  Is it not possible to directly compute the var group?
> 
> vg_dof_base() seems to do some direct calculation - can we use that
> instead?

vg_dof_base already has the variable group index vg, which is what we
can use for a direct lookup of data related to all variables in that
group; buried in that index buffer are subbuffers indexed by vg.

var_to_vg has the variable index v, and is trying to figure out vg.
We can only do that implicitly right now, since there may be an
arbitrary number of variables in each group.

I can think of ways to make that lookup faster in some cases, but
always at the cost of more space or more time in other cases or much
less code modularity, so I guess my first question is: what's your
case look like?

How many variables, and how are they grouped?

What is the heavy user of var_to_vg() for you?


A couple random ideas that might be worth trying:

We could cache a var_to_vg lookup table, and give every DofObject a
pointer to that table.  Less modular and an extra 8 bytes per object
but it'd be faster in at least some cases.

We could store n_vars in a cumulative sense, which would then let us
do a binary search instead of a linear search.  Slightly slower for
most use cases, but much faster for cases with lots of variable
groups.

We could try to avoid using var_to_vg-reliant methods in more places?
Or maybe that's already been done... we're consistently using
set_n_comp_group instead of set_n_comp, we're not using set_dof_number
internally anymore... but n_comp and dof_number both rely on
var_to_vg, and we don't use n_comp_group everywhere we might, and we
don't even *have* a dof_number_group, even though we use both n_comp
and dof_number in lots of loops that could be optimized in such a way.
---
Roy

Re: [Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Roy S. <roy...@ic...>]

On Tue, 19 Jun 2018, Derek Gaston wrote:

> Sorry to send two messages in a row

It's upsetting to me when people put lots of thought into how to
optimize libMesh?  Grr?  Argh?

> but I'm wondering if it would be ok to expose
> BoundaryInfo::_boundary_side_id ?

My first inclination is "hell no" but that may just mean I haven't
thought about it enough.

I don't like exposing members since that basically freezes our data
structures.  It'd be sad if e.g. we discovered we could shave
O(log(N)) off of some heavy code by switching from multimap to
unordered_multimap, but were then precluded from doing so by backwards
compatibility concerns.

I think I'd be willing to compromise with a getter tagged
libmesh_experimental().

> The issue I'm having is that calling boundary_ids() is "slow" (and
> causes memory allocations which is actually the thing I'm trying to
> remove) because it passes in a std::vector that it fills...

> I am reusing the vec_to_fill... but it's still not enough to keep
> down the memory allocation/deallocation associated with that
> routine.

Stupid question - why not?  If you're reusing the vector then there
should only be allocation O(log2(N)) times at most, where N is the
size of your loop.  You could get that down to O(1) by calling
reserve() on the vector yourself before starting.

> Barring that - would be it ok if I make boundary_ids() reserve()
> enough size in the vec_to_fill to potentially hold all of the IDs
> from the equal_range()?  That should help out with the reallocs.

Should it?  vector::clear() isn't allowed by the C++ standard to
modify vector capacity, so if your input vector was large enough to
hold the whole range then there shouldn't be any reallocs.

> Another option is to template boundary_ids() so that I can pass in
> my own container that is statically sized (over-sized to hold the
> max number of IDs) and won't do any allocation.

If we actually need this, I don't have any objections to it, but I
don't yet understand why we need it.
---
Roy

[Libmesh-devel] Exposing _boundary_side_id / Optimizing boundary_ids()

[Derek G. <fri...@gm...>]

Sorry to send two messages in a row - but I'm wondering if it would be ok
to expose BoundaryInfo::_boundary_side_id ?

The issue I'm having is that calling boundary_ids() is "slow" (and causes
memory allocations which is actually the thing I'm trying to remove)
because it passes in a std::vector that it fills...

Barring that - would be it ok if I make boundary_ids() reserve() enough
size in the vec_to_fill to potentially hold all of the IDs from the
equal_range()?  That should help out with the reallocs.

I am reusing the vec_to_fill... but it's still not enough to keep down the
memory allocation/deallocation associated with that routine.

Another option is to template boundary_ids() so that I can pass in my own
container that is statically sized (over-sized to hold the max number of
IDs) and won't do any allocation.

Derek

[Libmesh-devel] DofObject::var_to_vg()

[Derek G. <fri...@gm...>]

(Note: this falls into the category of "microptimization" but with my
current app that's important).

Something that consistently shows up in my timing with MOOSE and my new app
is DofObject::var_to_vg().

It is gets called in a number of places throughout DofObject... and I'm
wondering if it's currently implemented in the most optimal way.

It's doing a small loop in there and I'm wondering if it's necessary.  Is
it not possible to directly compute the var group?

vg_dof_base() seems to do some direct calculation - can we use that instead?

Derek

Re: [Libmesh-devel] libMesh/DistributedMesh Work

[Roy S. <roy...@ic...>]

On Wed, 2 May 2018, Tuzun, Robert wrote:

> Suppose a user of the class wanted to set or retrieve element n.
> This could be done by setting or retrieving bit n%8 of Bytes[n/8].
>
> From the previous email's remarks about race conditions, this would
> be thread-safe, correct, even when writing to?

Nope.  This is basically how vector<bool> already works under the hood.

The race condition happens when trying to set or retrieve that bit:
you look at the current byte, you OR that bit to 1, and you write the
result, right?  Nope.  Because if *another* thread tries to do the
same thing, and both look at the current byte at the same time, then
neither thread gets the set-to-1 version from the other thread, and
only one or the other write actually occurs.

There's probably some std::atomic way to fix this but nothing I'd feel
safe trying to futz with, and even a perfect fix would necessarily
ruin caching and thus speed.
---
Roy

Re: [Libmesh-devel] libMesh/DistributedMesh Work

[Roy S. <roy...@ic...>]

On Wed, 2 May 2018, Tuzun, Robert wrote:

> (1) define a typedef BoolLike to unsigned char, and also define constants TrueLike to 1 and FalseLike to 0.

true implicitly converts to 1, false implicitly to 0, so that last
part's mostly unnecessary...

Although if we *really* wanted to be able to ignore when we're not
using a real bool, we'd need a whole shim class, I think.  Otherwise

bool foo = 3, bar = 4;
assert(foo == bar);

suddenly breaks if you try to convert it to BoolLike.

> (2) use vector<BoolLike>
> 
> Nice, contiguous memory, no excess memory usage, and readable.

That's a good idea.  We should probably be doing that everywhere we're
currently using small vectors of semantically-bool contents.  Then it
doesn't matter that the question of what's faster is system-dependent;
if the answer changes then you just change the typedef accordingly.
---
Roy

Re: [Libmesh-devel] libMesh/DistributedMesh Work

[Roy S. <roy...@ic...>]

On Wed, 2 May 2018, Derek Gaston wrote:

> I'll add one more reason to hate std::vector<bool>: threading.

Yikes, I forgot that!  Glad I added you all to the Cc; thanks!

> std::vector<bool> is not at _all_ thread safe: multiple processors working on the
> same vector will clobber each other's values.

Assuming by "working on" you mean "writing to", then yeah:

"Notwithstanding (17.6.5.9), implementations are required to avoid
data races when the contents of the contained object in different
elements in the same container, excepting vector<bool>, are modified
concurrently."

You're supposed to still be able to read in multiple threads
concurrently just fine, though.

> I personally wish they had made std::vector<bool> take 8 times more
> memory... and then let you use something else if you needed the
> optimization... but that's just me.

You, and Hinnant... and honestly I agree too, just not quite as
strongly.  I'm deeply opposed to the "resizeable bitsets shouldn't
exist" attitude I see sometimes, but "resizeable bitsets shouldn't
have been named std::vector" is pretty hard to argue with.
---
Roy

Re: [Libmesh-devel] libMesh/DistributedMesh Work

[Derek G. <fri...@gm...>]

I'll add one more reason to hate std::vector<bool>: threading.
 std::vector<bool> is not at _all_ thread safe: multiple processors working
on the same vector will clobber each other's values.  This leads to hair
pulling out race conditions that take forever to track down.

I personally wish they had made std::vector<bool> take 8 times more
memory... and then let you use something else if you needed the
optimization... but that's just me.

Derek

On Wed, May 2, 2018 at 2:24 PM Roy Stogner <roy...@ic...> wrote:

>
> I hope you don't mind, but I'm going to Cc: this to libmesh-devel.
> These are good questions, and my opinions do not match those of all
> the other developers and it might be unfair not to let them chime in.
>
> On Wed, 2 May 2018, Tuzun, Robert wrote:
>
> > Greetings.  I'm still in the middle of my hiatus, but there was one
> > question I've been meaning to ask.  I notice, in several places, the
> > use of vector<bool> in the code.  Way back in 2001, Scott Meyers
> > (Effective STL, Item 18) recommended against the use of vector<bool>
> > and recommended using deque<bool> or bitset instead.
>
> "As an STL container, there are really only two things wrong with
> vector<bool>.  First, it's not an STL container.  Second, it doesn't
> hold bools.  Other than that, there's not much to object to." - Scott
> Meyers.  I seriously love his writing.  It takes something special to
> make programming language minutia so entertaining.
>
> This is indeed often annoying, because the inability to take real
> references to vector entries or to get a pointer to a underlying array
> means we often can't handle vector<bool> with the same generic code as
> we use to handle vector<literally_anything_else>.  I assume you first
> encountered vector<bool> in our parallel_implementation.h overloads,
> written to handle just that problem?
>
> However, the recommended alternatives are even worse in many cases.
> If a code is always fine using deque<bool>, if using EIGHT TIMES as
> much memory never fazes the author, then Meyers' hundreds of C++
> recommendations should be ignored in favor of "Item 1: Use Java
> Instead" (or maybe Python or Go these days?).  Conversely, if a code
> is always happy with bitset, and never needs an array that can be
> resized, then what are you writing, autotranslated Fortran 77?
>
> Personally, I'm mostly with Howard Hinnant:
> https://isocpp.org/blog/2012/11/on-vectorbool
>
> What's abominable about vector<bool> is the name, because that's
> basically a lie about supporting the whole std::vector interface, but
> the data structure itself is worthwhile and often useful.  And I'm
> even sympathetic to the name!  Once you create a few well-chosen
> overloads (like the parallel_implementation.h stuff) then the rest of
> your generic code can *mostly* support vector<bool> as if it were a
> real std::vector.  And when you miss a needed specialization, the
> resulting breakage is a compile-time failure, so it's a hassle rather
> than a disaster.
>
>
> > I don't know if this advice still holds, and my search on the Web
> > yields conflicting answers (just do a Google search on vector<bool>
> > and you'll see what I mean).  Is there reason for concern?
>
> Well, despite everything I said above: yes, there is.
>
>
> First, to Meyers' objections I'd also have added "Third, it requires
> bit-fiddling for every access", which means if you have a vector small
> enough to *not* worry about memory savings, maybe you ought to be
> using vector<unsigned char> instead.  Grepping through the code for
> short logically-boolean vectors, I see that we're doing the
> unsigned-char-really-holding-bool trick in dof_map.C, and
> mesh_communication_global_indices.C... but we're using small
> vector<bool> in about a dozen places?
>
> I'm not going to change those without benchmarking first, because
> "unsigned char is faster due to less bit fiddling" vs "bool is faster
> due to fitting more stuff in cache" is the sort of system-dependent
> question I can't answer a priori, and in the absence of data I think
> vector<bool> is more self-documenting for something that's logically
> boolean.  If anyone wants to try to prove me wrong in the libMesh
> context, I'd say try System::boundary_project_solution for the
> benchmark.  That has a few vector<bool> inside loops over elements,
> so it looks like the easiest place to expose performance differences.
>
>
> Second and more seriously, in libMesh whenever find ourselves in a
> situation where tactically vector<bool> is definitely the right
> choice, it may mean that strategically we've made a huge mistake!
>
> Our classic use case for vector<bool> are vectors indexed over nodes
> or over elements, where we need to support millions or billions of
> entries and so using 8 times as much space is clearly a bad idea.
>
> *However*, some of that code was a reasonable design only because it
> dates back to ReplicatedMesh-only days, where we couldn't get away
> from having data stored for every element on every processor.  These
> days, a DistributedMesh user should be able to avoid ever requiring
> O(N_elements) memory on a single MPI rank, which means that
> applications should never allocate such a long vector!  So the library
> should never be using one, at least on a non-debug-mode
> non-ReplicatedMesh-only code path.
>
> So let's see...
>
> Some of these uses are OK.  E.g. the MeshTools assertion code is all
> debug-mode-only, where readability is much more important than
> efficiency.
>
> Some are more dubious.  E.g.  UnstructuredMesh::all_first_order() only
> gets used in practice for writing serialized viz output of an error
> estimate, so making it perfectly scalable would require improving
> multiple other parts of that code path first, meaning it's suboptimal
> but not high priority.
>
> Some of these uses are probably outweighed by other distributed mesh
> incompatibility issues.  At first glance I'd guess
> MeshTools::Modification::Smooth may not even work on a distributed
> mesh, but we've just never tried to use it that way.  Looks like
> LaplaceMeshSmoother is distributed-mesh-aware, so maybe everyone who
> needs smoothing post-distribution has just been using that?  I don't
> think we have test coverage of it though so I wouldn't trust it.
>
> But a couple of these uses are just way past due to be fixed!  We're
> still relying on big serialized vectors (not just of bool; ErrorVector
> is a straight-up vector<float> under the hood!) in our error
> estimation and our mesh refinement flagging code.
>
>
> > A separate question: in looking at parallel_implementation.h, I note
> > several references to Communicator::get() (e.g., line 298).  Can you
> > tell me where this is defined?  I could not find it anywhere.  For
> > example, the command grep Communicator::get -r * from the main
> > directory did not yield the answer.
>
> And it probably doesn't help that our online Doxygen isn't generated
> from master/HEAD!
>
>
> http://libmesh.github.io/doxygen/classlibMesh_1_1Parallel_1_1Communicator.html#a4b6bd36c38eb7de350fcd56510306ced
>
> is still useful, though, even though refactoring has moved that method
> to include/parallel/communicator.h, it's still the same shim accessor
> method.  We often leave any definition that short in the class
> declaration itself, where it doesn't need the ClassName:: prefix.
> ---
> Roy
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Libmesh-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-devel
>

Re: [Libmesh-devel] libMesh/DistributedMesh Work

[Roy S. <roy...@ic...>]

I hope you don't mind, but I'm going to Cc: this to libmesh-devel.
These are good questions, and my opinions do not match those of all
the other developers and it might be unfair not to let them chime in.

On Wed, 2 May 2018, Tuzun, Robert wrote:

> Greetings.  I'm still in the middle of my hiatus, but there was one
> question I've been meaning to ask.  I notice, in several places, the
> use of vector<bool> in the code.  Way back in 2001, Scott Meyers
> (Effective STL, Item 18) recommended against the use of vector<bool>
> and recommended using deque<bool> or bitset instead.

"As an STL container, there are really only two things wrong with
vector<bool>.  First, it's not an STL container.  Second, it doesn't
hold bools.  Other than that, there's not much to object to." - Scott
Meyers.  I seriously love his writing.  It takes something special to
make programming language minutia so entertaining.

This is indeed often annoying, because the inability to take real
references to vector entries or to get a pointer to a underlying array
means we often can't handle vector<bool> with the same generic code as
we use to handle vector<literally_anything_else>.  I assume you first
encountered vector<bool> in our parallel_implementation.h overloads,
written to handle just that problem?

However, the recommended alternatives are even worse in many cases.
If a code is always fine using deque<bool>, if using EIGHT TIMES as
much memory never fazes the author, then Meyers' hundreds of C++
recommendations should be ignored in favor of "Item 1: Use Java
Instead" (or maybe Python or Go these days?).  Conversely, if a code
is always happy with bitset, and never needs an array that can be
resized, then what are you writing, autotranslated Fortran 77?

Personally, I'm mostly with Howard Hinnant:
https://isocpp.org/blog/2012/11/on-vectorbool

What's abominable about vector<bool> is the name, because that's
basically a lie about supporting the whole std::vector interface, but
the data structure itself is worthwhile and often useful.  And I'm
even sympathetic to the name!  Once you create a few well-chosen
overloads (like the parallel_implementation.h stuff) then the rest of
your generic code can *mostly* support vector<bool> as if it were a
real std::vector.  And when you miss a needed specialization, the
resulting breakage is a compile-time failure, so it's a hassle rather
than a disaster.


> I don't know if this advice still holds, and my search on the Web
> yields conflicting answers (just do a Google search on vector<bool>
> and you'll see what I mean).  Is there reason for concern?

Well, despite everything I said above: yes, there is.


First, to Meyers' objections I'd also have added "Third, it requires
bit-fiddling for every access", which means if you have a vector small
enough to *not* worry about memory savings, maybe you ought to be
using vector<unsigned char> instead.  Grepping through the code for
short logically-boolean vectors, I see that we're doing the
unsigned-char-really-holding-bool trick in dof_map.C, and
mesh_communication_global_indices.C... but we're using small
vector<bool> in about a dozen places?

I'm not going to change those without benchmarking first, because
"unsigned char is faster due to less bit fiddling" vs "bool is faster
due to fitting more stuff in cache" is the sort of system-dependent
question I can't answer a priori, and in the absence of data I think
vector<bool> is more self-documenting for something that's logically
boolean.  If anyone wants to try to prove me wrong in the libMesh
context, I'd say try System::boundary_project_solution for the
benchmark.  That has a few vector<bool> inside loops over elements,
so it looks like the easiest place to expose performance differences.


Second and more seriously, in libMesh whenever find ourselves in a
situation where tactically vector<bool> is definitely the right
choice, it may mean that strategically we've made a huge mistake!

Our classic use case for vector<bool> are vectors indexed over nodes
or over elements, where we need to support millions or billions of
entries and so using 8 times as much space is clearly a bad idea.

*However*, some of that code was a reasonable design only because it
dates back to ReplicatedMesh-only days, where we couldn't get away
from having data stored for every element on every processor.  These
days, a DistributedMesh user should be able to avoid ever requiring
O(N_elements) memory on a single MPI rank, which means that
applications should never allocate such a long vector!  So the library
should never be using one, at least on a non-debug-mode
non-ReplicatedMesh-only code path.

So let's see...

Some of these uses are OK.  E.g. the MeshTools assertion code is all
debug-mode-only, where readability is much more important than
efficiency.

Some are more dubious.  E.g.  UnstructuredMesh::all_first_order() only
gets used in practice for writing serialized viz output of an error
estimate, so making it perfectly scalable would require improving
multiple other parts of that code path first, meaning it's suboptimal
but not high priority.

Some of these uses are probably outweighed by other distributed mesh
incompatibility issues.  At first glance I'd guess
MeshTools::Modification::Smooth may not even work on a distributed
mesh, but we've just never tried to use it that way.  Looks like
LaplaceMeshSmoother is distributed-mesh-aware, so maybe everyone who
needs smoothing post-distribution has just been using that?  I don't
think we have test coverage of it though so I wouldn't trust it.

But a couple of these uses are just way past due to be fixed!  We're
still relying on big serialized vectors (not just of bool; ErrorVector
is a straight-up vector<float> under the hood!) in our error
estimation and our mesh refinement flagging code.


> A separate question: in looking at parallel_implementation.h, I note
> several references to Communicator::get() (e.g., line 298).  Can you
> tell me where this is defined?  I could not find it anywhere.  For
> example, the command grep Communicator::get -r * from the main
> directory did not yield the answer.

And it probably doesn't help that our online Doxygen isn't generated
from master/HEAD!

http://libmesh.github.io/doxygen/classlibMesh_1_1Parallel_1_1Communicator.html#a4b6bd36c38eb7de350fcd56510306ced

is still useful, though, even though refactoring has moved that method
to include/parallel/communicator.h, it's still the same shim accessor
method.  We often leave any definition that short in the class
declaration itself, where it doesn't need the ClassName:: prefix.
---
Roy

[Libmesh-devel] libMesh 1.3.0-rc1 released

[John P. <jwp...@gm...>]

Hello all,

There have been quite a few changes since the 1.2.x series in Summer 2017,
the biggest of which is probably that libMesh now requires a
C++11-conforming compiler.

For a more detailed list of changes in this release, see:
https://github.com/libMesh/libmesh/blob/v1.3.0-rc1/NEWS

For links to the packaged tarballs, visit:
https://github.com/libMesh/libmesh/releases/tag/v1.3.0-rc1

As always, we appreciate any feedback you can provide on your experiences
with downloading and building this release.

Thanks,
John

Re: [Libmesh-devel] [Libmesh-users] Is LIBS treated correctly?

[John P. <jwp...@gm...>]

On Tue, Feb 13, 2018 at 2:51 PM, Victor Eijkhout <eij...@ta...>
wrote:

>
>
> On Feb 13, 2018, at 2:12 PM, John Peterson <jwp...@gm...> wrote:
>
> I'd say you are missing a -lpython (and possibly other libraries) in your
> libmesh_optional_LIBS,
>
>
> Don’t I know it. I’ve been trying to figure out how to tell libmesh to add
> it.
>

You use the libmesh_LDFLAGS environment variable to add *all* the libraries
you need,

libmesh_LDFLAGS="-L/home1/apps/intel17/impi17_0/trilinos/12.10.1//lib
-lpytrilinos -lpython ..."

I don't know what the "..." should be. It depends on pytrilinos.

-- 
John

Re: [Libmesh-devel] [Libmesh-users] Is LIBS treated correctly?

[Victor E. <eij...@ta...>]

On Feb 13, 2018, at 2:12 PM, John Peterson <jwp...@gm...<mailto:jwp...@gm...>> wrote:

I'd say you are missing a -lpython (and possibly other libraries) in your libmesh_optional_LIBS,

Don’t I know it. I’ve been trying to figure out how to tell libmesh to add it.

That’s the first time you mention that variable to me, iirc. Is that an environment variable or a configure variable? If the latter, please dictate how to specify.

Victor.

Re: [Libmesh-devel] [Libmesh-users] Is LIBS treated correctly?

[John P. <jwp...@gm...>]

On Tue, Feb 13, 2018 at 10:14 AM, Victor Eijkhout <eij...@ta...>
wrote:

>
>
> On Feb 13, 2018, at 9:38 AM, John Peterson <jwp...@gm...> wrote:
>
> I think you may be using an older version of libmesh
>
>
> Could be. I’ve just moved up to 1.2.1. That does take away all the
> compiler options errors.
>
> Can you dictate me the configure flags to use my metis/parmetis rather
> than you compiling your own?
>

We currently only support the "--with-metis=PETSc" option to use the
Metis/ParMetis which was downloaded and installed during the PETSc
installation.



Configure & make logs for 1.2.1 attached.
>

I'd say you are missing a -lpython (and possibly other libraries) in your
libmesh_optional_LIBS, but this is just a guess based on Googling the first
undefined reference (PyType_IsSubtype).

The right way to link against Python might be by calling something like
"python-config --libs" on your system,  but since you are using
"pytrilinos" an even better approach might be to find a Trilinos Makefile
or configuration file that gives you the right list of libraries to link
against.

-- 
John

Re: [Libmesh-devel] [Libmesh-users] Is LIBS treated correctly?

[Victor E. <eij...@ta...>]

On Feb 13, 2018, at 9:38 AM, John Peterson <jwp...@gm...<mailto:jwp...@gm...>> wrote:

I think you may be using an older version of libmesh

Could be. I’ve just moved up to 1.2.1. That does take away all the compiler options errors.

Can you dictate me the configure flags to use my metis/parmetis rather than you compiling your own?

Configure & make logs for 1.2.1 attached.

Victor.