Re: [Valgrind-users] Fwd: "__malloc_initialize_hook is deprecated"&&co. warnings while building ast-open.2013-04-22 ...

[Roland M. <rol...@nr...>]

On Thu, May 16, 2013 at 9:13 PM, Philippe Waroquiers
<phi...@sk...> wrote:
> On Tue, 2013-05-14 at 04:28 +0200, Roland Mainz wrote:
>> On Thu, Apr 25, 2013 at 1:42 PM, Sebastian Feld
>> <seb...@gm...> wrote:
>> > On Wed, Apr 24, 2013 at 11:10 PM, Roland Mainz <rol...@nr...> wrote:
>> >> On Wed, Apr 24, 2013 at 10:14 PM, Roland Mainz <rol...@nr...> wrote:
>> >>> On Wed, Apr 24, 2013 at 12:45 AM, John Reiser <jr...@bi...> wrote:
>
>> >> $ valgrind  "--allocator-sym-redirect=sh_malloc=malloc,sh_free=free,sh_calloc=calloc"
>> >> ... # would instruct valgrind to take function |sh_malloc()| as an
>> >> alternative |malloc(), |sh_free()| as alternative |free()| version
>> >> etc. etc.
>> >>
>> >> The only issue is that if multiple allocators are active within a
>> >> single process we may need some kind of "grouping" to explain valgrind
>> >> that memory allocated by |sh_malloc()| can not be freed by |tcfree()|
>> >> or |_ast_free()| ... maybe it could be done using '{'- and '}'-pairs,
>> >> e.g. $ valgrind
>> >> "--allocator-sym-redirect={sh_malloc=malloc,sh_free=free,sh_calloc=calloc},{_ast_malloc=malloc,_ast_free=free,_ast_calloc=calloc}"
>> >> ... #
>> >
>> > The idea of (finally!) providing such an option sounds like a very
>> > good idea. Until now the only way to probe python and bash4 via
>> > valgrind is to poke in the valgrind sources (which should never
>> > happen).
> I think it would not be very difficult to extend the command line option
>     --soname-synonyms=syn1=pattern1,syn2=pattern2,... synonym soname
>               specify patterns for function wrapping or replacement.
>               To use a non-libc malloc library that is
>                   in the main exe:  --soname-synonyms=somalloc=NONE
>                   in libxyzzy.so:   --soname-synonyms=somalloc=libxyzzy.so
> to support also to give synonym for the "function" part of a redirection.
> Now that I understand better all this area, it should be relatively
> easy to allow to give synonyms for any (existing) redirection
> (library part or function part).
> In other words, to make -soname-synonyms "generic".

That would be helpfull (in our case it's the AST toolkit which has
it's own utility library called libast.so.1 and to avoid namespace
collisions on old operating systems many functions (including their
|malloc()| version) are prefixed with |_ast|, e.g. |_ast_malloc()|,
|_ast_free()| etc. on demand.

>> > I also think the idea to let valgrind detect mixing of different
>> > allocators is a very valuable feature since this has been a source of
>> > more and more bugs. Usually happens in complex projects with use many
>> > different shared libraries, all with their own memory allocators.
>
> However, the impact of this part is not as easy.

I know... I studied the valgrind code in the meantime and started to
gnaw my fingernails off... ;-/

> This implies to change the basic way the malloc interception is done,
> by adding an additional "grouping" parameter, and store this in each
> memory "chunk" managed by memcheck.
> More impact on memory, and on the interface between the core and
> the tools replacing the malloc, and a lot more difficult to make
> "generic".
> I suspect this will imply also some possibly heavy changes to
> the "core" redirection logic.

Right... but some day it needs to be done... ;-/
... for AFAIK (at least) two reasons:
1. Big applications depending on many utility libraries sooner or
later hit the issue of having multiple |malloc()| versions. For
passing a pointer obtained from |_chicken_malloc()| to
|_pterodactylus_free()| is likely causing some very interesting
trouble if the process doesn't get a SIGSEGV/SIGBUS walloped over the
head on the first try... ;-/
2. Some memory allocators (for example used by commercial C++
libraries... or take a look at Sun/Oracle is doing for the VM2 project
in Solaris 11.1/12) allow "pools" of memory (e.g. you create a pool
and then use the pool as argument to a |malloc()|-like function) where
from each pool memory can be allocated independently (this is usually
done to boost performance (for example each thread owns it's own pool
and doesn't have to use atomic operations unless new "backing store"
needs to be obtained), fight fragmentation, make
dealloc_everything_from_this_pool-in-one-go easier/faster etc.).
Beyond the usual trouble some implementations allow nesting of pools,
e.g. one pool provides the "backing store" for a couple of sub-pools
etc. (for example there is a "global" pool which is thread-safe and
the threads maintain their own pools (without the need for atomic
operations) and use the global pool to obtain the memory)

>> Uhm... was there any feedback yet for that idea ?
>
> Some feedback above :).

Thanks... :-)

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) rol...@nr...
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)