|
From: Philippe W. <phi...@sk...> - 2015-03-29 17:03:14
Attachments:
patch_decouple_main_main.txt
|
When linking a tool for a certain architecture (e.g. x86), the resulting
executable contains a significant proportion of the VEX library for
other architectures (amd64, arm, ppc, mips, s390).
After looking at the dependencies, the other architectures object
files are dragged in by various 'switch (arch)' in main_main.c.
For example,
case VexArchPPC64:
mode64 = True;
rRegUniv = getRRegUniverse_PPC(mode64);
isMove = (__typeof__(isMove)) isMove_PPCInstr;
getRegUsage = (__typeof__(getRegUsage)) getRegUsage_PPCInstr;
will drag various ppc64 objects.
I have quickly done a trial to see how much we can gain by having
main_main.c only dragging one architecture.
On x86, the text size of memcheck-x86-linux decreases
from about 3.8 MB to about 2MB.
The startup of a tool is also slightly faster : valgrind reads its own
debug info, and smaller executable means smaller debug info to read.
The mmap-ed size of the "dinfo" arena also decreases by about 9MB
(peak mmaped decreases by 14MB).
So, it looks interesting to avoid dragging all the other archs VEX
objects.
I see 3 ways to do the above:
1. Using a few conditional macros in main_main.c, ensure only
the functions needed for the compiled architecture
are referenced
This is easy to do.
However, this means the compile VEX library can only be used with
one single architecture : host and guest will be the same, and
will be the one for which the VEX lib is compîled.
I do not know if VEX lib is supposed to be usable to have
host and guest different.
If VEX host and guest are in any case supposed to be the same,
then solution 1 is the easiest.
2. Same as 1, but allow a configure option to still allow all
architectures to be compiled in.
A little bit more work than 1.
Advantage: if someone needs a multi-arch VEX lib, it can be
decided at configure time.
3. Decouple main_main.c from the 'backend VEX' by extending VexArchInfo
and/or adding a VexBackEndInfo structure, containing pointers to the
arch specific functions.
I have started doing this (half done patch attached, only works
for x86, as the decoupling infrastructure is incomplete).
All that is not very straightforward/is a lot more work.
And of course, the VEX user will have to call a 'arch dependent'
initialise procedure (one of the things not yet done in the patch).
Personally, 1 (or maybe 2) looks good enough to me,
but that assumes there is no (significant) need for a multi-arch VEX
lib.
Feedback/comments/suggestions ?
Philippe
|
|
From: Florian K. <fl...@ei...> - 2015-03-29 20:46:58
|
On 29.03.2015 18:03, Philippe Waroquiers wrote: > > So, it looks interesting to avoid dragging all the other archs VEX > objects. > Yes, please. +10 > I see 3 ways to do the above: > > 1. Using a few conditional macros in main_main.c, ensure only > the functions needed for the compiled architecture > are referenced > This is easy to do. > However, this means the compile VEX library can only be used with > one single architecture : host and guest will be the same, and > will be the one for which the VEX lib is compîled. > I do not know if VEX lib is supposed to be usable to have > host and guest different. Julian can give the full history. There is plenty of evidence that the original VEX design goal was to support host != guest. But when I began looking at V code in 2006 the code wasn't clean in that respect. And it hasn't gone any better. Here is a quote from Julian from Dec 2014 clarifying: This guest-vs-host stuff is still partly alive as a result of a hope I had that someone might want to do a cross-valgrind one day, eg ARM32 guest on AMD64 host. But it's been 12+ years and I've never once heard any mention of such a thing. So perhaps it's time to give up on that one. I think that settles your question :) > If VEX host and guest are in any case supposed to be the same, > then solution 1 is the easiest. Yes. It's funny.. I looked at this very same issue a few weeks back but could not figure out the autotools stuff. Note, though, that we still want to compile all VEX sources and not just the ones pertaining to the current architecture. Florian |
|
From: Philippe W. <phi...@sk...> - 2015-03-29 21:15:50
|
On Sun, 2015-03-29 at 22:46 +0200, Florian Krohm wrote: > Julian can give the full history. > There is plenty of evidence that the original VEX design goal was to > support host != guest. But when I began looking at V code in 2006 the > code wasn't clean in that respect. And it hasn't gone any better. > Here is a quote from Julian from Dec 2014 clarifying: > > This guest-vs-host stuff is still partly alive as a result of a > hope I had that someone might want to do a cross-valgrind one day, > eg ARM32 guest on AMD64 host. But it's been 12+ years and I've > never once heard any mention of such a thing. So perhaps it's > time to give up on that one. > > I think that settles your question :) Yes :). > > > If VEX host and guest are in any case supposed to be the same, > > then solution 1 is the easiest. > > Yes. > > It's funny.. I looked at this very same issue a few weeks back but could > not figure out the autotools stuff. > Note, though, that we still want to compile all VEX sources and not just > the ones pertaining to the current architecture. Yes, for sure, that is the idea. I am busy now doing the solution 1. I have already eliminated s390 :). Philippe |
|
From: Philippe W. <phi...@sk...> - 2015-03-30 19:14:51
Attachments:
patch_option1_link.txt
|
On Sun, 2015-03-29 at 23:20 +0200, Philippe Waroquiers wrote: > I just finished implementing option 1. Find attached the patch for option 1. It has been compiled/validated on x86/amd64/ppc32/ppc64/s390. Changes for arm/arm64/mips32/mips64 are done but uncompiled/untested (I am reasonably confident it will work, as the changes are very mechanical). Some more comments ? Any feedback about option 2.5 ? (i.e. have an additional .o 'multi_arch_main_main.o' always built) ? Philippe |
|
From: Yan <ya...@ya...> - 2015-03-29 21:28:27
|
Hi guys, We use VEX pretty heavily in research here at UC Santa Barbara (resulting in PyVEX [1] and some published [2] and in-submission academic papers), and for us, the ability to handle multiple architectures with the same libVEX is really crucial. As it is already, we have to patch VEX pretty heavily (see the patch in [3]) to get it to work statically, but the changes in option 1 sound like they'll make things even worse for us :-( Of course, no one's under any obligation to make our lives over here easier. In fact, I talked to Philippe and Julian at FOSDEM '14 about me sending in some patches to change up the VEX interface to be more flexible (basically, option 3) and Julian seemed receptive at the time. However, with PhD stuff, grant progress reports, and paper deadlines, I haven't had any time to actually do it. If it saves multi-arch VEX, though, we can make it a top priority over here and devote a few good guys to it to get it done... So, in summary, if option 3 could be made more attractive with some manpower, UCSB can provide that manpower, because we use VEX in a multi-arch, static way for our research. - Yan [1] http://github.com/zardus/pyvex [2] http://www.internetsociety.org/doc/firmalice-automatic-detection-authentication-bypass-vulnerabilities-binary-firmware [3] https://github.com/zardus/pyvex/blob/master/patches/valgrind_static_3.9.0.patch On Sun, Mar 29, 2015 at 1:16 PM, Philippe Waroquiers < phi...@sk...> wrote: > On Sun, 2015-03-29 at 22:46 +0200, Florian Krohm wrote: > > Julian can give the full history. > > There is plenty of evidence that the original VEX design goal was to > > support host != guest. But when I began looking at V code in 2006 the > > code wasn't clean in that respect. And it hasn't gone any better. > > Here is a quote from Julian from Dec 2014 clarifying: > > > > This guest-vs-host stuff is still partly alive as a result of a > > hope I had that someone might want to do a cross-valgrind one day, > > eg ARM32 guest on AMD64 host. But it's been 12+ years and I've > > never once heard any mention of such a thing. So perhaps it's > > time to give up on that one. > > > > I think that settles your question :) > Yes :). > > > > > If VEX host and guest are in any case supposed to be the same, > > > then solution 1 is the easiest. > > > > Yes. > > > > It's funny.. I looked at this very same issue a few weeks back but could > > not figure out the autotools stuff. > > Note, though, that we still want to compile all VEX sources and not just > > the ones pertaining to the current architecture. > Yes, for sure, that is the idea. > I am busy now doing the solution 1. > I have already eliminated s390 :). > > Philippe > > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, > sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for > all > things parallel software development, from weekly thought leadership blogs > to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |
|
From: Philippe W. <phi...@sk...> - 2015-03-29 22:19:34
|
On Sun, 2015-03-29 at 14:27 -0700, Yan wrote: > Hi guys, > > > We use VEX pretty heavily in research here at UC Santa Barbara > (resulting in PyVEX [1] and some published [2] and in-submission > academic papers), and for us, the ability to handle multiple > architectures with the same libVEX is really crucial. As it is > already, we have to patch VEX pretty heavily (see the patch in [3]) to > get it to work statically, but the changes in option 1 sound like > they'll make things even worse for us :-( > > > Of course, no one's under any obligation to make our lives over here > easier. In fact, I talked to Philippe and Julian at FOSDEM '14 about > me sending in some patches to change up the VEX interface to be more > flexible (basically, option 3) and Julian seemed receptive at the > time. However, with PhD stuff, grant progress reports, and paper > deadlines, I haven't had any time to actually do it. If it saves > multi-arch VEX, though, we can make it a top priority over here and > devote a few good guys to it to get it done... > > > So, in summary, if option 3 could be made more attractive with some > manpower, UCSB can provide that manpower, because we use VEX in a > multi-arch, static way for our research. Isn't option 2 good enough ? I just finished implementing option 1. In option 1, we have the following comment: /* For each architecture <arch>, we define 2 macros: <arch>FN that has as argument a pointer to a function. <arch>ST that has as argument a statement. If VEX is compiled for <arch>, then these macros just expand their arg. Otherwise, the macros expands to respectively NULL and vassert(0). These macros are used to avoid introducing dependencies to object files not needed for the (only) architecture we are compiling for. To still compile the below for all supported architectures, define VEXMULTIARCH. */ // #define VEXMULTIARCH 1 So, a very trivial patch is enough to have VEX compiled in multiarch with option 1. Of course, option 2 now only means adding a configure option. If needed, we could maybe have option 2.5: We compile main_main.c twice: One with VEXMULTIARCH not defined to produce an object main_main.o and one with VEXMULTIARCH defined to produce another object multi_arch_main_main.o Then, I suppose that when linking, if you put multi_arch_main_main.o before the VEX library .a, that the linker will choose multi_arch_main_main.o to resolve the needed LibVEX functions defined in main_main.c Philippe |
|
From: Yan <ya...@ya...> - 2015-03-30 20:01:32
|
Hey! Sorry about the laggy response; I was traveling yesterday :-( For UCSB's purposes, option 2 works ok! If I understand it correctly, it's basically exactly what happens currently, right? Option 2.5 is fine for us as well. - Yan On Mon, Mar 30, 2015 at 12:15 PM, Philippe Waroquiers < phi...@sk...> wrote: > On Sun, 2015-03-29 at 23:20 +0200, Philippe Waroquiers wrote: > > I just finished implementing option 1. > > Find attached the patch for option 1. > It has been compiled/validated on x86/amd64/ppc32/ppc64/s390. > Changes for arm/arm64/mips32/mips64 are done but uncompiled/untested > (I am reasonably confident it will work, as the changes are > very mechanical). > > Some more comments ? > > Any feedback about option 2.5 ? > (i.e. have an additional .o 'multi_arch_main_main.o' always built) ? > > Philippe > > |
|
From: Philippe W. <phi...@sk...> - 2015-03-30 21:13:46
|
On Mon, 2015-03-30 at 13:00 -0700, Yan wrote: > Hey! Sorry about the laggy response; I was traveling yesterday :-( > > > For UCSB's purposes, option 2 works ok! If I understand it correctly, > it's basically exactly what happens currently, right? Option 2.5 is > fine for us as well. Option 1 and option 2 are effectively mostly similar (just need to tweak configure to have option 2). When VEXMULTIARCH is defined, then main_main.c is compiled similarly to the current trunk. As you can see in the patch, the changes are trivial. Option 2.5 implies some changes to Makefile.am (not completely trivial changes, at least for someone like me that knows close to 0 about automake/conf magic) An advantage of option 2.5 is that main_main.c will always be compiled in both VEXMULTIARCH and single arch mode, to help catching an unlikely compilation bug in VEXMULTIARCH not detected otherwise. A user that needs multi arch can then have it just by giving 2 libs to link (in the good order) : libvexmultiarch-x86-linux.a libvex-x86-linux.a where the first lib only contains main_main.c compiled with VEXMULTIARCH defined. The 2nd lib contains main_main.c compiled in single arch, and all the other VEX object files, and is the one to use by default. Philippe |
|
From: Philippe W. <phi...@sk...> - 2015-03-31 23:24:33
|
On Mon, 2015-03-30 at 23:13 +0200, Philippe Waroquiers wrote: > An advantage of option 2.5 is that main_main.c will always be compiled > in both VEXMULTIARCH and single arch mode, to help catching an unlikely > compilation bug in VEXMULTIARCH not detected otherwise. > > A user that needs multi arch can then have it just > by giving 2 libs to link (in the good order) : > libvexmultiarch-x86-linux.a libvex-x86-linux.a > where the first lib only contains main_main.c compiled with VEXMULTIARCH > defined. The 2nd lib contains main_main.c compiled in single arch, > and all the other VEX object files, and is the one to use by default. > Option '2.5' committed as vex: r3113 So, the VEX user can choose at link time whether single arch or multi-arch VEX is linked in. Example: # this produces t1single (calling LibVEX_Translate), that is single arch: gcc -o t1single t1.o -LInst/lib/valgrind -lvex-x86-linux -lgcc # this produces t1multi, that is multi arch: gcc -o t1multi t1.o -LInst/lib/valgrind -lvexmultiarch-x86-linux -lvex-x86-linux -lgcc Philippe |
|
From: Julian S. <js...@ac...> - 2015-03-31 18:24:57
|
On 29/03/15 18:03, Philippe Waroquiers wrote: > When linking a tool for a certain architecture (e.g. x86), the resulting > executable contains a significant proportion of the VEX library for > other architectures (amd64, arm, ppc, mips, s390). There are two reasons why it is like it currently is: 1. To retain the possibility of people using Vex for binary translation, and being able to do this simply by passing in suitable parameters to LibVEX_Translate. That implies linking in all of the front ends and back ends. We still have some folks who need that ability, for example Yan (for binary analysis) and recent post "Guest ARM, Host X86". 2. To avoid the code falling into a mass of ifdefs, and so having the possibility of build failures on some targets but not others. The simplest way to avoid that was simply to compile and link all files on all targets. I can see that it might be useful to reduce the binary sizes. But I'd still want a way to be able to retain both (1) and (2). I would prefer the proposed configure time scheme, so that (eg) maybe by default, only the front- and back-end object for the primary and secondary arch get linked in. But if --all-vex-archs is given then it will be built as at present, so that requirement (2) above is still easily satisfied on developer machines and nightly builder.s. J |