|
From: Christopher L. <cl...@an...> - 2007-02-17 08:11:56
|
Pretty basic one here it seems. I'm running a daemon process through callgrind. 2 initial dumpfiles are created which are most likely parent dumps of some sort before it fork()s to child/daemon land. I don't particularly care about those dumpfiles. What happens afterwards is that no dumpfile is created (0 byte etc) until I either force a dump through callgrind_control or terminate the process. Now, the non-creation of the child dumpfile is just a small problem, I can workaround that, it's not that big a deal. In one of my attempts to do a suitable workaround, I simply decided to run the process through callgrind with it's daemonizing mode turned off. This creates the initial 0 byte dumpfile - but here's the catch: the daemon suid's to a lower privledged user as part of it's daemonization logic but unfortunately it looks like CLG_init_files() is being called before that happens. So the result is that a 0 byte dumpfile is created with ownership of root, but when termination time comes, callgrind has assumed the uid of the lower-privledged user - and hence cannot write to the file. I am using --base, but by looking at the source it doesn't look like this would affect the logic of things - just seems that a possible solution would be to setuid back to the original process creator in order to write the dumpfile out. I could just be totally off and my logic is wrong. Also, I have no idea why the dump file is essentially not there when I CG the process as a daemon, unless there's some kind of open/unlink stuff going on. But based on the fact that it creates the initial 0-byte dump when I run the process non-daemon, it just seems strange that it wouldn't when it is in daemon mode. However, the overall issue seems to be the part of not jumping back to the original uid before writing out the dump. -cl |
|
From: Josef W. <Jos...@gm...> - 2007-02-18 02:30:54
|
On Saturday 17 February 2007, Christopher Layne wrote: > Pretty basic one here it seems. I'm running a daemon process through > callgrind. 2 initial dumpfiles are created which are most likely parent dumps > of some sort before it fork()s to child/daemon land. I don't particularly > care about those dumpfiles. What happens afterwards is that no dumpfile is > created (0 byte etc) I am not sure that initial creation of an empty dump file is the right thing to do. It currently has two somewhat dubious uses: (1) Check for write permissions for the dump file. If the dump file can not be created at startup, there is not much use of the profile run, as the results (most probably) will not be able to be written. Therefore, callgrind stops with an error when permissions are not enough. (2) The initial creation of a dumpfile allows to immediately start KCachegrind on this empty file. The current behavior is to load all dump "parts" on a "reload" action inside of KCachegrind (e.g. with callgrind_control -d), and allow for the "force dump" action in KCachegrind to work (which exactly triggers a new dump and reloads it). You found one problem with (1). It is not sure that after a initial write check, the file will always be writable at program termination. We also could unlink the file again directly after the permission check. Usage (2) is nice, but kind of a strange work flow, mixing command line with GUI usage. You probably either run callgrind on the command line and use callgrind_control if needed, or you want to use a GUI for the full online profiling, ie. it would be good to allow KCachegrind to start callgrind itself, which currently is not possible. When I get around to add some further control to KCachgrind to start callgrind itself, the usage (2) goes away. Neverless. The fact that callgrind is not creating an empty dumpfile in the child after a fork (using the new PID) can be seen as a bug. This bug is not visible when directly afterwards, an exec() is done, which will restart a new callgrind process, creating that empty file. But the current shortcoming really is that callgrind does not get notified when a fork happens. The "not-creation" of the emtpy dumpfile in the child is only one effect. More important, the child inherits all the profiling costs of the parent, so that these costs later will show up both in the child and the parent. This for sure is a (known) bug :-( > until I either force a dump through callgrind_control > or terminate the process. Now, the non-creation of the child dumpfile is > just a small problem, I can workaround that, it's not that big a deal. In > one of my attempts to do a suitable workaround, I simply decided to run > the process through callgrind with it's daemonizing mode turned off. This > creates the initial 0 byte dumpfile - but here's the catch: the daemon > suid's to a lower privledged user as part of it's daemonization logic but > unfortunately it looks like CLG_init_files() is being called before that > happens. So the result is that a 0 byte dumpfile is created with ownership > of root, but when termination time comes, callgrind has assumed the uid of > the lower-privledged user - and hence cannot write to the file. Your logic is probably right. The best thing for you would be to disallow callgrind to create an emtpy dumpfile at all, wouldn't it? I can introduce a command line option for this. > I am using > --base, but by looking at the source it doesn't look like this would affect > the logic of things - just seems that a possible solution would be to setuid > back to the original process creator in order to write the dumpfile out. > > I could just be totally off and my logic is wrong. Also, I have no idea why > the dump file is essentially not there when I CG the process as a daemon, > unless there's some kind of open/unlink stuff going on. But based on the fact > that it creates the initial 0-byte dump when I run the process non-daemon, > it just seems strange that it wouldn't when it is in daemon mode. See above for a reasonable explanation :-) > However, > the overall issue seems to be the part of not jumping back to the original > uid before writing out the dump. I do not think that this would be the correct solution. Josef |
|
From: Christopher L. <cl...@an...> - 2007-02-18 10:18:19
|
On Sun, Feb 18, 2007 at 03:26:12AM +0100, Josef Weidendorfer wrote:
> I am not sure that initial creation of an empty dump file is the right thing
> to do. It currently has two somewhat dubious uses:
> (1) Check for write permissions for the dump file. If the dump file can
> not be created at startup, there is not much use of the profile run, as the
> results (most probably) will not be able to be written. Therefore, callgrind
> stops with an error when permissions are not enough.
This seems like sane logic to me. The user callgrinding something probably
should have the same if not higher privledges than that they are callgrinding,
so I don't think it's an issue. Except in these kind of corner cases. :(
> (2) The initial creation of a dumpfile allows to immediately start KCachegrind
> on this empty file. The current behavior is to load all dump "parts" on a
> "reload" action inside of KCachegrind (e.g. with callgrind_control -d), and
> allow for the "force dump" action in KCachegrind to work (which exactly
> triggers a new dump and reloads it).
>
> You found one problem with (1). It is not sure that after a initial write check,
> the file will always be writable at program termination. We also could unlink
> the file again directly after the permission check.
>
> Usage (2) is nice, but kind of a strange work flow, mixing command line with
> GUI usage. You probably either run callgrind on the command line and use
> callgrind_control if needed, or you want to use a GUI for the full online
> profiling, ie. it would be good to allow KCachegrind to start callgrind itself,
> which currently is not possible. When I get around to add some further control
> to KCachgrind to start callgrind itself, the usage (2) goes away.
#2 is what I'm trying to achieve. Get callgrind initally running via cmdline,
and then periodically check things with the visualizer.
I typically run callgrind via a script like so:
#!/bin/bash
valgrind \
-v \
--num-callers=50 \
--trace-children=yes \
--tool=callgrind \
--simulate-cache=yes \
--collect-jumps=yes \
--dump-instr=yes \
--dump-line=yes \
--base=$1 \
$*
I then can use callgrind_control -d to create a dump. In the case of daemon
mode, this create it as the daemonized user. Just remembered a couple of bugs
here btw:
1. When i do callgrind_control -d, it keeps dumping continuously. It won't
stop dumping until I 'cp /devnull callgrind.cmd'. The actual ownership of
callgrind.cmd is:
-rw-r--r-- 1 root root 4 2007-02-18 02:02 callgrind.cmd
I think the issue here is that it cannot clear that file after executing a
dump, since it's running under daemon perms.
2. callgrind.cmd should be ${base}.cmd.
Sure enough, I just verified bug 1. I did a callgrind_control -d, it kept
on dumping continously. I then did a chown user.user callgrind.cmd and it
stopped.
Bug 2 seems an easy fix.
> Neverless.
> The fact that callgrind is not creating an empty dumpfile in the child after a
> fork (using the new PID) can be seen as a bug.
> This bug is not visible when directly afterwards, an exec() is done, which will
> restart a new callgrind process, creating that empty file.
Yep. Is that secondary empty file also unlinked btw? Because I never see it
until I force a dump or terminate the process.
> But the current shortcoming really is that callgrind
> does not get notified when a fork happens. The "not-creation" of the emtpy
> dumpfile in the child is only one effect. More important, the child inherits
> all the profiling costs of the parent, so that these costs later will show
> up both in the child and the parent. This for sure is a (known) bug :-(
If the parent is just doing standard pre-daemon work, this shouldn't be that
big an issue right? Or is it something else I'm not seeing?
In reference to the other point, is it possible to do some kind of getuid()
vs geteuid() comparison to determine when a dumpfile should be created or
deferred creation? In this case, it's not necessarily the nature of a fork()
which is causing the issue. It's the fact that witch the fork, the uids are
demoted - causing all of the ensuing permission conflicts.
> Your logic is probably right.
> The best thing for you would be to disallow callgrind to create an emtpy
> dumpfile at all, wouldn't it?
> I can introduce a command line option for this.
Thinking about it more, how about just not creating an empty dumpfile at all
until either a dump is requested or callgrind is on it's way out?
Wouldn't this solve the uid issue? Ah, but then we are back to the catch-22
because one wouldn't be able to select the file within kcachegrind without
doing a manual dump first.
> > However,
> > the overall issue seems to be the part of not jumping back to the original
> > uid before writing out the dump.
>
> I do not think that this would be the correct solution.
>
> Josef
Yes, you're correct. The real goal is to be able to create the empty dumpfile
on the child and not the parent, really.
Valgrind does have the notion of --trace-children, so there must be some kind
of fork() detection present already. Or is it using other mechanisms?
-cl
|
|
From: Christopher L. <cl...@an...> - 2007-02-18 14:26:35
|
On Sun, Feb 18, 2007 at 01:47:35PM +0100, Josef Weidendorfer wrote:
> The best is to have a solution without any corner cases. At least a way to
> bypass every corner case. In this case to be able to choose different
> "dump backends". Not only files, but also a network connection.
Absolutely agreed.
> > I then can use callgrind_control -d to create a dump. In the case of daemon
> > mode, this create it as the daemonized user. Just remembered a couple of bugs
> > here btw:
> >
> > 1. When i do callgrind_control -d, it keeps dumping continuously. It won't
> > stop dumping until I 'cp /devnull callgrind.cmd'. The actual ownership of
> > callgrind.cmd is:
> > -rw-r--r-- 1 root root 4 2007-02-18 02:02 callgrind.cmd
> > I think the issue here is that it cannot clear that file after executing a
> > dump, since it's running under daemon perms.
> > 2. callgrind.cmd should be ${base}.cmd.
>
> OK. This is again a problem because of the usage of files for control of
> callgrind. A optional network connection would solve this.
Definitely agreed. Either a bound network connection or a domain socket would
suffice.
> > Sure enough, I just verified bug 1. I did a callgrind_control -d, it kept
> > on dumping continously. I then did a chown user.user callgrind.cmd and it
> > stopped.
>
> You do not need callgrind_control. You instead can create the callgrind.cmd
> file directly:
>
> echo d > callgrind.cmd
Alternatively for now I can run as non-daemon, and change the perms of the 0byte
dump to the same user as the process (since it setuids).
> > Bug 2 seems an easy fix.
>
> Interesting. Actually, "callgrind.cmd" is deprecated but still supported
> because of KCachegrind. It should be "callgrind.cmd.<PID>" now. Otherwise,
> you can not control multiple callgrind runs which are started from the
> same working directory.
>
> The real bug here is that KCachegrind is directly creating a "callgrind.cmd"
> file without using callgrind_control. The later would decouple the control
> logic, allowing to change it more easily.
This also goes back to socket control as the preferred route as well.
> > Yep. Is that secondary empty file also unlinked btw? Because I never see it
> > until I force a dump or terminate the process.
>
> When exec() is run and "--follow-children" is active, you get a new callgrind
> run, and there should be no difference to the standard way.
Shouldn't be, but for instance, running as non-daemon now:
-rw------- 1 root root 0 2007-02-18 06:11 tor.31893
(no problem, cept i'll have to change that owner before I dump, of course)
Run as daemon, I get 2 initial dumps like so:
-rw------- 1 root root 450776 2007-02-17 01:43 tor.31457
-rw------- 1 root root 447492 2007-02-17 01:43 tor.31452
(i ignore these as we mentioned. strange there are 2 though,
31452 contains no reference to fork() where 31457 does as expected)
And this last one is absolutely not visible until I terminate the process.
-rw------- 1 tor tor 1559955 2007-02-17 02:00 tor.31458
> > In reference to the other point, is it possible to do some kind of getuid()
> > vs geteuid() comparison to determine when a dumpfile should be created or
> > deferred creation? In this case, it's not necessarily the nature of a fork()
> > which is causing the issue. It's the fact that witch the fork, the uids are
> > demoted - causing all of the ensuing permission conflicts.
>
> This smells like a specific solution for a specific corner case without solving
> the 999 other corner cases.
> Why should a Valgrind tool be concerned about permissions?
>
> What about keeping the dump file open from the beginning? Demoting permissions
> doesn't change permission on open files.
Sorry, when I said permissions here I meant ownership. Keeping the dump file
open from the beginning would be fine - except for a type of case where vg/cg
doesn't know ahead of time that it's going to setuid.
> > Wouldn't this solve the uid issue? Ah, but then we are back to the catch-22
> > because one wouldn't be able to select the file within kcachegrind without
> > doing a manual dump first.
>
> Yup.
Back to the socket idea :)
> > Valgrind does have the notion of --trace-children, so there must be some kind
> > of fork() detection present already. Or is it using other mechanisms?
>
> The currently is no special hook in VGs tool interface for it. There was some
> talk about it lately. We will find a solution ;-)
>
> Josef
Yes, I'm not too concerned about that, that's for sure. I really do love callgrind,
Josef. I think it opens up amazing possibilities for analysis and I don't think the
VG/CG guys get enough spotlight because the tools are indespensable.
-cl
|