You can subscribe to this list here.
| 2003 |
Jan
|
Feb
|
Mar
(58) |
Apr
(261) |
May
(169) |
Jun
(214) |
Jul
(201) |
Aug
(219) |
Sep
(198) |
Oct
(203) |
Nov
(241) |
Dec
(94) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(137) |
Feb
(149) |
Mar
(150) |
Apr
(193) |
May
(95) |
Jun
(173) |
Jul
(137) |
Aug
(236) |
Sep
(157) |
Oct
(150) |
Nov
(136) |
Dec
(90) |
| 2005 |
Jan
(139) |
Feb
(130) |
Mar
(274) |
Apr
(138) |
May
(184) |
Jun
(152) |
Jul
(261) |
Aug
(409) |
Sep
(239) |
Oct
(241) |
Nov
(260) |
Dec
(137) |
| 2006 |
Jan
(191) |
Feb
(142) |
Mar
(169) |
Apr
(75) |
May
(141) |
Jun
(169) |
Jul
(131) |
Aug
(141) |
Sep
(192) |
Oct
(176) |
Nov
(142) |
Dec
(95) |
| 2007 |
Jan
(98) |
Feb
(120) |
Mar
(93) |
Apr
(96) |
May
(95) |
Jun
(65) |
Jul
(62) |
Aug
(56) |
Sep
(53) |
Oct
(95) |
Nov
(106) |
Dec
(87) |
| 2008 |
Jan
(58) |
Feb
(149) |
Mar
(175) |
Apr
(110) |
May
(106) |
Jun
(72) |
Jul
(55) |
Aug
(89) |
Sep
(26) |
Oct
(96) |
Nov
(83) |
Dec
(93) |
| 2009 |
Jan
(97) |
Feb
(106) |
Mar
(74) |
Apr
(64) |
May
(115) |
Jun
(83) |
Jul
(137) |
Aug
(103) |
Sep
(56) |
Oct
(59) |
Nov
(61) |
Dec
(37) |
| 2010 |
Jan
(94) |
Feb
(71) |
Mar
(53) |
Apr
(105) |
May
(79) |
Jun
(111) |
Jul
(110) |
Aug
(81) |
Sep
(50) |
Oct
(82) |
Nov
(49) |
Dec
(21) |
| 2011 |
Jan
(87) |
Feb
(105) |
Mar
(108) |
Apr
(99) |
May
(91) |
Jun
(94) |
Jul
(114) |
Aug
(77) |
Sep
(58) |
Oct
(58) |
Nov
(131) |
Dec
(62) |
| 2012 |
Jan
(76) |
Feb
(93) |
Mar
(68) |
Apr
(95) |
May
(62) |
Jun
(109) |
Jul
(90) |
Aug
(87) |
Sep
(49) |
Oct
(54) |
Nov
(66) |
Dec
(84) |
| 2013 |
Jan
(67) |
Feb
(52) |
Mar
(93) |
Apr
(65) |
May
(33) |
Jun
(34) |
Jul
(52) |
Aug
(42) |
Sep
(52) |
Oct
(48) |
Nov
(66) |
Dec
(14) |
| 2014 |
Jan
(66) |
Feb
(51) |
Mar
(34) |
Apr
(47) |
May
(58) |
Jun
(27) |
Jul
(52) |
Aug
(41) |
Sep
(78) |
Oct
(30) |
Nov
(28) |
Dec
(26) |
| 2015 |
Jan
(41) |
Feb
(42) |
Mar
(20) |
Apr
(73) |
May
(31) |
Jun
(48) |
Jul
(23) |
Aug
(55) |
Sep
(36) |
Oct
(47) |
Nov
(48) |
Dec
(41) |
| 2016 |
Jan
(32) |
Feb
(34) |
Mar
(33) |
Apr
(22) |
May
(14) |
Jun
(31) |
Jul
(29) |
Aug
(41) |
Sep
(17) |
Oct
(27) |
Nov
(38) |
Dec
(28) |
| 2017 |
Jan
(28) |
Feb
(30) |
Mar
(16) |
Apr
(9) |
May
(27) |
Jun
(57) |
Jul
(28) |
Aug
(43) |
Sep
(31) |
Oct
(20) |
Nov
(24) |
Dec
(18) |
| 2018 |
Jan
(34) |
Feb
(50) |
Mar
(18) |
Apr
(26) |
May
(13) |
Jun
(31) |
Jul
(13) |
Aug
(11) |
Sep
(15) |
Oct
(12) |
Nov
(18) |
Dec
(13) |
| 2019 |
Jan
(12) |
Feb
(29) |
Mar
(51) |
Apr
(22) |
May
(13) |
Jun
(20) |
Jul
(13) |
Aug
(12) |
Sep
(21) |
Oct
(6) |
Nov
(9) |
Dec
(5) |
| 2020 |
Jan
(13) |
Feb
(5) |
Mar
(25) |
Apr
(4) |
May
(40) |
Jun
(27) |
Jul
(5) |
Aug
(17) |
Sep
(21) |
Oct
(1) |
Nov
(5) |
Dec
(15) |
| 2021 |
Jan
(28) |
Feb
(6) |
Mar
(11) |
Apr
(5) |
May
(7) |
Jun
(8) |
Jul
(5) |
Aug
(5) |
Sep
(11) |
Oct
(9) |
Nov
(10) |
Dec
(12) |
| 2022 |
Jan
(7) |
Feb
(13) |
Mar
(8) |
Apr
(7) |
May
(12) |
Jun
(27) |
Jul
(14) |
Aug
(27) |
Sep
(27) |
Oct
(17) |
Nov
(17) |
Dec
|
| 2023 |
Jan
(10) |
Feb
(18) |
Mar
(9) |
Apr
(26) |
May
|
Jun
(13) |
Jul
(18) |
Aug
(5) |
Sep
|
Oct
|
Nov
|
Dec
|
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
|
|
|
|
|
|
1
(11) |
2
|
|
3
(5) |
4
(8) |
5
(33) |
6
(11) |
7
(1) |
8
(3) |
9
(4) |
|
10
(3) |
11
(12) |
12
(17) |
13
(10) |
14
(13) |
15
(7) |
16
|
|
17
(2) |
18
(22) |
19
(9) |
20
(7) |
21
(1) |
22
(4) |
23
(1) |
|
24
|
25
(1) |
26
(3) |
27
(8) |
28
(3) |
29
(16) |
30
(4) |
|
31
|
|
|
|
|
|
|
|
From: Nicholas N. <nj...@ca...> - 2003-08-19 07:45:39
|
On Mon, 18 Aug 2003, Jason Gauthier wrote:
> Is it possible to actually link against a valgrind library, or can one only
> run it through the executable?
Let's see:
[~/grind/annelid] gcc a.o memcheck/vgskin_memcheck.so coregrind/valgrind.so
[~/grind/annelid] VG_ARGS=--suppressions=default.supp; a.out
==26034== Memcheck, a.k.a. Valgrind, a memory error detector for x86-linux.
==26034== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward.
==26034== Using valgrind-20030725, a program supervision framework for x86-linux.
==26034== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward.
==26034== Estimated CPU clock rate is 1404 MHz
==26034== For more details, rerun with: -v
==26034==
==26034==
==26034== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==26034== malloc/free: in use at exit: 0 bytes in 0 blocks.
==26034== malloc/free: 0 allocs, 0 frees, 0 bytes allocated.
==26034== For a detailed leak analysis, rerun with: --leak-check=yes
==26034== For counts of detected errors, rerun with: -v
So the answer seems to be yes, with some provisos:
- You have to link with a particular skin. I linked with Memcheck
above.
- You have to put the skin .so before valgrind.so when linking.
- You have to setup $VG_ARGS, by adding a --suppressions= entry, at the
least.
I'd be interested to know why you want to do this.
N
|
|
From: Nicholas N. <nj...@ca...> - 2003-08-19 07:38:42
|
On Mon, 18 Aug 2003, Steve G wrote: > I think this thread has strayed way beyond Jeremy's > original post. He was asking if it was worthwhile to have > something like a Hellegrind race detector. Hear hear! > Should something be done to help locate problems? In one of > the first e-mails of this thread, I wrote an algorithm that > may help locate those problems. Does it work? Can anyone > see a better way to do it? As Jeremy said, it sounds like it would be difficult to do. His Helgrind/Eraser-like algorithm sounds more suitable. However, adding it would make Crocus much more complicated, and I have enough things on my plate as it is. Someone else is very welcome to have a go at adding the functionality. > Am I the only one that believes a race detector would be helpful? I think you are much more aware of these things than most programmers, due to your experience with daemons, etc. N |
|
From: Tom H. <th...@cy...> - 2003-08-19 06:19:41
|
In message <3F4...@re...>
Steve Fink <sf...@re...> wrote:
> I was mostly wondering if, now that valgrind supports MMX and some SSE,
> it is now reporting that it is an MMX/SSE cpu? If so, is there some way
> of making it act stupid again? (I don't know whether my problem is with
> MMX or SSE instructions, actually.)
I sent a patch to this list several weeks ago to stop valgrind
reporting the CPU as SSE capable for people with your problem.
There shouldn't be any problem with MMX as valgrind's MMX support
is supposed to be complete.
Tom
--
Tom Hughes (th...@cy...)
Software Engineer, Cyberscience Corporation
http://www.cyberscience.com/
|
|
From: Jeremy F. <je...@go...> - 2003-08-19 06:13:58
|
On Mon, 2003-08-18 at 06:47, Steve G wrote:
> I suppose an approach might be something like this: Each
> time a signal is about to be delivered, check the mask. If
> any unblocked signal (besides the one about to be
> delivered) has a user defined signal handler, fake
> delivering a signal to it and only note the addresses it
> wanted to read data from or write data to and what the
> access type is (read or write). Don't let it actually
> write, reading is ok since its non-destructive. Then after
> faking a signal to all user defined handlers, check the
> list each time the current signal handler accesses a memory
> location. Only read/write or write/write collisions matter.
> The list gets thrown away at the end of the signal handler.
So you're saying speculatively run the signal handler without side
effects? I think that's pretty tricky. For one, it may have a read
which depends on one of its earlier writes. Even more tricky, it may
write to memory with a syscall, which would be hard to make
side-effect-free.
I think a helgrind-like algorithm would be more appropriate:
For each memory location, maintain a state. There are 4 states:
1. untouched
2. private - only touched by mainline code or a handler for a
particular signal, but not both
3. read-shared - only ever read by mainline and signal handlers
4. RW-shared - both read and written by mainline and signal
handlers
In RW-shared state, it keeps a set of signal handlers which touch the
memory location. If any code touches a location in RW-shared state, it
must have that set of signals blocked, otherwise it reports an error.
The tricky part is trying to determine if a given instruction is being
run as part of a signal handler or not. Entering the signal handler is
pretty obvious, since Valgrind has to go to some effort to deliver a
signal to a thread. Detecting when the application leaves the handler
is more tricky. If the handler does a normal return it's easy enough,
but if it longjmps out of the handler it's harder to tell. I think the
sure-fire way which works in both cases is to detect when ESP goes above
the signal stack frame - this will happen in both the longjmp and
sigreturn cases.
The other potentially tricky thing to handle is recursive signals. I
need to think about it more - it may just fall out in the algorithm
above. I think if you consider the mainline code to also be a
particular form of signal handler (SIGMAIN) then it comes out naturally.
J
|
|
From: Steve F. <sf...@re...> - 2003-08-19 02:34:57
|
I used to be able to valgrind my app, but these days I always run into unimplemented instructions. I have changed several pieces of the puzzle since the last time I was able to successfully run, but I was hoping someone might have an idea. I am using the NVIDIA driver with the __GL_FORCE_GENERIC_CPU flag. I have no reason to suspect that there is any problem with this. I am also using Intel's IPP libraries that make heavy use of MMX and SSE acceleration. They provide versions that are not supposed to contain any MMX code, and I am linking with these. I suspect there may be a problem here, especially since I have upgraded versions since my last successful valgrind run. By liberally spreading printouts through my code and running it under valgrind, it appears that what is happening is that my application calls something in libpng, which calls the pow() function, which has been overridden by the IPP libraries. When disassembled by gdb, IPP's pow() seems to autodetect whether the CPU it is running on supports MMX, and dynamically selects a version of the function to run based on that. (I don't know *why* it is bothering to autodetect anything, since I told it to link with the non-MMX version of the library.) That detection appears to be resulting in it believing that it is running on an MMX processor (which it is, except for the valgrind VM layer). Fall down go boom. Or maybe I'm just horribly confused, and this isn't what's happening at all. I was mostly wondering if, now that valgrind supports MMX and some SSE, it is now reporting that it is an MMX/SSE cpu? If so, is there some way of making it act stupid again? (I don't know whether my problem is with MMX or SSE instructions, actually.) If it would help, I'll post the actual error message. I can't now, since it's on the other side of a very long recompile cycle. This is with valgrind-20030725, and also a CVS checkout from about a week ago. If nobody has any brilliant ideas, I guess I'll slog through downgrading each piece of the puzzle until I can valgrind again. Thanks! Steve |
|
From: Steve G <lin...@ya...> - 2003-08-19 01:14:22
|
>No, cache coherency suffices, really! You can't say that without having a particular problem in mind. For example, cache coherency doesn't suffice when I/O is in the mix. I think this thread has strayed way beyond Jeremy's original post. He was asking if it was worthwhile to have something like a Hellegrind race detector. Rather than pontificating the various ways a signal handler can work or not and how to synchronize, why don't we discuss the *merits* of a race detector? I think the sum of all of this thread is that signal handlers can be tricky. There are many that are bad. Should something be done to help locate problems? In one of the first e-mails of this thread, I wrote an algorithm that may help locate those problems. Does it work? Can anyone see a better way to do it? Am I the only one that believes a race detector would be helpful? This is an interesting subject, but I think we are off track. Jeremy's original question was thoughtful and deserves more consideration. Best Regards, -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com |
|
From: Jason G. <jga...@la...> - 2003-08-19 00:40:44
|
Is it possible to actually link against a valgrind library, or can one only run it through the executable? Thanks, Jason |
|
From: John R.
|
Erik Corry wrote:
> On Mon, Aug 18, 2003 at 01:08:11PM -0700, John Reiser wrote:
>
>>Dan Kegel wrote:
>>
>>>John Reiser wrote:
>>>
>>>
>>>>>Well, surely the buffer pointer updates need to be protected, unless
>>>>>you're using strictly atomic operations?
>>>>
>>>>
>>>>No, the buffer pointer updates do not need to be "protected".
>>>
>>[snip]
>> >> the significant exceptions are on "8-bit"
>>
>>>>(or less) microcontrollers where a pointer is wider than the memory bus.
>>>
>>>
>>>Don't forget the other significant exception, which is on SMP
>>>machines. You may need memory barrier instructions to
>>>avoid caching problems if variables are used by more than
>>>one processor.
>>
>>Cache coherency suffices.
>
>
> No, it's also necessary that the CPU not reorder writes. x86 never does, but
> I think the Alpha, for example, does unless you put in a memory barrier.
>
No, cache coherency suffices, really! At least for the control pointers;
I agree that a write ordering memory barrier is necessary after writing
the data and before writing the control pointer.
Remember, each processor writes only _one_ control pointer. None of
the memory re-ordering schemes allow re-ordering writes to the _same_
address. If one [the only] processor writes successive values 1, 2, 3, 4, 5
to the _same_ memory address (say, 0x89abc), then it is not legal
for any sequence of reads of that address 0x89abc, by _any_ [one]
processor (including the processor that does the writing), to return
1, 3, 2, 5, 4; or anything except a monotonic subsequence of the
ordered sequence {<some prior value>, 1, 2, 3, 4, 5}.
|
|
From: Erik C. <er...@ar...> - 2003-08-18 20:47:08
|
On Mon, Aug 18, 2003 at 01:08:11PM -0700, John Reiser wrote: > Dan Kegel wrote: > > John Reiser wrote: > > > >>> Well, surely the buffer pointer updates need to be protected, unless > >>> you're using strictly atomic operations? > >> > >> > >> No, the buffer pointer updates do not need to be "protected". > [snip] > >> the significant exceptions are on "8-bit" > >> (or less) microcontrollers where a pointer is wider than the memory bus. > > > > > > Don't forget the other significant exception, which is on SMP > > machines. You may need memory barrier instructions to > > avoid caching problems if variables are used by more than > > one processor. > > Cache coherency suffices. No, it's also necessary that the CPU not reorder writes. x86 never does, but I think the Alpha, for example, does unless you put in a memory barrier. See also http://www-1.ibm.com/servers/esdd/articles/power4_mem.html Also, you have to consider the risk that the compiler will reorder writes. It can do this if there is no difference in a single-CPU scenario. To avoid this you need a compiler-level memory barrier. Linux uses: __asm__ __volatile__ ("": : :"memory") #define wmb() __asm__ __volatile__ ("": : :"memory") which is an empty assembler statement that has "clobbering memory" as a side effect. The effect is that gcc can't move a store across it. Probably on other compilers you can achieve the same effect with lower performance by inserting a call to an extern function. See also http://groups.google.com/groups?selm=linux.kernel.200007272355.QAA03148%40pizda.ninka.net -- Erik Corry er...@ar... |
|
From: John R.
|
Dan Kegel wrote: > John Reiser wrote: > >>> Well, surely the buffer pointer updates need to be protected, unless >>> you're using strictly atomic operations? >> >> >> No, the buffer pointer updates do not need to be "protected". [snip] >> the significant exceptions are on "8-bit" >> (or less) microcontrollers where a pointer is wider than the memory bus. > > > Don't forget the other significant exception, which is on SMP > machines. You may need memory barrier instructions to > avoid caching problems if variables are used by more than > one processor. Cache coherency suffices. [If not cache coherent, then there is not _a_ memory: each cache is a different memory.] In one sense, cache coherency is stronger than necessary: as long as a write eventually propagates to the other cache [and becomes the value fetched by a read], then communication succeeds. The delay can affect throughput and latency, but not correctness. Multiple writes to the same control pointer [which are done by only one processor] may even be "optimized" by suppressing all but the most recent one. > > Semaphores provide the needed memory barriers. I'm sure > some pthread stuff does, too. > Volatile variables don't, I think (except maybe in Java :-) > > It's such a pain that I personally use synchronous signal > delivery for absolutely everything... > - Dan > |
|
From: Steve G <lin...@ya...> - 2003-08-18 19:35:47
|
>Semaphores provide the needed memory barriers. You cannot make any semaphore calls from a signal handler. >I'm sure some pthread stuff does, too. You cannot make any pthread calls either. Earlier you posted a link to the SUS, but if you look at the signal concepts page: http://www.opengroup.org/onlinepubs/007904975/functions/xsh_chap02_04.html#tag_02_04 You will see sem_post() is the only interprocess communication call allowed in a signal handler. Signal handlers are somewhat like multithreaded programming but an entirely different animal. They have to be much simpler and only call the "whitelisted" functions if they call anything at all. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com |
|
From: Nicholas N. <nj...@ca...> - 2003-08-18 19:35:09
|
On Mon, 18 Aug 2003, Ing, Graeme wrote: > As per your FAQ, nothing in my executable modifies (or even reads) > environment variables. > Command line is: > valgrind ./debug/CSAssistServer > > VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error: > what = 1 > ld_preload_str = `' Are you completely sure nothing modifies LD_PRELOAD? I ask because, as you can see, it's printed out when this case occurs and it's empty. Unless something else weird is happening... N |
|
From: Ing, G. <gi...@so...> - 2003-08-18 19:13:13
|
Here's an email I just sent to jseward, in case anyone else on this list has ideas or experience with this. Thanks! I received one of your "impossible errors". Have you found a cure for these yet? gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7) Red Hat Linux release 8.0 (Psyche) glib-1.2.10-8 valgrind-20030725 As per your FAQ, nothing in my executable modifies (or even reads) environment variables. Command line is: valgrind ./debug/CSAssistServer Any help would be welcome, alas the project is too large to send you as a sample. I'm just about to see if I can generate the error with a test file. <Valgrind works fine on other executables, there is clearly something about this particular one, which is a very large server> Graeme ==9058== Memcheck, a.k.a. Valgrind, a memory error detector for x86-linux. ==9058== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward. ==9058== Using valgrind-20030725, a program supervision framework for x86-linux. ==9058== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward. ==9058== Estimated CPU clock rate is 1390 MHz ==9058== For more details, rerun with: -v ==9058== VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error: what = 1 ld_preload_str = `' ld_library_path_str = `/usr/local/lib/valgrind::/m1/homes/ging/projects/lib:/opt/app/oracle/produc t/8.1.6/lib:/m1/homes/ging/STLport-4.5.3/lib' vg_prel = `(null)' sk_prel = `(null)' coredir2 = `(null)' ==4920== Memcheck, a.k.a. Valgrind, a memory error detector for x86-linux. ==4920== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward. ==4920== Using valgrind-20030725, a program supervision framework for x86-linux. ==4920== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward. ==4920== ==4920== My PID = 4920, parent PID = 9893. Prog and args are: ==4920== ./debug/CSAssistServer ==4920== Estimated CPU clock rate is 1391 MHz ==4920== For more details, rerun with: -v ==4920== VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error: what = 1 ld_preload_str = `' ld_library_path_str = `/usr/local/lib/valgrind::/m1/homes/ging/projects/lib:/opt/app/oracle/produc t/8.1.6/lib:/m1/homes/ging/STLport-4.5.3/lib' vg_prel = `(null)' sk_prel = `(null)' coredir2 = `(null)' VG_LIBDIR = `/usr/local/lib' Note that this is often caused by mis-installation of valgrind. Correct installation procedure is: ./configure --prefix=/install/dir make install And then use /install/dir/bin/valgrind Moving the installation directory elsewhere after 'make install' will cause the above error. Hand-editing the paths in the shell scripts is also likely to cause problems. valgrind: the `impossible' happened: VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH) failed Basic block ctr is approximately 300000 sched status: Thread 1: status = Runnable, associated_mx = 0x0, associated_cv = 0x0 ==4921== at 0x420B0357: __execve (in /lib/i686/libc-2.3.2.so) ==4921== by 0x420B0491: execv (in /lib/i686/libc-2.3.2.so) ==4921== by 0x408A1B05: execv (in /usr/local/parasoft/lib.linux2/libtql_c_gcc.so) ==4921== by 0x409D9F4D: Insure::Process::nativeNewProcess(Insure::NewProcessArgs const &, Insure::ThisThread *, Insure::LibraryCall &) (in /usr/local/parasoft/lib.linux2/libinsure.so) ==4921== by 0x409A2AD1: Insure::Process::newProcess(Insure::NewProcessArgs const &, Insure::ThisThread *, Insure::LibraryCall &) (in /usr/local/parasoft/lib.linux2/libinsure.so) ==4921== by 0x4097A08D: Insure::Pipe::newPipeline(Insure::Pipe *&, Insure::Pipe *&, Insure::Process *&, Insure_Argv_ThisThreadPtr const &, Insure::ThisThread *, Insure::LibraryCall &) (in /usr/local/parasoft/lib.linux2/libinsure.so) Note: see also the FAQ.txt in the source distribution. It contains workarounds to several common problems. If that doesn't help, please report this bug to: js...@ac... In the bug report, send all the above text, the valgrind version, and what Linux distro you are using. Thanks. Graeme "Fippy" Ing Senior Engineer Sony Online Entertainment Inc. gi...@so... (858) 577 3130 |
|
From: Dan K. <da...@ke...> - 2003-08-18 18:33:37
|
John Reiser wrote: >> Well, surely the buffer pointer updates need to be protected, unless >> you're using strictly atomic operations? > > No, the buffer pointer updates do not need to be "protected". As long > as the width of the pointer equals the width of the memory bus [or pointer > is narrower and the memory has natural operations of that width, and the > code uses them], and the storage for the pointer itself is naturally > aligned > [width divides address], then ordinary load and store operations suffice. > This is commonly the case; the significant exceptions are on "8-bit" > (or less) microcontrollers where a pointer is wider than the memory bus. Don't forget the other significant exception, which is on SMP machines. You may need memory barrier instructions to avoid caching problems if variables are used by more than one processor. Semaphores provide the needed memory barriers. I'm sure some pthread stuff does, too. Volatile variables don't, I think (except maybe in Java :-) It's such a pain that I personally use synchronous signal delivery for absolutely everything... - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
|
From: John R.
|
Jeremy Fitzhardinge wrote: > On Mon, 2003-08-18 at 06:59, John Reiser wrote: >>Even in more complicated situations, a circular buffer between the >>signal handler as producer and the mainline code as consumer, >>would violate the rule about "no touching" the control pointers >>of the circular buffer. > > > Well, surely the buffer pointer updates need to be protected, unless > you're using strictly atomic operations? No, the buffer pointer updates do not need to be "protected". As long as the width of the pointer equals the width of the memory bus [or pointer is narrower and the memory has natural operations of that width, and the code uses them], and the storage for the pointer itself is naturally aligned [width divides address], then ordinary load and store operations suffice. This is commonly the case; the significant exceptions are on "8-bit" (or less) microcontrollers where a pointer is wider than the memory bus. It may be argued that this makes the ordinary read+write of the control pointers "strictly atomic", but that is much of the point. The ordinary operations are enough; you don't need privilege, assembly code, a special compiler, or special tricks. This a major point of a [properly implemented] circular buffer. No bus interlock, no compare-and-swap, no semaphore. Ordinary load and store are atomic enough. The interlocks are implicit in the logic of the control pointers. For review: A circular buffer implements a one-way reliable channel between two process[es|ors] having read+write access to a common memory, without any hardware interlocks except ordinary read+write memory operations that are monotonic in time-causality. The buffer consists of a region of memory and four indices: FIRST, IN, OUT, LAST. At all times FIRST <= IN <= OUT <= LAST. FIRST and LAST are constant for the duration of the channel. Only the producer writes IN, and only the consumer writes OUT. If OUT==IN then the buffer is empty. If OUT < IN then the region beginning at memory[OUT] and extending upto, but not including, memory[IN] has data from the producer for the consumer. If IN < OUT then the region beginning at memory[OUT] and continuing upto, but not including, memory[LAST] has the first part of data, and the region from memory[FIRST] upto, but not including, memory[IN] has the second part of data from the producer for the consumer. Both the producer and consumer can operate simultaneously and achieve reliable one-way communication with low overhead. An early commercial implementation appeared in the Control Data Corporation (CDC) 6600/6500/6400 machines in the late 1960s. They had 11 or 12 processors: 1 or 2 CPU, plus 10 PPU (Peripheral Processor Unit). Each PPU had a 4K-word (18 bit) memory of its own, plus access to the 60-bit wide main memory. All I/O channels were connected to the PPUs, and not to the main CPU memory. Ordinarily, user code ran only on the CPU, and only the operating system ran PPUs. The only hardware interlock in the machine was the power-up lock which disabled every processor except PPU0 for system initialization. All I/O to/from main memory was conducted by a PPU using a circular buffer in main memory. The protocol was such that it was possible to stream a magnetic tape drive (10.5 inch reel of 0.5 inch tape, 2400 feet long, 556/800/1600 char-per-inch) and have a block size of 30 megabytes [the whole tape, with no inter-record gaps], with a user- mode program in the CPU doing meaningful processing of the data [not just copying]. The usual case with records of a few kilobytes or less [and gaps of a fraction of an inch with no data on the tape, which enabled the drive to start and stop if necessary without losing data] was even easier to stream. |
|
From: Dan K. <da...@ke...> - 2003-08-18 18:11:26
|
Dan Kegel wrote: > Jeremy Fitzhardinge wrote: > >>> Even in more complicated situations, a circular buffer between the >>> signal handler as producer and the mainline code as consumer, >>> would violate the rule about "no touching" the control pointers >>> of the circular buffer. > > Yep. In other words, there's something iffy about the > whole idea of signal handlers Arrgh. Clicked 'send' by mistake. I meant to say "classic signal handlers". The current Unix standard, http://www.opengroup.org/onlinepubs/007904975/functions/sigaction.html has this to say about the issue: "In order to prevent errors arising from interrupting non-reentrant function calls, applications should protect calls to these functions either by blocking the appropriate signals or through the use of some programmatic semaphore (see semget() , sem_init() , sem_open(), and so on). ..." - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
|
From: Jeremy F. <je...@go...> - 2003-08-18 18:01:05
|
On Mon, 2003-08-18 at 10:19, Steve G wrote: > >In my experience, however, programs which use such a > >simple scheme are prone to signal race deadlocks. > > Not really...it just depends on what your job is. I > re-wrote the signal handlers in proftpd about a year and > half ago using flags. The signal handlers set a flag when > something of interest occurs. The handler is call from the > main loop which periodically polls the flag. The handlers > reset the flag first, and then do the work. They completely > process everything pending. The worst case is that the same > signal is delivered while processing that flag. This causes > a call to the signal handler for which no work is waiting. Yes, this is the case where sigwait/sigtimedwait allows you to consume pending signals without async delivery - which makes the whole issue of calling handlers moot. > There's no way to protect data structures using process > synchronization techniques from a signal handler. Either > you have to do you homework up front in the design making > sure nothing steps on each other, or using signals as flags > and not as hardware interrupt handlers. Well, that's not really true. You can use sigprocmask to defer delivery of a signal for as long as you need to, so you can use it to protect your critical sections. > In case you missed the way that stunnel works, what it does > is create a signal pipe. Whenever a signal comes in, it > writes to the pipe. The pipe is added to the fd_set that > select watches. Whenever a signal occurs, the pipe becomes > readable and it goes off to handle that event. Write() is a > legal function call from a signal handler. What's written > to the pipe is an integer which is sigatomic in size. Right, that's what I do in the autofs4 autmount as well. It means that you're not relying on the signal to interrupt the syscall directly; you're using the normal select mechanisms. There's a theoretical risk of deadlock if the pipe fills up, but that just means you need to select/poll often enough. > Xinetd does the same thing except it uses a different > integer for each signal. The only issue you have to watch > when using the pipe technique for serialization is to call > fcntl(fd, F_SETFD, CLOEXEC); so you don't leak the pipe > descriptor. If you're forking child processes (no exec) you also have to be careful not to end up sharing your message pipe. (I'd really like CLOFORK.) > > >I think the most common case of manipulating data > >structures from a signal handler is in programs which > >keep track of their child processes: > > These are all fundamentally broken. See > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-1563 > for a good reason why stunnel moved to the current > technique. I disagree, because you can use sigprocmask to get it right. On the other hand, it is error-prone (like any multi-threaded programming), so it would be nice to have a tool to test for correctness. Hence this thread on this list. J |
|
From: Steve G <lin...@ya...> - 2003-08-18 17:56:11
|
>In my experience, however, programs which use such a >simple scheme are prone to signal race deadlocks. Not really...it just depends on what your job is. I re-wrote the signal handlers in proftpd about a year and half ago using flags. The signal handlers set a flag when something of interest occurs. The handler is call from the main loop which periodically polls the flag. The handlers reset the flag first, and then do the work. They completely process everything pending. The worst case is that the same signal is delivered while processing that flag. This causes a call to the signal handler for which no work is waiting. >If it relies on the signal to interrupt a syscall, then >it almost certainly has deadlock bugs. Almost all daemons depend on a signal to get them out of select(). >Well, surely the buffer pointer updates need to be >protected, unless you're using strictly atomic >operations? There's no way to protect data structures using process synchronization techniques from a signal handler. Either you have to do you homework up front in the design making sure nothing steps on each other, or using signals as flags and not as hardware interrupt handlers. In case you missed the way that stunnel works, what it does is create a signal pipe. Whenever a signal comes in, it writes to the pipe. The pipe is added to the fd_set that select watches. Whenever a signal occurs, the pipe becomes readable and it goes off to handle that event. Write() is a legal function call from a signal handler. What's written to the pipe is an integer which is sigatomic in size. Xinetd does the same thing except it uses a different integer for each signal. The only issue you have to watch when using the pipe technique for serialization is to call fcntl(fd, F_SETFD, CLOEXEC); so you don't leak the pipe descriptor. >I think the most common case of manipulating data >structures from a signal handler is in programs which >keep track of their child processes: These are all fundamentally broken. See http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-1563 for a good reason why stunnel moved to the current technique. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com |
|
From: Dan K. <da...@ke...> - 2003-08-18 16:54:01
|
Jeremy Fitzhardinge wrote: >>Even in more complicated situations, a circular buffer between the >>signal handler as producer and the mainline code as consumer, >>would violate the rule about "no touching" the control pointers >>of the circular buffer. Yep. In other words, there's something iffy about the whole idea of signal handlers > > > Well, surely the buffer pointer updates need to be protected, unless > you're using strictly atomic operations? > > >> What did you really mean? > > > Erm, what I said, I think. > > I think the most common case of manipulating data structures from a > signal handler is in programs which keep track of their child processes: > they seem to do a lot of work in their SIGCHLD handlers. > > J > > > > ------------------------------------------------------- > This SF.Net email sponsored by: Free pre-built ASP.NET sites including > Data Reports, E-commerce, Portals, and Forums are available now. > Download today and enter to win an XBOX or Visual Studio .NET. > http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 > _______________________________________________ > Valgrind-users mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-users > > -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
|
From: Jeremy F. <je...@go...> - 2003-08-18 15:59:18
|
On Mon, 2003-08-18 at 06:59, John Reiser wrote: > > The general idea that if you're touching a piece of memory and the > > handler for signal N also touches that memory, you'd better have signal > > N blocked. > > But only for "complex" situations. For "simple" cases the _recommended_ > methodolgy violates this rule. Namely, the signal handler sets (or > increments) a global variable, and the non-signal handling code checks > this variable at convenient times (to see if it has changed from the > last time the non-handler code checked its value). Well, you probably wouldn't bother running it under crocus to test its correctness in that case. In my experience, however, programs which use such a simple scheme are prone to signal race deadlocks. If its a program which can truly work with only a flag set in the signal handler, it would probably be much better off to simply block all signals and poll for pending signals with sigwait(). If it relies on the signal to interrupt a syscall, then it almost certainly has deadlock bugs. > Even in more complicated situations, a circular buffer between the > signal handler as producer and the mainline code as consumer, > would violate the rule about "no touching" the control pointers > of the circular buffer. Well, surely the buffer pointer updates need to be protected, unless you're using strictly atomic operations? > What did you really mean? Erm, what I said, I think. I think the most common case of manipulating data structures from a signal handler is in programs which keep track of their child processes: they seem to do a lot of work in their SIGCHLD handlers. J |
|
From: John R.
|
Zeljko Vrba wrote: > On Thu, Aug 14, 2003 at 11:29:44AM -0700, John Reiser wrote: > >>>Lucky for us, so does the x86. There is a flag you can set in some >>>register or other which makes the CPU trap on unaligned access. So long >>>as we don't do it internally, you could set that and watch all the >>>unaligned accesses in your program (even without Valgrind). >> >>Yes, it's the AC [Alignment Check] bit of the processor flags, >>which is bit 18 (the bit with positional value (1<<18) or 0x40000). >>It checks for unaligned access in user mode. >> > > But not only AC. As I recall, some other bit in some control register (CR) > must be enabled also. And even then, AC occurs only in user mode (ring 3). Did you try the example? It runs as expected in Linux 2.4.20, namely: generates SIGBUS for unaligned access from user mode (ring 3). So the AM bit (also bit 18, but in CR0) is enabled by the kernel. Valgrind (memcheck) applies only to user mode, so there is no additional restriction on this method of checking alignment of memory accesses. Even better, you can check any executable without recompilation: $ gdb a.elf (gdb) b main # any place you want to start checking (gdb) run Breakpoint 1 (gdb) set var $ps |= 1<<18 # set AlignmentCheck bit of eflags (gdb) continue and now gdb will catch SIGBUS for any unaligned access. To continue past each unaligned access: (gdb) handle SIGBUS nopass print stop # notify gdb+user, and not a.elf (gdb) set var $ps &= ~(1<<18) # clear AlignmentCheck bit of eflags (gdb) stepi # perform unaligned access (gdb) set var $ps |= (1<<18) # set AlignmentCheck bit of eflags (gdb) continue and those commands may be put into a gdb macro command for ease of use. |
|
From: John R.
|
> The general idea that if you're touching a piece of memory and the > handler for signal N also touches that memory, you'd better have signal > N blocked. But only for "complex" situations. For "simple" cases the _recommended_ methodolgy violates this rule. Namely, the signal handler sets (or increments) a global variable, and the non-signal handling code checks this variable at convenient times (to see if it has changed from the last time the non-handler code checked its value). Even in more complicated situations, a circular buffer between the signal handler as producer and the mainline code as consumer, would violate the rule about "no touching" the control pointers of the circular buffer. What did you really mean? |
|
From: Steve G <lin...@ya...> - 2003-08-18 14:09:25
|
>The general idea that if you're touching a piece of >memory and the handler for signal N also touches that >memory, you'd better have signal N blocked. I suspect if you see code that does this, it should be redesigned. Some good examples of serialization are stunnel 4.04 or xinetd-2.3.12. Come to think of it, postfix may be another example of proper serialization. I suppose an approach might be something like this: Each time a signal is about to be delivered, check the mask. If any unblocked signal (besides the one about to be delivered) has a user defined signal handler, fake delivering a signal to it and only note the addresses it wanted to read data from or write data to and what the access type is (read or write). Don't let it actually write, reading is ok since its non-destructive. Then after faking a signal to all user defined handlers, check the list each time the current signal handler accesses a memory location. Only read/write or write/write collisions matter. The list gets thrown away at the end of the signal handler. I don't think performance is an issue since signals don't get delivered very often in normal applications. The operating system even delays delivery until a system call is made. I think this is managable so long as preventing writes from the other signal handlers is possible. I have no idea if this is tricky in the virtual cpu. I think finding these problems does matter and is worthwhile. Does anyone see issues with the above algorithm? Of course, I am not familiar with the inner workings of valgrind and someone else may have to code it. Best Regards, -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com |
|
From: Zeljko V. <zel...@av...> - 2003-08-18 10:49:31
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, Aug 14, 2003 at 11:29:44AM -0700, John Reiser wrote: > >Lucky for us, so does the x86. There is a flag you can set in some > >register or other which makes the CPU trap on unaligned access. So long > >as we don't do it internally, you could set that and watch all the > >unaligned accesses in your program (even without Valgrind). > > Yes, it's the AC [Alignment Check] bit of the processor flags, > which is bit 18 (the bit with positional value (1<<18) or 0x40000). > It checks for unaligned access in user mode. > But not only AC. As I recall, some other bit in some control register (CR) must be enabled also. And even then, AC occurs only in user mode (ring 3). - -- v Zeljko Vrba, dipl. ing. http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x3B375F1C zel...@av... Advanced Simulation Technologies http://www.avl.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQE/QKw2zT2jbzs3XxwRAhUCAJ0dDXnGKC/OAxbGRYXg/HFufg21CACfXs78 /n9YBcjBGtSdVYUM3p+e790= =5Z3H -----END PGP SIGNATURE----- |
|
From: Benjamin L. <ben...@us...> - 2003-08-18 10:02:13
|
Do you mean something like the file attached? (I could be completely wrong, by the way -- I just ripped off __NR_wait4). ;-) On Monday, 2003-08-18 at 02:21:01 AM, Sefer Tov scribbled: > Hi! > > I have recently tried an old program of mine using valgrind (latest cvs > version) and encountered an error stating it's missing an implementation > syscall no. 7 on my Mandrake 9.1 (glibc 2.3.2) box. > I looked through the header files and it seems to be coming from a missing > implementation of "waitpid" (called via __libc_waitpid). > > I'd appreciate it if anyone could offer a patch to fix that and integrate > it into the head branch. > > Thanks, > Sefer. -- Benjamin Lee Melbourne, Australia "Always real." http://www.realthought.net/ __________________________________________________________________________ Only those who leisurely approach that which the masses are busy about can be busy about that which the masses take leisurely. -- Lao Tsu |