[Valgrind-users] Program in valgrind behaves slightly different comparing to normal run

[olologin <olo...@gm...>]

Hi everyone,

First of all I want to thank everyone involved valgrind development, 
this set of tools, especially memcheck and helgrind saved me a lot of 
time. There were so many hard-to-find bugs in our code that were found 
only thanks to these tools.

Now to the question: I've been using memcheck and helgrind for almost a 
year now, and recently I noticed small instability in results provided 
by our code. For example if I run my program 10 times with the same 
output, 5/10 times I get wrong output, and I cannot understand why. So I 
tried memcheck, but when I run my software in it I always get right 
output, no matter how often I rerun it - output is always stable, and of 
course memcheck doesn't report any issues. Same issue happens with 
helgrind. I can suspect one of dynamic libraries for this instability, 
but I cannot find exact place that causes this issue. I have sources of 
that library, but it's quite massive, and I'm not original developer of 
it and unfortunately I cannot show it since it's closed-source. Most 
probably this issue is not caused by race-conditions, because it happens 
even when I run everything in single thread.

So far I tried to play with:
--smc-check=all
--redzone-size=*
--fair-sched=yes
--expensive-definedness-check=yes


Is there anything else I can try? I feel like there is some legitimate 
way of executing random branch of code, and somehow when I launch my 
program in valgrind - it always runs only good parts of code, this is 
why I don't see my program failing, and I don't see any issues reported 
in valgrind. But is there a way to make it run in same way as without 
valgrind? I mean to make it produce bad output in valgrind?

I'm using valgrind-3.12.0, my program is compiled with gcc6.3

Thanks in advance.