Menu
▾
▴
valgrind-users
|
From: mnaret <mn...@ci...> - 2013-06-10 08:23:37
|
Hello, Recently I'm getting lot's of "invalid read/invalid write" valgrind errors which point out at memory allocated for the stack. However the code doesn't crush and finish running successfully. I'm trying to understand where the error comes from - and will be grateful fo any help wih this issue. Here's what I can see using vgdb: (gdb) monitor v.info last_error ==10259== Invalid write of size 4 ==10259== at 0x28686C: vsnprintf (in /lib/libc-2.12.so) ==10259== Address 0x4b43040 is 45,120 bytes inside a block of size 65,536 alloc'd ==10259== at 0x4005DB9: memalign (vg_replace_malloc.c:727) ==10259== by 0x4005E68: posix_memalign (vg_replace_malloc.c:876) -The memory for the stack is allocated using memalign and then the upper and lower parts of it are protected using mprotect, so the stack looks like this: 16k protected with mprotect, 32K valid for usage, another 16K protected. The problem happens only in the valid area. -The problem always seems to happen in the end of the first page of the valid memory. For example, the stack above starts with the address 0x4b44000 and goes down, the first inaccessible address is 0x4b4304b. -The address pointed out as inaccessible is above the stack pointer: (gdb) info registers eax 0x100 256 ecx 0x0 0 edx 0x0 0 ebx 0x3acff4 3854324 esp 0x4b42f44 0x4b42f44 ebp 0x4b4304c 0x4b4304c esi 0x4b43b70 78920560 edi 0x4b431d4 78918100 eip 0x28686c 0x28686c <vsnprintf+12> -Even though addressibility is not supposed to be affected by mprotect, when I comment out the calls to mprotect the poblem doen't happen any more. -There's no specific line in the code that is causing the problem. It seems that he problem always happens in the end of the first page of the stack. I tried checkig the memory at the point of allocation - and the pointed out address is valid. It also seems valid when the thread starts running and becoms invalid only when the stack reaches near 4K. Considering the fact there's no crash in the program and it runs normally and correctly in spite of those errors, I don't understand what is the reason of valgrind componains. Any help with the cause of the problem or with further evaluation will be highly appreciated, as after spending a few days on this I'm run out of ideas. Thank you, Masha. -- View this message in context: http://valgrind.10908.n7.nabble.com/Valgrind-shows-Invalid-write-os-size-4-for-memory-allocated-for-the-stack-tp45597.html Sent from the Valgrind - Users mailing list archive at Nabble.com. |
|
From: John R. <jr...@bi...> - 2013-06-10 14:12:53
|
> (gdb) monitor v.info last_error > ==10259== Invalid write of size 4 > ==10259== at 0x28686C: vsnprintf (in /lib/libc-2.12.so) > ==10259== Address 0x4b43040 is 45,120 bytes inside a block of size 65,536 > alloc'd > ==10259== at 0x4005DB9: memalign (vg_replace_malloc.c:727) > ==10259== by 0x4005E68: posix_memalign (vg_replace_malloc.c:876) > > > -The memory for the stack is allocated using memalign and then the upper and > lower parts of it are protected using mprotect, so the stack looks like > this: 16k protected with mprotect, 32K valid for usage, another 16K > protected. The problem happens only in the valid area. There is an inherent conflict between memcheck and mprotect. On one side, it is desirable that after mprotect which removes all access privileges then memcheck's accounting bits should reflect the current state, including no access. On the other side, it is desirable that after a following mprotect which restores the previous access privileges to the same region, then memcheck's accounting bits should reflect the state *AS IF* those two mprotect() never had occurred. That is, it would be nice if memcheck remembered which bytes were defined and undefined, even though the region "disappeared behind the mprotect NO-ACCESS curtain" for a while. I don't know the exact situation now, but some time in the past it was impossible to have both properties because the accounting bits couldn't handle it. -- |
|
From: Philippe W. <phi...@sk...> - 2013-06-12 19:43:41
|
On Mon, 2013-06-10 at 01:23 -0700, mnaret wrote: > Hello, > Recently I'm getting lot's of "invalid read/invalid write" valgrind errors > which point out at memory allocated for the stack. However the code doesn't > crush and finish running successfully. > I'm trying to understand where the error comes from - and will be grateful > fo any help wih this issue. Do you have a small (compilable) reproducer ? Philippe |
|
From: Masha N. (mnaret) <mn...@ci...> - 2013-06-13 06:40:31
|
Not yet, Following the first answer in the forum I tried several code modifications around mprotect: 1. When releasing the memory, in addition to setting mprotect to PROT_READ and PROT_WRITE, set it PROT_EXEC 2. When allocating the memory, in addition to calling mprotect for the upper and lower parts of the allocated stack with PROT_NONE, Call mprotect with PROT_WRITE and PROT_READ for the valid part of the stack. Each one of the above solutions solved the problem. (Separately) Regarding the small reproducer: The problem only appears after the tests bring the process up and down several times, And only after the pages where the problem appears were allocated for some other stack, released, and then allocated again for the current stack. I'm not sure how to reproduce this behavior in a small executable, I'll try now. Thank you for your reply, Masha. -----Original Message----- From: Philippe Waroquiers [mailto:phi...@sk...] Sent: Wednesday, June 12, 2013 10:44 PM To: Masha Naret (mnaret) Cc: val...@li... Subject: Re: [Valgrind-users] Valgrind shows "Invalid write os size 4" for memory allocated for the stack On Mon, 2013-06-10 at 01:23 -0700, mnaret wrote: > Hello, > Recently I'm getting lot's of "invalid read/invalid write" valgrind > errors which point out at memory allocated for the stack. However the > code doesn't crush and finish running successfully. > I'm trying to understand where the error comes from - and will be > grateful fo any help wih this issue. Do you have a small (compilable) reproducer ? Philippe |
|
From: Masha N. (mnaret) <mn...@ci...> - 2013-07-17 09:10:11
|
Hello,
Attaching the reproducer - please compile it with gcc -g tmpthread.c -o tmpthread -lpthread
And then run: tmpthread argsFile
Also attached argsFile and valgrind log.
There are two "Invalid write" errors in the log, but only the second one seems to be the same as in the original program.
Some more explanation:
In the original program I noticed valgrind reported the problem when the stack, while growing, grew from area with permissions 'rwx' to area with permissions 'rw'.
The difference in premissions happened since each time thread stack is created, it's lower and upper part are protected with mprotect(NONE) and when it's out of use the protection is set back to READ | WRITE.
(exactly as in reproducer)
However, when running the program with valgrind, for some reason initially the all the momery is set to 'rwx'.
Which is weird, since normally the memory for the stack shouldn't have the 'x' permission.
When running the program without valgrind, there's no memory with 'rwx', only 'rw'
If I modify the original program to restore also EXEC persmission, valgrind doesn't report any error, however this doesn't seem like a correct thing to do.
The same happens with the reproducer.
If you modify the code of the function restoring permissions to add PROT_EXEC persmissions :
static void ReleaseThreadStack(int stacksize, void** stackbase)
{
int ret = 0;
void* ptStack = NULL;
//remove protection
ptStack = *stackbase - STACK_GUARD_SIZE;
ret = mprotect(ptStack, STACK_GUARD_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC );
ptStack = *stackbase + stacksize;
ret = mprotect(ptStack, STACK_GUARD_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC);
return;
}
The second "invalid write" problem will disappear!
If there’s need of any further logs, please let me know, I’ll send you valgrind debugger outputs.
Thank you very much for your support,
Masha.
-----Original Message-----
From: Philippe Waroquiers [mailto:phi...@sk...]
Sent: Wednesday, June 12, 2013 10:44 PM
To: Masha Naret (mnaret)
Cc: val...@li...
Subject: Re: [Valgrind-users] Valgrind shows "Invalid write os size 4" for memory allocated for the stack
On Mon, 2013-06-10 at 01:23 -0700, mnaret wrote:
> Hello,
> Recently I'm getting lot's of "invalid read/invalid write" valgrind
> errors which point out at memory allocated for the stack. However the
> code doesn't crush and finish running successfully.
> I'm trying to understand where the error comes from - and will be
> grateful fo any help wih this issue.
Do you have a small (compilable) reproducer ?
Philippe
|
|
From: Masha N. (mnaret) <mn...@ci...> - 2013-07-24 14:06:09
|
Hello,
Could anyone please comment the issue described below:
Valgrind shows "invalid write" for memory allocated on the stack while the program runs normally.
I've attached a reproducer for this problem,
During the run two errors show in valgrind.log,
However only the second one is relevant (is the same as the one I have in code)
There are more details in my previous posts on this subject.
Attching once again
Thank you in advance,
Masha.
-----Original Message-----
From: Masha Naret (mnaret)
Sent: Wednesday, July 17, 2013 12:10 PM
To: 'Philippe Waroquiers'; 'lop...@gm...'
Cc: 'val...@li...'; Yanai, Omer (OY...@nd...)
Subject: RE: [Valgrind-users] Valgrind shows "Invalid write os size 4" for memory allocated for the stack
Hello,
Attaching the reproducer - please compile it with gcc -g tmpthread.c -o tmpthread -lpthread And then run: tmpthread argsFile Also attached argsFile and valgrind log.
There are two "Invalid write" errors in the log, but only the second one seems to be the same as in the original program.
Some more explanation:
In the original program I noticed valgrind reported the problem when the stack, while growing, grew from area with permissions 'rwx' to area with permissions 'rw'.
The difference in premissions happened since each time thread stack is created, it's lower and upper part are protected with mprotect(NONE) and when it's out of use the protection is set back to READ | WRITE.
(exactly as in reproducer)
However, when running the program with valgrind, for some reason initially the all the momery is set to 'rwx'.
Which is weird, since normally the memory for the stack shouldn't have the 'x' permission.
When running the program without valgrind, there's no memory with 'rwx', only 'rw'
If I modify the original program to restore also EXEC persmission, valgrind doesn't report any error, however this doesn't seem like a correct thing to do.
The same happens with the reproducer.
If you modify the code of the function restoring permissions to add PROT_EXEC persmissions :
static void ReleaseThreadStack(int stacksize, void** stackbase) {
int ret = 0;
void* ptStack = NULL;
//remove protection
ptStack = *stackbase - STACK_GUARD_SIZE;
ret = mprotect(ptStack, STACK_GUARD_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC );
ptStack = *stackbase + stacksize;
ret = mprotect(ptStack, STACK_GUARD_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC);
return;
}
The second "invalid write" problem will disappear!
If there’s need of any further logs, please let me know, I’ll send you valgrind debugger outputs.
Thank you very much for your support,
Masha.
-----Original Message-----
From: Philippe Waroquiers [mailto:phi...@sk...]
Sent: Wednesday, June 12, 2013 10:44 PM
To: Masha Naret (mnaret)
Cc: val...@li...
Subject: Re: [Valgrind-users] Valgrind shows "Invalid write os size 4" for memory allocated for the stack
On Mon, 2013-06-10 at 01:23 -0700, mnaret wrote:
> Hello,
> Recently I'm getting lot's of "invalid read/invalid write" valgrind
> errors which point out at memory allocated for the stack. However the
> code doesn't crush and finish running successfully.
> I'm trying to understand where the error comes from - and will be
> grateful fo any help wih this issue.
Do you have a small (compilable) reproducer ?
Philippe
|
|
From: Philippe W. <phi...@sk...> - 2013-07-24 20:08:49
|
On Wed, 2013-07-24 at 14:05 +0000, Masha Naret (mnaret) wrote: > Hello, > > Could anyone please comment the issue described below: > Valgrind shows "invalid write" for memory allocated on the stack while the program runs normally. > I've attached a reproducer for this problem, Thanks for the reproducer, I started looking at this problem (limited time permitting :) some days ago. Stack management in valgrind is somewhat tricky (see e.g. revision r13467: fix 321960 pthread_create() then alloca() causing invalid stack write errors). However, r13647 does not fix your specific case. (you might however try the last SVN version on your full application). At this moment, the hypothesis is that if the stack memory is managed by the user, then when the thread terminates, the stack memory is not marked by memcheck as addressible again. I need to investigate more in depth to confirm this hypothesis. Philippe |
|
From: Philippe W. <phi...@sk...> - 2013-07-30 20:44:42
|
On Wed, 2013-07-24 at 22:08 +0200, Philippe Waroquiers wrote: > At this moment, the hypothesis is that if the stack memory is > managed by the user, then when the thread terminates, the stack memory > is not marked by memcheck as addressible again. > I need to investigate more in depth to confirm this hypothesis. Investigations have confirmed the hyposthesis and have lead to https://bugs.kde.org/show_bug.cgi?id=323020 which summarises/collects several strange aspects of thread stack handling. At this point, the only possible fix I see is to add a valgrind client request in your code to mark the released stack as accessible. No clear way how this can be done automatically by valgrind, without introducing the risk of more false positive and/or false negative. Philippe |
Thanks for helping keep SourceForge clean.
X