|
From: 邓尧 <to...@gm...> - 2012-12-14 03:36:16
|
Hi, Under 64-bit linux, a program could mmap() a very large space (at least 1T) when PROT_NONE is used. But when the program is running under valgrind, it could mmap() at most 2G, otherwise mmap() would return EINVAL. Is this a limitation, bug or feature of valgrind ? Thanks Yao |
|
From: John R. <jr...@bi...> - 2012-12-14 03:53:26
|
> Under 64-bit linux, a program could mmap() a very large space (at least 1T) when PROT_NONE is used. But when the program is running under valgrind, it could mmap() at most 2G, otherwise mmap() would return EINVAL.
> Is this a limitation, bug or feature of valgrind ?
It works for me in valgrind-3.8.1.
-----
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/mman.h>
main(int argc, char *argv[])
{
unsigned long len = 0;
sscanf(argv[1], "%li", &len);
void *addr = mmap(0, len, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
if (MAP_FAILED==addr) {
printf("%ld{0x%lx) bytes failed.\n", len, len);
return 1;
}
else {
printf("%ld(0x%lx) bytes at %p\n", len, len, addr);
return 0;
}
}
-----
$ gcc -g -o mmap-big mmap-big.c
$ ./mmap-big 0x100000000 ## 4G
4294967296(0x100000000) bytes at 0x7f82fc92d000
$ valgrind ./mmap-big 0x100000000
==18060== Memcheck, a memory error detector
==18060== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==18060== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==18060== Command: ./mmap-big 0x100000000
==18060==
==18060== Warning: set address range perms: large range [0x3aee0000, 0x13aee0000) (noaccess)
4294967296(0x100000000) bytes at 0x3aee0000
==18060==
==18060== HEAP SUMMARY:
==18060== in use at exit: 0 bytes in 0 blocks
==18060== total heap usage: 1 allocs, 1 frees, 256 bytes allocated
==18060==
==18060== All heap blocks were freed -- no leaks are possible
==18060==
==18060== For counts of detected and suppressed errors, rerun with: -v
==18060== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
$
--
|
|
From: 邓尧 <to...@gm...> - 2012-12-14 07:30:00
|
My apologies. I did some more tests, it's more than 2G, but still couldn't
map more than 16G. I'm using valgrind-3.7.0 on linux-3.2.0 x86_64
On Fri, Dec 14, 2012 at 11:54 AM, John Reiser <jr...@bi...> wrote:
> > Under 64-bit linux, a program could mmap() a very large space (at least
> 1T) when PROT_NONE is used. But when the program is running under valgrind,
> it could mmap() at most 2G, otherwise mmap() would return EINVAL.
> > Is this a limitation, bug or feature of valgrind ?
>
> It works for me in valgrind-3.8.1.
> -----
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/types.h>
> #include <sys/mman.h>
>
> main(int argc, char *argv[])
> {
> unsigned long len = 0;
> sscanf(argv[1], "%li", &len);
> void *addr = mmap(0, len, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
> if (MAP_FAILED==addr) {
> printf("%ld{0x%lx) bytes failed.\n", len, len);
> return 1;
> }
> else {
> printf("%ld(0x%lx) bytes at %p\n", len, len, addr);
> return 0;
> }
> }
> -----
> $ gcc -g -o mmap-big mmap-big.c
> $ ./mmap-big 0x100000000 ## 4G
> 4294967296(0x100000000) bytes at 0x7f82fc92d000
> $ valgrind ./mmap-big 0x100000000
> ==18060== Memcheck, a memory error detector
> ==18060== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
> ==18060== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
> ==18060== Command: ./mmap-big 0x100000000
> ==18060==
> ==18060== Warning: set address range perms: large range [0x3aee0000,
> 0x13aee0000) (noaccess)
> 4294967296(0x100000000) bytes at 0x3aee0000
> ==18060==
> ==18060== HEAP SUMMARY:
> ==18060== in use at exit: 0 bytes in 0 blocks
> ==18060== total heap usage: 1 allocs, 1 frees, 256 bytes allocated
> ==18060==
> ==18060== All heap blocks were freed -- no leaks are possible
> ==18060==
> ==18060== For counts of detected and suppressed errors, rerun with: -v
> ==18060== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
> $
>
> --
>
>
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
>
|
|
From: Julian S. <js...@ac...> - 2012-12-14 10:38:17
|
On Friday, December 14, 2012, 邓尧 wrote:
> My apologies. I did some more tests, it's more than 2G, but still couldn't
> map more than 16G. I'm using valgrind-3.7.0 on linux-3.2.0 x86_64
Yes, that is as expected. By default V will only use 32GB of memory,
and that is divided more or less evenly between the application and
Valgrind itself. Hence your app will not be able to use more than
16GB.
The patch below doubles those limits, so you can go up to 32GB.
Give it a try. I would like to commit it, but it needs testing.
Oh, and upgrade to 3.8.1.
J
Index: memcheck/mc_main.c
===================================================================
--- memcheck/mc_main.c (revision 13179)
+++ memcheck/mc_main.c (working copy)
@@ -167,10 +167,10 @@
#else
-/* Just handle the first 32G fast and the rest via auxiliary
+/* Just handle the first 64G fast and the rest via auxiliary
primaries. If you change this, Memcheck will assert at startup.
See the definition of UNALIGNED_OR_HIGH for extensive comments. */
-# define N_PRIMARY_BITS 19
+# define N_PRIMARY_BITS 20
#endif
@@ -6524,11 +6524,11 @@
tl_assert(sizeof(Addr) == 8);
tl_assert(sizeof(UWord) == 8);
tl_assert(sizeof(Word) == 8);
- tl_assert(MAX_PRIMARY_ADDRESS == 0x7FFFFFFFFULL);
- tl_assert(MASK(1) == 0xFFFFFFF800000000ULL);
- tl_assert(MASK(2) == 0xFFFFFFF800000001ULL);
- tl_assert(MASK(4) == 0xFFFFFFF800000003ULL);
- tl_assert(MASK(8) == 0xFFFFFFF800000007ULL);
+ tl_assert(MAX_PRIMARY_ADDRESS == 0xFFFFFFFFFULL);
+ tl_assert(MASK(1) == 0xFFFFFFF000000000ULL);
+ tl_assert(MASK(2) == 0xFFFFFFF000000001ULL);
+ tl_assert(MASK(4) == 0xFFFFFFF000000003ULL);
+ tl_assert(MASK(8) == 0xFFFFFFF000000007ULL);
# endif
}
Index: coregrind/m_aspacemgr/aspacemgr-linux.c
===================================================================
--- coregrind/m_aspacemgr/aspacemgr-linux.c (revision 13179)
+++ coregrind/m_aspacemgr/aspacemgr-linux.c (working copy)
@@ -1649,7 +1649,7 @@
aspacem_minAddr = (Addr) 0x04000000; // 64M
# if VG_WORDSIZE == 8
- aspacem_maxAddr = (Addr)0x800000000 - 1; // 32G
+ aspacem_maxAddr = (Addr)0x1000000000ULL - 1; // 64G
# ifdef ENABLE_INNER
{ Addr cse = VG_PGROUNDDN( sp_at_startup ) - 1;
if (aspacem_maxAddr > cse)
|
|
From: Roland M. <rol...@nr...> - 2012-12-14 12:15:48
|
On Fri, Dec 14, 2012 at 11:38 AM, Julian Seward <js...@ac...> wrote: > On Friday, December 14, 2012, 邓尧 wrote: >> My apologies. I did some more tests, it's more than 2G, but still couldn't >> map more than 16G. I'm using valgrind-3.7.0 on linux-3.2.0 x86_64 > > Yes, that is as expected. By default V will only use 32GB of memory, > and that is divided more or less evenly between the application and > Valgrind itself. Hence your app will not be able to use more than > 16GB. > > The patch below doubles those limits, so you can go up to 32GB. > Give it a try. I would like to commit it, but it needs testing. [snip] WIthout looking at the sources myself... ... would it be hard to make this kind of change a command-line option ? ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) rol...@nr... \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 3992797 (;O/ \/ \O;) |
|
From: Julian S. <js...@ac...> - 2012-12-14 12:57:21
|
On Friday, December 14, 2012, Roland Mainz wrote: > WIthout looking at the sources myself... > ... would it be hard to make this kind of change a command-line option ? Difficult to do without taking a performance hit in Memcheck. How much memory do you need? What is currently needed is someone to verify that, with this patch, the system can be run up to 32G (user) / 64G (total) without problems. I only have a machine with 8GB memory, so I can't test it myself. J |
|
From: John R. <jr...@bi...> - 2012-12-14 22:48:06
|
> Yes, that is as expected. By default V will only use 32GB of memory,
> and that is divided more or less evenly between the application and
> Valgrind itself. Hence your app will not be able to use more than
> 16GB.
>
> The patch below doubles those limits, so you can go up to 32GB.
> Give it a try. I would like to commit it, but it needs testing.
Using the patch to valgrind-3.8.1 on a machine with 32GB RAM and 80GB swap,
I can mmap 0x7f0000000 bytes (32GiB - 256MiB) to PROT_NONE,
but not 0x800000000 bytes (32GiB): [running Fedora 18-Final-TC2]
-----
$ /usr/local/valgrind-svn/bin/valgrind ./mmap-big 0x7f0000000
==15519== Memcheck, a memory error detector
==15519== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==15519== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
==15519== Command: ./mmap-big 0x7f0000000
==15519==
==15519== Warning: set address range perms: large range [0x805ce6000, 0xff5ce6000) (noaccess)
34091302912(0x7f0000000) bytes at 0x805ce6000
-----
When not running under valgrind, I can get almost to 128TiB:
-----
$ ./mmap-big 0x800000000000
140737488355328{0x800000000000) bytes failed.
$ ./mmap-big 0x7f0000000000
139637976727552(0x7f0000000000) bytes at 0x76dad8c000
$
-----
--
|
|
From: John R. <jr...@bi...> - 2012-12-14 22:56:59
|
> Using the patch to valgrind-3.8.1 on a machine with 32GB RAM and 80GB swap, Sorry; make that 16GB RAM and 80GB swap. -- |