From: Matt Z. <md...@de...> - 2003-04-22 00:47:08
|
I noticed this when trying to build gcc 3.2 under UML. root is an ext2 filesystem on a ubd, and the build is taking place on a hostfs mount (/mnt). Here is the output from repeated invocations of gcc: (none):/mnt/home/mdz/data/src/deb/notmine/gcc-3.2-3.2.3ds8/build/libiberty# !gc gcc -save-temps -c -DHAVE_CONFIG_H -g -I. -I../../src/libiberty/../include -W -Wall -Wtraditional -pedantic -fpic ../../src/libiberty/regex.c -o pic/regex.o 2>&1 |head gcc: Internal error: Segmentation fault (program cpp0) Please submit a full bug report. See <URL:http://gcc.gnu.org/bugs.html> for instructions. For Debian GNU/Linux specific bugs, please see /usr/share/doc/debian/bug-reporting.txt. (none):/mnt/home/mdz/data/src/deb/notmine/gcc-3.2-3.2.3ds8/build/libiberty# gcc -save-temps -c -DHAVE_CONFIG_H -g -I. -I../../src/libiberty/../include -W -Wall -Wtraditional -pedantic -fpic ../../src/libiberty/regex.c -o pic/regex.o 2>&1 |head In file included from ../../src/include/xregex.h:26, from ../../src/libiberty/regex.c:195: ../../src/include/xregex2.h:572:4: warning: null character(s) ignored ../../src/include/xregex2.h:572:8: warning: null character(s) ignored ../../src/include/xregex2.h:572:16: warning: null character(s) ignored ../../src/include/xregex2.h:572:20: warning: null character(s) ignored ../../src/include/xregex2.h:572:22: warning: null character(s) ignored ../../src/include/xregex2.h:572:24: warning: null character(s) ignored ../../src/include/xregex2.h:572:31: warning: null character(s) ignored ../../src/include/xregex2.h:572:40: warning: null character(s) ignored (none):/mnt/home/mdz/data/src/deb/notmine/gcc-3.2-3.2.3ds8/build/libiberty# gcc -save-temps -c -DHAVE_CONFIG_H -g -I. -I../../src/libiberty/../include -W -Wall -Wtraditional -pedantic -fpic ../../src/libiberty/regex.c -o pic/regex.o 2>&1 |head In file included from ../../src/include/xregex.h:26, from ../../src/libiberty/regex.c:195: ../../src/include/xregex2.h:572:3: warning: null character(s) ignored ../../src/include/xregex2.h:573:4: warning: null character(s) ignored ../../src/include/xregex2.h:573:13: warning: null character(s) ignored ../../src/include/xregex2.h:573:18: warning: null character(s) ignored ../../src/include/xregex2.h:573:22: warning: null character(s) ignored ../../src/include/xregex2.h:573:26: warning: null character(s) ignored ../../src/include/xregex2.h:573:34: warning: null character(s) ignored ../../src/include/xregex2.h:573:40: warning: null character(s) ignored (none):/mnt/home/mdz/data/src/deb/notmine/gcc-3.2-3.2.3ds8/build/libiberty# gcc -save-temps -c -DHAVE_CONFIG_H -g -I. -I../../src/libiberty/../include -W -Wall -Wtraditional -pedantic -fpic ../../src/libiberty/regex.c -o pic/regex.o 2>&1 |head In file included from ../../src/libiberty/regex.c:649: ../../src/libiberty/regex.c:8378:47: missing terminating ' character ../../src/libiberty/regex.c:8825:1: warning: null character(s) ignored gcc: Internal error: Segmentation fault (program cpp0) Please submit a full bug report. See <URL:http://gcc.gnu.org/bugs.html> for instructions. For Debian GNU/Linux specific bugs, please see /usr/share/doc/debian/bug-reporting.txt. Occasionally, it will succeed. Usually, it acts as if there is garbage at the end of a file (the file xregex2.h is 571 lines, and gcc starts reporting errors at line 572). Sometimes it's all NULLs, other times more or less random data. Sometimes, cpp segfaults (possibly due to bugs in cpp which arise when running it on random binary garbage). I noticed this on the todo page: > I built UML inside itself twice on hostfs, with the output going to a log. > The first failed, and when the second ran, the log filled with garbage. This sounds vaguely similar. If the input file is larger than a certain size, calculated relative to the page size returned from getpagesize, cpp will use mmap() rather than read(), and it seems to be in these cases that the corruption occurs. So I suspect this is mmap-related. As far as I can tell, the mmap call is OK (it is mapping the correct number of bytes on a valid file descriptor), and the same compiler works OK outside of UML. Here's an strace excerpt: 12545 open("../../src/include/xregex.h", O_RDONLY|O_NOCTTY) = 4 12545 fstat64(4, {st_mode=S_IFREG|0770, st_size=783, ...}) = 0 12545 read(4, "/* This file redefines all regex"..., 783) = 783 12545 close(4) = 0 12545 open("../../src/include/xregex2.h", O_RDONLY|O_NOCTTY) = 4 12545 fstat64(4, {st_mode=S_IFREG|0770, st_size=21622, ...}) = 0 12545 old_mmap(NULL, 21622, PROT_READ, MAP_PRIVATE, 4, 0) = 0x4019b000 12545 close(4) = 0 12545 brk(0) = 0x83cd000 12545 brk(0x83ce000) = 0x83ce000 12545 brk(0) = 0x83ce000 12545 brk(0x83cf000) = 0x83cf000 12545 brk(0) = 0x83cf000 12545 brk(0x83d1000) = 0x83d1000 12545 brk(0) = 0x83d1000 12545 brk(0x83d3000) = 0x83d3000 12545 old_mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x401f7000 12545 write(2, "In file included from ../../src/"..., 233) = 233 [...crash and burn...] Note the close() of the mmapped fd immediately after mmap(). This sent me looking at the cpp code some more, and it looks like it essentially substitutes mmap for read in an open()-read()-close(), so the mmapped region is still being accessed after the close. I had to look this up, but apparently it is correct and this is supposed to work. Could this be where the problem lies? -- - mdz |
From: Matt Z. <md...@de...> - 2003-04-27 20:24:36
|
On Mon, Apr 21, 2003 at 08:45:02PM -0400, Matt Zimmerman wrote: > I noticed this when trying to build gcc 3.2 under UML. root is an ext2 > filesystem on a ubd, and the build is taking place on a hostfs mount > (/mnt). Here is the output from repeated invocations of gcc: I don't have much idea how to go about debugging this, but I'm very much able to reproduce it. The file which normally fails first (xregex2.h) does not depend on anything else in gcc, and I can run the gcc-3.2 preprocessor on it and get the bug more often than not. I can copy the same file onto a filesystem on a ubd, and the bug disappears. Likewise on the host, with the same versions of all software concerned (except the kernel). I've caught some of the output from the preprocessor a few times (usually cpp chokes so badly that it won't output anything), and the garbage seems to be pieces of other files I've worked with (cache?). This works with any file large enough for cpp to want to mmap it. I have not yet been able to reproduce it with any test programs, mostly because I don't understand what is going on in gcc here. Using gcc-2.95 instead, the preprocessor seems to work, but I run into another strange problem where the compiler gets a SIGTERM out of nowhere. I found this: http://www.geocrawler.com/archives/3/709/2001/8/50/6375430/ which says that this was known, but seemed to be fixed. -- - mdz |
From: Matt Z. <md...@de...> - 2003-04-27 23:19:35
|
On Sun, Apr 27, 2003 at 04:23:58PM -0400, Matt Zimmerman wrote: > I have not yet been able to reproduce it with any test programs, mostly > because I don't understand what is going on in gcc here. So I built gcc for debugging and spent some time tracking this down, and it turned out that the buffer wasn't null-terminated in this case, while it was in the working case: (gdb) print (*parse_in->buffer).buf[21621] $16 = 10 '\n' (gdb) print (*parse_in->buffer).buf[21622] $17 = 59 ';' vs. (gdb) print (*parse_in->buffer).buf[21621] $19 = 10 '\n' (gdb) print (*parse_in->buffer).buf[21622] $20 = 0 '\0' shortly thereafter, I came upon this comment in src/gcc/cppfiles.c: /* Use mmap if the file is big enough to be worth it (controlled by MMAP_THRESHOLD) and if we can safely count on there being at least one readable NUL byte after the end of the file's contents. This is true for all tested operating systems when the file size is not an exact multiple of the page size. */ And wrote a simple test program to look at the byte just after the end of an mmaped region (/ is ubd, /mnt is hostfs): sh-2.05b# ./mmaptest /etc/motd 0 sh-2.05b# ./mmaptest /etc/motd 0 sh-2.05b# ./mmaptest /etc/motd 0 sh-2.05b# ./mmaptest /etc/motd 0 sh-2.05b# ./mmaptest /etc/motd 0 sh-2.05b# ./mmaptest /mnt/etc/motd 0 sh-2.05b# ./mmaptest /mnt/etc/motd 8 sh-2.05b# ./mmaptest /mnt/etc/motd 8 sh-2.05b# ./mmaptest /mnt/etc/motd 8 It always comes up with a zero everywhere except on hostfs. It seems like this should be as simple as zeroing the page, since it only expects this when the size is not a multiple of the page size. I wouldn't know where to change this in UML, though. -- - mdz |
From: Henrik N. <hn...@ma...> - 2003-04-28 07:36:34
|
On Sun, 27 Apr 2003, Matt Zimmerman wrote: > by MMAP_THRESHOLD) and if we can safely count on there being > at least one readable NUL byte after the end of the file's > contents. This is true for all tested operating systems when > the file size is not an exact multiple of the page size. Which is for good reasons.. not zeroing the remainder of the page is a security issue. memset(buffer, 0, PAGE_CACHE_SIZE) just before the call to read_file in hostfs_readpage should address this problem I think. Note: A quick inspection of the hostfs writepage function makes me think there is another bug in flushing the last page of modified mmaped files with an even page size. Regards Henrik |
From: Matt Z. <md...@de...> - 2003-04-29 00:39:19
|
On Mon, Apr 28, 2003 at 09:36:20AM +0200, Henrik Nordstrom wrote: > Which is for good reasons.. not zeroing the remainder of the page is a > security issue. Indeed. > memset(buffer, 0, PAGE_CACHE_SIZE) just before the call to read_file > in hostfs_readpage should address this problem I think. It does, thank you. Now I have only the SIGTERM problem to deal with. If I restart the build enough times, it will eventually complete, but this is not ideal. :-) > Note: A quick inspection of the hostfs writepage function makes me think > there is another bug in flushing the last page of modified mmaped files > with an even page size. I'm afraid I am not familiar enough with the kernel to quickly grasp that function. What is the meaning of page->index? -- - mdz |
From: Jeff D. <jd...@ka...> - 2003-04-29 20:04:25
|
Nice diagnosis. hn...@ma... said: > memset(buffer, 0, PAGE_CACHE_SIZE) just before the call to read_file > in hostfs_readpage should address this problem I think. Yup. > Note: A quick inspection of the hostfs writepage function makes me > think there is another bug in flushing the last page of modified > mmaped files with an even page size. Yeah, I'll make sure about that. > I'm afraid I am not familiar enough with the kernel to quickly grasp that > function. What is the meaning of page->index? md...@de... said: > I'm afraid I am not familiar enough with the kernel to quickly grasp that > function. What is the meaning of page->index? Unless Henrik is seeing something else, a quick look at that code suggests that count == 0 for the last page of a file whose size is an exact multiple of the page size. md...@de... said: > It does, thank you. Now I have only the SIGTERM problem to deal with. Put a breakpoint on send_sig_info conditional on sig == 15 and get a backtrace. There shouldn't be too many SIGTERMs floating around the kernel, so that should provide a good clue. Jeff |
From: Henrik N. <hn...@ma...> - 2003-04-29 21:02:32
|
On Tue, 29 Apr 2003, Jeff Dike wrote: > Unless Henrik is seeing something else, a quick look at that code suggests > that count == 0 for the last page of a file whose size is an exact multiple > of the page size. Exacly. But as Matt I do not feel familiar enough with this area of the kernel to say this actually is the case without first making a test case, which I have not.. Regards Henrik |
From: Matt Z. <md...@de...> - 2003-04-29 21:43:54
|
On Tue, Apr 29, 2003 at 03:37:58PM -0400, Jeff Dike wrote: > md...@de... said: > > It does, thank you. Now I have only the SIGTERM problem to deal with. > > Put a breakpoint on send_sig_info conditional on sig == 15 and get a > backtrace. There shouldn't be too many SIGTERMs floating around the > kernel, so that should provide a good clue. Thanks for the quick start. It turns out it was just the OOM killer, and so entirely my fault for not giving it enough memory to do what I asked. -- - mdz |