Share

flex: the fast lexical analyser

Tracker: Bugs

5 Broken output on ia64 when input is piped in - ID: 2782996
Last Update: Attachment added ( srivasta )

On ia64, flex appears to generate a completely different (and broken)
output depending on whether the input file is specified using shell
redirection or via a pipe.

This was traced back commit 3971e1a917548977cff71418a7c3575ffbc9571f,
by Alex Nixon <alex.nixon@citrix.com>:
VFS: increase pseudo-filesystem block size to PAGE_SIZE.
PAGE_SIZE on ia64 kernels is larger than most architectures - 16K is
what most distributions (including Debian) use. An effect of the above
changeset, as verified by strace, is that the read() and write()
system calls issued by flex on a pipe will use this bigger
size. fread() appears to be properly hiding the actual read size and
is returning 8192 byte buffers to flex upon request. I verified by
this by instrumenting YY_INPUT to compare the fread data with the
original input file.

Its probably useful to note that the corruption happening here is that
the last 382 lines of the input file is being prefixed to the output
file. The only differences in the remainder of the file are the #line
numbers. I've also found that bumping YY_READ_BUF_SIZE up to 16384
seems to workaround the problem - but that may just be putting it back
into hiding. Note that increasing the buffer size on other
architectures cause problems in corner cases not caught by the test
suite. So it is better to not increase the buffer size unilatereally
for all architectures, but to just increase it for the architecture it
is required on, namely, IA64, and leave it unchanged on other
architectures.

The tradeoff here is between correct operation, by makeing flex's
buffer at least as big as the system buffer -- and not bigger, so that
we do not waste memory. At this point, fixing the isue seems more
important than worrying that flex consumes and additional 8KB of
memory.

Also, please note that just increasing YY_READ_BUF_SIZE is not enough,
YY_BUF_SIZE has to be increased as well. In the general case, the
latter is twice the former. Therefore set it to the same ratio in the
ia64 case.

Patch attached.

manoj


Manoj Srivastava ( srivasta ) - 2009-04-28 16:45

5

Open

None

Nobody/Anonymous

Compiling flex

2.5.35

Public


Comments




Log in to comment.

No follow-up comments have been posted.

Attached File ( 1 )

Filename Description Download
flex-bufsize-fix.patch Fix for bufsize increase on IA64 Download

Change ( 1 )

Field Old Value Date By
File Added 324794: flex-bufsize-fix.patch 2009-04-28 16:45 srivasta