RE: [PATCH] Suggested fix for ICU bug 4176 that works for Unix, Mac, and Windows newlines

[Gertzfield, B. <bge...@pa...>]

The patch did not introduce any new buffer overruns, but I did detect an =
existing one in u_fgets().  I have submitted another ICU bug 4271 with a =
patch; there's a buffer overrun if n > 512 and any long lines > 512 =
UChars are encountered.

=20

Ben

=20

________________________________

From: Gertzfield, Ben=20
Sent: Wednesday, December 01, 2004 2:35 PM
To: 'George Rhoten'
Cc: Freeland, Carl; Lasker, David; icu...@os...; =
Wang, June
Subject: RE: [PATCH] Suggested fix for ICU bug 4176 that works for Unix, =
Mac, and Windows newlines

=20

Thanks, I've submitted ICU jitterbug #4270.  I couldn't find a proper =
way to attach patches so I copied-and-pasted it, but I'm sure there are =
line-wrapping issues there.

=20

I'm currently testing the code more thoroughly to make sure there aren't =
any new buffer overruns introduced by this fix, however.

=20
Ben

=20

________________________________

From: George Rhoten [mailto:gr...@us...]=20
Sent: Wednesday, December 01, 2004 1:58 PM
To: Gertzfield, Ben
Cc: Freeland, Carl; Lasker, David; icu...@os...; =
Wang, June
Subject: Re: [PATCH] Suggested fix for ICU bug 4176 that works for Unix, =
Mac, and Windows newlines

=20


It looks like the main difference is the "currDelim =3D 1;"  around line =
522 for ICU 3.2 (when we hit the first newline possibility).  That looks =
like a good change.  Please submit your patch to our jitterbug database. =


http://www.jtcsv.com/cgibin/icu-bugs/=20

George Rhoten
IBM Globalization Center of Competency/ICU  San Jos=E9, CA, USA=20

Ben Gertzfield <bge...@pa...>=20
Sent by: icu...@ww...=20

12/01/2004 12:31 PM=20

To

"icu...@os..." =
<icu...@ww...>, George Rhoten/San =
Jose/IBM@IBMUS=20

cc

"Wang, June" <jw...@pa...>, Carl Freeland <cfr...@pa...>, =
"Lasker, David" <dl...@pa...>=20

Subject

[PATCH] Suggested fix for ICU bug 4176 that works for Unix, Mac, and =
Windows newlines

=20

=20

=20




Hi again all,

I think the correct fix to allow both non-combined newlines (\n or \r) =
and
combined newlines (\r\n) to work is just to set currDelim to 1 in the =
case
where CAN_HAVE_COMBINED_STRING_DELIMITER(*alias) returns false.

I've attached a patch to implement this change against ICU 3.0's =
ustdio.c
(pristine -- this patch also includes the changes that were part of the
original fix for 4176).  Please see the attached file,
ustdio.c_buffer_fix_v2.diff.txt.

Here's my test program, extended to try Unix, Mac, and Windows newlines.
They all work now on 512 byte buffer boundaries, when I apply the =
attached
patch.  (The output of the -m argument looks a little funny since my
terminal obeys carriage return as "return to start of current line.)

#include <stdio.h>
#include <unicode/ustdio.h>

const char *windows_linefeed =3D "\x0d\x0a";
const char *mac_linefeed =3D "\x0d";
const char *unix_linefeed =3D "\x0a";

void write_test_file(const char *filename, int buffer_size, const char
*linefeed)
{
   char filler[buffer_size];
   FILE *file;
   int linefeed_size;

   linefeed_size =3D strlen(linefeed);

   file =3D fopen(filename, "w");
   if (file =3D=3D NULL)
   {
       fprintf(stderr, "Could not open %s\n", filename);
       exit(1);
   }


   memset(filler, 'X', buffer_size - linefeed_size);
   filler[buffer_size - linefeed_size] =3D '\0';

   // Write buffer_size "X"s and a linefeed twice
   fprintf(file, "%s%s", filler, linefeed);
   fprintf(file, "%s%s", filler, linefeed);

   fclose(file);
}

void read_test_file(const char *filename, UChar *dest, int max_len)
{
   UFILE *file =3D u_fopen(filename, "r", "en_US", "us-ascii");
   if (file =3D=3D NULL)
   {
       fprintf(stderr, "Could not open %s\n", filename);
       exit(1);
   }

   u_fgets(dest, max_len, file);

   u_fclose(file);
}

int main(int argc, char **argv)
{
   UChar buf[2000];
   UFILE *u_stdout;
   const char *linefeed;

   if (argc < 3)
   {
       fprintf(stderr, "Usage: %s [-w|-m] filename bufsize\n\n", =
argv[0]);
       exit(1);
   }

   linefeed =3D unix_linefeed;
   if (strcmp(argv[1], "-w") =3D=3D 0)
   {
       linefeed =3D windows_linefeed;
       argv++;
   }
   else if (strcmp(argv[1], "-m") =3D=3D 0)
   {
       linefeed =3D mac_linefeed;
       argv++;
   }

   write_test_file(argv[1], atoi(argv[2]), linefeed);
   read_test_file(argv[1], buf, 2000);

   u_stdout =3D u_finit(stdout, "en_US", "us-ascii");

   u_fprintf(u_stdout, "The following should only have one newline "
             "between the brackets: [%S]\n", buf);
}


[bgertzfield@hyper2:~/tmp]% ./test test.txt 512
The following should only have one newline between the brackets:
[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
]
[bgertzfield@hyper2:~/tmp]% ./test -m test.txt 512
The following should only have one newline between the brackets:
[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]XXXXXXXXXXXXXXXX
[bgertzfield@hyper2:~/tmp]% ./test -w test.txt 512
The following should only have one newline between the brackets:
[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
XXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
]

[attachment "ustdio.c_buffer_fix_v2.diff.txt" deleted by George =
Rhoten/San Jose/IBM]=20