#40 6.4.2 `format` wrote past the buffer it allocated

[Batch Job 1]

format wrote past the buffer it allocated

What happened. A field width or a precision above about 200 made format write past the end of the block it had just allocated.

(printout t "before" crlf)
(printout t "width [" (format nil "%1000s" "x") "]" crlf)
(printout t "after" crlf)

killed the process with Trace/BPT trap: 5. The stack named the next allocation, not the write:

  libsystem_malloc.dylib  mfm_alloc
  clips                   genalloc
  clips                   genrealloc
  clips                   ExpandStringWithChar
  clips                   ScanSymbol
  clips                   GetToken
  clips                   RouteCommand
  clips                   BatchStar

format had already returned; malloc aborted when the batch reader asked for memory and found the heap damaged. (format nil "%.500f" 1.0) did the same. Whether a given run aborts depends on what sits after the block: the same %1000s call inside a longer script returned 1000 characters and carried on.

Separately, (format nil "%2147483648s" "x") returned the empty string — the padding gone and the one-character argument gone with it.

Why it is a fix and not a preference. Section 12.4.6 of the Basic Programming Guide says of the field width that "if M is used, at least M characters will be output" and "if more than M characters are required to display the value, format expands the field as needed". For %f and %e it says the precision is "the number of digits following the decimal point". A conversion asked for 1000 characters and the buffer held 207, and a conversion asked for 500 decimal places into a buffer holding 206.

The empty string from %2147483648s fails the same sentence: at least 2147483648 characters were promised and none came out.

Where it came from. Two places, both in iofun.c.

PrintFormatFlag sized the buffer it passes to gensprintf, which is vsprintf, from the conversion string and the argument:

        theLength = strlen(formatString) + strlen(theResult.lexemeValue->contents) + 200;
        printBuffer = (char *) gm2(theEnv,(sizeof(char) * theLength));
        gensprintf(printBuffer,formatString,theResult.lexemeValue->contents);

and from the conversion string alone for the numeric conversions. Neither sum has the width in it, and neither has the precision. CreateConversionString rebuilds %1000s as six characters, so the allocation was 6 + 1 + 200 = 207 bytes and vsprintf wrote 1001. %.500f rebuilt to six characters gave 206 bytes for a 503-character result.

FindFormatFlag accumulated the width in an int:

      minWidth = (minWidth * 10) + (inchar - '0');

After the digits 214748364 that holds 214748364, and the next digit 8 asks for 2147483648, which is not an int. What minWidth held after that was negative, so CreateConversionString emitted a negative width — a left-justify flag with a nonsense width — and nothing came out.

The change. ConversionBufferLength in iofun.c returns the conversion string length, the argument length, the width, the precision and 400. All five sizings in PrintFormatFlag call it, after CreateConversionString, which is where the width gets its final value — for a %s conversion that function adds the difference between the argument's bytes and its characters, so a UTF-8 argument pads to the width in characters.

The 400 covers what neither the argument nor the width accounts for: the 309 digits DBL_MAX has before the point under %f, a sign, a point, an exponent and the terminator.

FindFormatFlag accumulates the width and the precision in size_t and stops at INT_MAX, which is the bound that matters because both reach vsprintf inside the conversion string, where a width and a precision are int. A digit string asking for more comes back as '-', the value ControlStringCheck already reports as an invalid flag, and it reports before any argument is converted. AppendDigitRun copies the digits the program wrote into the flag text, so the message names %2147483648s rather than the number the accumulation stopped at.

What was checked. Widths of 1000, 2000, 900, 700, 800, 1200 and 1500 on %s, %d, %x, %o, %c, %f and %e: each outputs exactly that many characters. The padded character lands at the right end under %1000s and at the left under %-1000s. A width of 300 on café outputs 300 characters and the argument comes out whole, so the character-versus-byte adjustment still happens. A width of two million runs, which puts the new bound on the type rather than on some smaller number.

Precisions: %.500f on 1.0 gives 502 characters starting 1.00, %.4000f gives 4002, %.500f on the largest double gives 810 — its 309 integral digits, the point and 500 places — %.400e gives 406, %.400g gives 3 because 1.5 needs three significant digits, and %.400d pads an integer to 400.

Widths and precisions past INT_MAX: %2147483648s, %.2147483648f, %2147483648d and a 26-digit width. All four are reported as invalid flags, the report names the text the program wrote, and none returns a length. A call after them still evaluates.

Then every example in section 12.4.6, which all use small widths that the old sizing happened to cover, plus %c on an integer, a precision on a UTF-8 %s, and %%. All unchanged.

The reason to care about this one: the abort lands in whatever allocates next, so the stack names a reader or a scanner and the format call is already off it.

Present in 7.0? Not checked.