#303 Bad chars in COFF file

[Patrick Hayes]

When building with SDCC, v1.5.x outputs non-printable chars in the COFF file which causes MPLABX to fail loading the resulting binary. See examples below, extracted with gpov:
v1.4.x - ok

---

COFF File and Optional Headers
COFF version         0x1240: MICROCHIP_MAGIC_v2
Processor Type       16f887
Time Stamp           05-05-2017 21:59:13
Number of Sections   zu
Number of Symbols    308
Characteristics      0x2
  File is executable.

Section Header
Name                    sharebank
Physical address        0x70
Virtual address         0x70
Size of Section         16
Number of Relocations   zu
Number of Line Numbers  zu
Flags                   0x5080
  Uninitialized data.
  Absolute.
  Overlaid with other sections from different objects modules.

---

v.1.5.0-1 - ERROR in the first section name

COFF File and Optional Headers
COFF version         0x1240: MICROCHIP_MAGIC_v2
Processor Type       16f887
Time Stamp           05-05-2017 22:02:31
Number of Sections   zu
Number of Symbols    233
Characteristics      0x2
  File is executable.

Section Header
Name                    U
Physical address        0x70
Virtual address         0x70
Size of Section         16
Number of Relocations   zu
Number of Line Numbers  zu
Flags                   0x5080
  Uninitialized data.
  Absolute.
  Overlaid with other sections from different objects modules.

---

build v1.5.1.1301 - ERROR - first section header

COFF File and Optional Headers
COFF version         0x1240: MICROCHIP_MAGIC_v2
Processor Type       16f887
Time Stamp           05-05-2017 22:24:29
Number of Sections   zu
Number of Symbols    233
Characteristics      0x2
  File is executable.

Section Header
Name                    

Physical address        0x70
Virtual address         0x70
Size of Section         16
Number of Relocations   zu
Number of Line Numbers  zu
Flags                   0x5080
  Uninitialized data.
  Absolute.
  Overlaid with other sections from different objects modules.

Also the number of symbols is rather reduced in v1.5.x, maybe due to linker optimizations?

[Molnár Károly]

gpov --> gpvo

Which operating system did the compile of gputils? What was the compiler? Clang, gcc or something else? For something is wrong:

"Number of Relocations zu"

This is in the source code:

printf("Number of Relocations %zu\n", Section->relocation_list.num_nodes);

It seems that the C compiler does not know the "%zu" formatting string.

Also the number of symbols is rather reduced in v1.5.x, maybe due to linker optimizations?

This may be due to this: gplink String Table Optimisation

Molnár Károly

[Patrick Hayes]

I am just using the prebuilt binaries from this site - not homebuilt.
My test environment is Win10 x64 and probuilt SDCC v3.6.0

[Molnár Károly]

Probably only windows binary is wrong because no such bugs have been reported so far. I've made an improved version, but I'm not sure it's all right. [r1303]

gputils-20170507-1303-setup.exe

[Patrick Hayes]

Thanks for the fast reply and fix.
Actually, there are some issues, they have just not been reported here, or are not obvious to the end users.

Like this www.microchip.com/forums/m987524.aspx

Just tried the 1303 build. Now i get an failed assertion. See below.

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
Assertion failed!

Program: C:\PROGRA~2\gputils\bin\gplink.exe
File: gpcod.c, Line 63

Expression: length < (int)Pascal_max_size

Thanks in advanced

[Molnár Károly]

Interesting, if I runs on Linux, then there is nothing wrong with it. I still have to work with it.

Károly

[Patrick Hayes]

Let me know if you need anything.

Patrick

[Molnár Károly]

Thank you for the good intent, but I can solve the testing. :-) I just need a little time.

Károly

[Patrick Hayes]

ok, just took a look at the assert.
your commit 1302->1303 in gpcod.c:

+static const char* m_day_names[] = {
+  "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
+};

-  length = strftime(temp, sizeof(temp), "%d%b%g", localtime(&now));
-  assert(length < Pascal_max_size);
+  t = localtime(&now);
+  length = snprintf(temp, sizeof(temp),
+                    "%02d%s%02d\n", t->tm_mday, m_day_names[t->tm_mon], (t->tm_year + 1900) % 100);
+  assert(length < (int)Pascal_max_size);

the added sbprintf format string will always be 8 chars long - same as Pascal_max_size in the following assert that it should be shorter than.

[Patrick Hayes]

If I remove the trailing \n in the format string, the assert goes away.
Now it just produces an invalid? coff file. MPLABX can't load it.

My best guess is empty sections with uninitialized names (shows up with random chars/values)

[Molnár Károly]

If I remove the trailing \n in the format string, the assert goes away.

I was forgetful and left it there the \n character. I've fixed this in the [r1304] version.

[Patrick Hayes]

Great, thanks. Assert fixed.
But the COFF file is still broken. MPLABX will not load it.
Some section names contains random chars. Only for uninitialized/empty sections?

[Molnár Károly]

I need an example which generates the error.

[Patrick Hayes]

See attached zipfile

The asm file is generated by sdcc.

commands:

gpasm -o main.o -c main.asm
gplink -c -m -w -r -o main.hex main.o libsdcc.lib pic16f887.lib

Output from mplabx using sdcc 3.6.0 and gputils1.5.x

CLEAN SUCCESSFUL (total time: 63ms)
make -f nbproject/Makefile-default.mk SUBPROJECTS= .build-conf
make[1]: Entering directory 'D:/Projekter/MPLABX/RobotBMW/sdcctest.X'
make  -f nbproject/Makefile-default.mk dist/default/production/sdcctest.X.production.hex
make[2]: Entering directory 'D:/Projekter/MPLABX/RobotBMW/sdcctest.X'
gnumkdir -p "build/default/production" 
rm -f  build/default/production/main.o 
"C:\Program Files\SDCC\bin\SDCC.exe" --use-non-free -c -mpic14 -p16f887 main.c  -obuild/default/production/main.o
gnumkdir -p dist/default/production 
"C:\Program Files\SDCC\bin\SDCC.exe" -Wl-c -Wl-m --use-non-free -mpic14 -p16f887 build/default/production/main.o -odist/default/production/sdcctest.X.production.hex 
message: Using default linker script "C:\Program Files (x86)\gputils\lkr\16f887_g.lkr".
warning: Relocation symbol "_cinit" [0x0000] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x0004] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x001E] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x0022] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x0000] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x0004] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x001E] has no section. (pass 0)
warning: Relocation symbol "_cinit" [0x0022] has no section. (pass 0)
make[2]: Leaving directory 'D:/Projekter/MPLABX/RobotBMW/sdcctest.X'
make[1]: Leaving directory 'D:/Projekter/MPLABX/RobotBMW/sdcctest.X'

BUILD SUCCESSFUL (total time: 390ms)
The program file could not be loaded: 4

Hope it helps

[Patrick Hayes]

Just found this comment in a MPLABX SDK file called "CoffSectionHeader.java":

• The section name is either a string or a reference into the string table.
• Strings of fewer than seven characters are stored directly, and all others
• are stored in the string table. If the first four characters of the string
• are 0, then the last four bytes are assumed to be an offset into the
• string table. This is a bit nasty as it is not strictly conforming to the
• ANSI specification (i.e., type munging is undefined behaviour by the
• standard), but it's effective and it maintains binary compatibility with
• the System V layout, which other options would not do. This implementation
• has the advantage of mirroring the standard System V structure used for
• long symbol names.

Could that explain the problem?

[Molnár Károly]

Strings of fewer than seven characters are stored directly, and all others are stored in the string table. If the first four characters of the string are 0, then the last four bytes are assumed to be an offset into the string table.

From the libgputils/gpcoff.h file:

#define COFF_SSYMBOL_NAME_MAX   8

This is not seven, but eight. So far nobody was troubled this difference, at least nobody indicated that it would cause a mistake. In turn it has been so for years.

From the libgputils/gpreadobj.c file:

static char *
_read_symbol_name(const uint8_t *File, const char *String_table, const gp_binary_t *Data)
{
  char     buffer[COFF_SSYMBOL_NAME_MAX + 1];
  uint32_t string_offset;

  /*   's_zeros'  -- first four characters are 0 */
  if (_check_getl32(&File[0], Data) == 0) {
    /* Long name, read this from the string table. */
    string_offset = _check_getl32(&File[4], Data);
    /* 's_offset' -- pointer to the string table */
    return GP_Strdup(&String_table[string_offset]);
  }

  /*   'name'     -- symbol name if less than 8 characters */
  memcpy(buffer, &File[0], COFF_SSYMBOL_NAME_MAX);
  /* The name can occupy all 8 chars without a null terminator. */
  buffer[COFF_SSYMBOL_NAME_MAX] = '\0';
  return GP_Strdup(buffer);
}

From the libgputils/gpwriteobj.c file:

static void
_add_name(const char *Name, uint8_t *Table, FILE *Fp)
{
  uint32_t length;
  uint32_t offset;

  if (Name == NULL) {
    return;
  }

  length = (uint32_t)strlen(Name);

  if (length <= COFF_SSYMBOL_NAME_MAX) {
    /* The string will fit in the structure. */
    if (length > 0) {
      /* 'name' */
      gp_fputvar(Name, length, Fp);     /* symbol name if less then 8 characters */
    }

    if (length < COFF_SSYMBOL_NAME_MAX) {
      gp_fputzero(COFF_SSYMBOL_NAME_MAX - length, Fp);
    }
  }
  else {
    offset = _add_string(Name, Table);

    /* write zeros and offset */
    /* 's_zeros' */
    gp_fputl32(0, Fp);                  /* first four characters are 0 */
    /* 's_offset' */
    gp_fputl32(offset, Fp);             /* pointer to the string table */
  }
}

I ran the following commands:

gpvo main.o > main.o.dump
wine gpvo.exe main.o > main.o.txt
diff -up --strip-trailing-cr main.o.txt main.o.dump > main.o.dump.diff

Content of the main.o.dump.diff file:

--- main.o.txt  2017-05-17 19:03:35.475280735 +0200
+++ main.o.dump 2017-05-17 19:00:05.103510392 +0200
@@ -1,7 +1,7 @@
 COFF File and Optional Headers
 COFF version         0x1240: MICROCHIP_MAGIC_v2
 Processor Type       16f887
-Time Stamp           2017. 05. 16. 21:55:04
+Time Stamp           2017. máj. 16., kedd, 21.55.04 CEST
 Number of Sections   6
 Number of Symbols    164
 Characteristics      0

It's almost the same.

Which program was produced the main.txt file? It is not like the output of the gpvo.exe program.

[Anonymous]

I ran the command
gpvo.exe main.cof > main.txt

This is the COFF file produced by the linker - required by MPLABX in order to show memory usage in non-debug builds. The output from the assembler (main.o) looks just fine, but the output from the linker (main.cof) looks strange.

[Molnár Károly]

It turned out now that we both was thinking about other things. :-( The Microchip and gputils treat slightly differently the coff format. Therefore the gpvo is not capable of correctly processed by Microchip's coff files. The same is true vice versa also. I assume that the format was once uniform, but Microchip changed something. It seems that few and rarely use together the mplabx and gputils, because so far there was no such complaint.

[Patrick Hayes]

I traced the coff-file problem to first appear in build 1219 (between 1217 and 1219). One of the things changed, apart from a lot of renaming, is massive changes to the libgputils\gpcoffgen.c and related files. Massive changes, probably related to the introduction of the -b flag on gplink.

[Molnár Károly]

I'm looking for the bug, but lately I have little time. I temporarily put a printf call on the scanning process. Then I did a debug list with the following command line:

gpvo main.cof > main.cof.dump-1.5.2

So the list starts:

gpreadobj.c:_read_symbol_name() -- string_offset: 4, string: Ï.
gpreadobj.c:_read_symbol_name() -- string_offset: 14, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1690, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1707, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1722, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 26, string: AT.
gpreadobj.c:_read_symbol_name() -- string_offset: 36, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1739, string: ..
gpreadobj.c:_read_symbol_name() -- string_offset: 49, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 69, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 81, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1838, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 105, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 116, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1851, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1873, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1909, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1920, string: _EECON2
gpreadobj.c:_read_symbol_name() -- string_offset: 1956, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 1992, string: þÿ
gpreadobj.c:_read_symbol_name() -- string_offset: 2028, string: ebank
gpreadobj.c:_read_symbol_name() -- string_offset: 2039, string: errupt
gpreadobj.c:_read_symbol_name() -- string_offset: 2075, string: g_2007_MAIN.O
gpreadobj.c:_read_symbol_name() -- string_offset: 2086, string: .O
gpreadobj.c:_read_symbol_name() -- string_offset: 2377, string: ic16f887_9
gpreadobj.c:_read_symbol_name() -- string_offset: 2388, string: UD_abs_pic16f887_a
gpreadobj.c:_read_symbol_name() -- string_offset: 2402, string: 87_a
gpreadobj.c:_read_symbol_name() -- string_offset: 2507, string: s_pic16f887_10
gpreadobj.c:_read_symbol_name() -- string_offset: 2517, string: 7_10
gpreadobj.c:_read_symbol_name() -- string_offset: 2528, string: _pic16f887_11
gpreadobj.c:_read_symbol_name() -- string_offset: 2542, string: UD_abs_pic16f887_12
gpreadobj.c:_read_symbol_name() -- string_offset: 146, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 2581, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 162, string:
gpreadobj.c:_read_symbol_name() -- string_offset: 178, string: .
gpreadobj.c:_read_symbol_name() -- string_offset: 197, string:
.
.
.

It appears that main.cof stores the strings differently as the main.o file. The internal addresses point to different location.

[Anonymous]

Yes, something happend to the coff output from the linker. v1.4.x works fine, so MPLABX users (all 5 of us) will most likely just use that version for now.

[Molnár Károly]

I looked also with a very old (0.13.7) gpvo version:

gpvo main.cof > main.cof.dump-0.13.7

.
.
.
Section Header
Name                    Ď
Address                 0x70
Size of Section         16
.
.
.
Section Header
Name                    AT”
Address                 0xf8
Size of Section         0
.
.
.
Section Header
Name                    >
Address                 0xf7
Size of Section         2
Number of Relocations   0
.
.
.

It behaves also the same as the present versions.

[Patrick Hayes]

Yes, looks wrong. But maybe the problem is not with main.o, but with one of the libraries that gets linked in. (libsdcc.lib or pic16f887.lib)? What do they look like inside?

Btw, i have been working a bit on a MPLABX plugin for gputils, so gpasm can be selected as a build tool, just like mpasm or any other compiler. Is it of any interest?

It can be found on https://github.com/mc6pac/toolchainGPUTILS

It is in a very rough preliminary version, but somewhat functional. No binaries yet.

[Molnár Károly]

This error should be search in libgputils.

It can be found on https://github.com/mc6pac/toolchainGPUTILS

I encourage you to continue the development.

[Molnár Károly]

This is a bugfix:

gputils-src-20170702-1309.tar.bz2
gputils-20170702-1309-setup.exe

Please try it out.

Károly