#18 Adding dfu-suffix to file results in suffix data programmed to flash...

[digitall]

Using dfu-util v0.8 on Gentoo Linux programming an STM32F4, I have been seeing odd issues resulting in data corruption.

I have finally tracked down the issue. Due to the "dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!" I used the dfu-suffix tool to add a suffix to my binary files.

This programs fine, but if I then upload the resulting flash, the suffix data has been errorneously programmed into the flash memory after the expected binary data.

While not critical for most applications (the extra "spurious" data does not cause issues as unused), my application skips a Flash page which is used to hold configuration data i.e. it uses two dfu files or a dfu-pack.py file to flash the first region, skips a page and then the rest of the flash ... This is actually fairly common in real world systems which use Flash as Configuration and ST have a technical note on doing this. ... Unfortunately due to the suffix overprogramming, this region is corrupted by the first flash resulting in application malfunction.

I have checked the commits since v0.8 to v0.9 and I don't think this has been fixed, but it should be fairly easy to prove i.e. create a file with suffix, download and then upload full flash region and check for the suffix data after the expected...

For now, I have worked around this by dropping the suffix and using the values passed to dfuse-pack.py instead... but this should be fixed if the suffix is going to be mandatory in future!

[Tormod Volden]

I am a bit surprised because the dfuse code should remove any suffix on the files. I am a bit confused as well since you say you are using dfu-pack.py. Do you see the issue both with (plain) binary files and with .dfu files?

Do you add a suffix to a binary before you pack it into .dfu? If you use a suffix for dfuse you must add it to the finished .dfu file.

[Anonymous]

Ah, I think I am seeing the issue.

I was originally using bare binary files for this i.e. objcopy to get the elf to a binary file with gap fill of 0xff (unprogrammed) and then dd to cut this to create two images for the pages.

Due to the "dfu-util: Invalid DFU suffix signature dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!" message, I used dfu-suffix on these two files to remove the message by adding suffix and thus removing the need to specify the USB VID/PID etc. on dfu-util command line... but we still needed two invocations of dfu-util to program the device at different base addresses.

However, I then used dfu-pack.py on these two files to pack them together into a single dfu with the correct base addresses for each image... This did not cause any errors to be emitted from dfu-pack.py and no error from dfu-util when I flashed with this and no need to add suffix as the system seemed to read the suffix fine (probably from the second combined file?) ... So no error message, but silent "overprogramming" outside the region leading to subtle flash corruption as I was doing these steps now in the wrong order. Not good.

I suggest the following:
1. improve user documentation i.e. make it clear in README of expected workflow for packing raw binary file into dfu and then adding suffix.
2. add input validation checks to dfu-pack.py to abort if any of the input files already contains a suffix. Either indicate that this should be removed and re-run, or automatically do this within the tool i.e. make the pack deal with the suffix as well since there is overlap in the PID/VID? and add the suffix to the completed file?

[digitall]

Damn forgot to login to SF.net ... See the message trapped in moderation.

[Tormod Volden]

Good ideas, can you suggest a text? I need to look at this again, but a DfuSe file must have a suffix so I guess dfu-pack.py always adds one? I think we can require bare binary files as input to dfu-pack.py otherwise we'll have a parallel implementation of suffix decoding in there.

[digitall]

Hmm.. Text. Something as follows:
"Please note that the DFU programming tool, dfu-util supports programming with raw binary files as produced by objcopy ie. arm-none-eabi-objcopy --gap-fill=0xff -O binary image.elf output_image.bin.

However, programming with a raw binary file will require the USB VID/PID and other parameters to be specified on the dfu-util command line. See dfu-util --help for details.

To avoid this, the binary file should have a DFU prefix and suffix added to create a DFU file. This will likely be mandatory in a future release to avoid programming mistakes. To add these, the dfu-prefix and dfu-suffix tools should be used to add these. Since these modify the file in place, it is recommended to first copy the binary file to a new name ending in .dfu i.e. cp output_image.bin output_image.dfu && dfu-prefix -a output_image.dfu && dfu-suffix -a output_image.dfu -p <PID> -v <vid> <other parameters="">
It is recommend to consult dfu-prefix --help and dfu-suffix-help for details"

[digitall]

"This removes the need to specify most parameters to the dfu-util programming tool on command line as the DFU prefix and suffix will contain these parameters.

Further to this, ST support a variant of the DFU file format called DfuSe which allows multiple binary images with different load addresses to be merged into a single DFU programming file. To support this, the DfuSe packing tool, dfuse-pack.py is used. This will require a installed Python interpreter to run.Run dfuse-pack.py with no parameters for details on usage. This accepts raw binary files as input with the load addresses and USB VID/PID specified on the command line. You should not use any files which have had suffixes added. These can be removed to restore these to raw binary form using the dfu-suffix and dfu-prefix tools with the -D (delete) option, though the -c (check) option should be used first to note down the required load addresses to be passed to dfuse-pack.py. Failure to do this may result in the dfu-suffix data being interpreted as programming data and being programmed after the expected range in the Flash memory of the target device."

[digitall]

All done. I suggest fixing this in code as well in the Python, by at least having validation of the input files to ensure they are "raw"... I have checked the Python code and this does not call dfu-prefix or dfu-suffix as far as I can see, though it may do the same job with it's own code... I am slightly unsure, but I think it may be adding the suffix in the python code, though it may read the prefix if present... and I am needing to specify the --intf 0 --alt 0 options to dfu-util still as it can not accept these on the command line as data for suffix... It would be good to sort this so that dfu-util purely takes the file as the only parameter for optimal "user-proofing" :)

[Tormod Volden]

I have added a note in the dfuse-pack.py usage text, and also added a check for any DFU suffix in the input binary files. We could probably need a man page for dfuse-pack.py...

[digitall]

Thanks for sorting that out. I would tend to agree on a man page, even a simple one would be fine. Let me know if you want any help writing it which is what I assume the ellipsis indicated :)

[Tormod Volden]

Yes, the ellipsis carried some hope it would be done by someone :) I would appreciate any contribution.

We have none for dfu-suffix and dfu-prefix either.