#34 dfu-util requires -a X when flashing .dfu (DfuSE) files

[Roger Wolff]

I thought that the DFU file format was meant to tell the flashing utility stuff like "the ALT port to use" and "The address to flash to".

So I created a DFU file, told it to use address 0x08000000, and when I dump it it says; "ALT 0". But when I try to program it, it sees the ALT0 (flash) and ALT1 (option bytes) as separate devices and requires me to sepecify -a 0 on the commandline. IMHO that is defeating the purpose of the DFU file format.

[Tormod Volden]

FYI, dfuse-pack.py can now generate files with multiple alternate interface numbers (thus in separate "DFU images" in the DFuSE file).

[Hendry Kaak]

Nice! That's some quick progress. This makes testing a lot easier, thanks.

[Matthijs Kooijman]

Hi, coworker from Hendry here, he asked me to chip in as well.

I haven't looked at the patches in detail and haven't followed all of the previous discussion, but I do have some thoughts about how the CLI interface for this should look. AFAIK, without these patches, you have to always explicitly specify the alternate setting to be used (implemented from the .dfu perspective where a file does not contain alternate setting info and contains just one image). With dfuse files, you can have multiple images and each is annotated using an alternate settings. I can imagine three cases:
- You specify e.g -a 0 along with a dfuse file. This causes just the image with alt=0 to be flashed, ignoring any others. This is current behavior. If the dfuse file does not have an alt=0 file, nothing is flashed and it errors out.
- You specify e.g. -a 0 -a 1 along with a dfuse file. This causes the images with alt=0 and alt=1 to be flashed. If the dfuse file does not contain both alt=0 and alt=1, no flashing happens and it errors out.
- You specifiy -a all along with a dfuse file. This causes all images from the file to be flashed to their specified alt interface. This makes it explicit that you want to flash everything, rather than defaulting to flashing everything when no -a is passed (which reduces the chances of accidentally flashing things you did not want to flash because you forgot to specify -a).
- You specify e.g. -a 0 -a 1 or -a all along with a dfu file. This makes no sense and errors out.

If your image elements are listed with alternate interface numbers in e.g. the order @1 @1 @2 it will make a DfuSe file with two DFU images, the first having two image elements. If the order is @1 @2 @1 you will get three DFU images.

I'm not sure what it would mean to have the same alt setting specified multiple times in the same file. One could just use either the first or the last one, but that seems arbitrary, so I think it would be better if dfu-util would just outright reject these files entirely (or maybe only when the duplicate alt setting is actually requested).

[Tormod Volden]

Hi Matthijs, thanks for your thoughts on this.

"specify e.g -a 0 along with a dfuse file" - Yes, we'd like to support this, for backwards compatibility and to have a way for expert users to try out dfuse files.

I don't see a pressing need to allow specifying a subset of alternate interface. DfuSe files is a vehicle for firmware packagers (who know what they do) to provide all-in-one files to end users (who might not know what they are doing). It is not for shipping a catalogue of firmware parts that the user should pick and choose from.

For an "-a all" option, this should be the default. Users expect that the DfuSe file is safe and complete, and the firmware packager assumes that the user will apply it wholesale. There could be cases where applying by part would brick a device (from a user point of view). I assume other tools like ST's also apply all parts of a DfuSe file without questions.

"the same alt setting specified multiple times in the same file" - this is the DfuSe concept of multiple DFU image elements inside one DFU image (= alt setting) where you have for instance an IAP bootloader at 0x08000000, an application at 0x08003000 and some default application flash settings at 0x08001e000 and you don't want to have this is in one large, padded element. There could even be intermediate flash banks that you don't want to have erased so you can't pad everything in between.

[Matthijs Kooijman]

I don't see a pressing need to allow specifying a subset of alternate interface. DfuSe files is a vehicle for firmware packagers (who know what they do) to provide all-in-one files to end users (who might not know what they are doing). It is not for shipping a catalogue of firmware parts that the user should pick and choose from.

Yeah, I guess that makes sense. And if you support selecting a single alt, then you can just call dfu-util multiple times to select multiple alts (but not all).

For an "-a all" option, this should be the default. Users expect that the DfuSe file is safe and complete, and the firmware packager assumes that the user will apply it wholesale. There could be cases where applying by part would brick a device (from a user point of view). I assume other tools like ST's also apply all parts of a DfuSe file without questions.

My inuition would say otherwise, but I'm not too familiar with the the ecosystem around this yet, and I suppose you are :-) The bricking argument is a fair one, as well.

"the same alt setting specified multiple times in the same file" - this is the DfuSe concept of multiple DFU image elements inside one DFU image (= alt setting) where you have for instance an IAP bootloader at 0x08000000, an application at 0x08003000 and some default application flash settings at 0x08001e000 and you don't want to have this is in one large, padded element. There could even be intermediate flash banks that you don't want to have erased so you can't pad everything in between.

Yeah, Hendry pointed at that an image can have multiple elements, which should indeed just be flashed together. However, I was referring to a slightly different case: When there are multiple images with the same alt setting (so not one image with multiple elements), which the file format obviously makes possible, and AFAIU, the packer tool also allows creating such files. However, I cannot think of a usecase for such files: If you want both the images to be flashed together, then just include them in a single image? Lacking clear semantics about how to interpret these files, I would be inclined to simply reject these files (and probably also refuse creating them them in the packer).

Or could the usecase for such files be that you need to flash the same thing twice? E.g. set certain option bytes, then write to flash, that set certain option bytes back? Maybe such a thing could be useful to remove and re-apply locks in the option bytes or so?

[Tormod Volden]

I have now pushed the changes to git.

One way to test it out is to pack a DfuSe file from multiple binary files with several alternate interfaces, and for the non-flash targets, use a RAM address (one that doesn't clash with bootloader operation!):

./dfuse-pack.py \
    -b 0x08001000@0:/tmp/at4k.bin \
    -b 0x08002000@0:/tmp/at8k2k.bin \
    -b 0x20004000@1:/tmp/at4k.bin \
    -b 0x20004400@2:/tmp/at4k.bin \
    -b 0x20004800@2:/tmp/at4k.bin \
    ../testalt.dfu

The resulting DfuSe file can be verified with:
./dfuse-pack.py ../testalt.dfu
Now use the force modifier to ignore the "unknown" RAM addresses since they are not declared by the bootloader. To program all elements of the file:
src/dfu-util -D ../testalt.dfu -s :force

Like before, to only program elements belonging to one target:
src/dfu-util -D ../testalt.dfu -s :force -a 1

BTW, Hendry mentioned seeing a libusb memory leak, and I can confirm these when enabling libusb debug output:
libusb: warning [libusbexit] device 2.4 still referenced
However they are not due to these changes.

Matthijs, I would leave the user and bootloader all possibilities for programming elements in arbitrary order with possible side-effects and not add artificial limitations.

Note also that the current implementation leaves it to the DfuSe file packager to make sure one programmed element doesn't get erased by a subsequent element in the same flash erase page.

[Matthijs Kooijman]

Cool!

Matthijs, I would leave the user and bootloader all possibilities for programming elements in arbitrary order with possible side-effects and not add artificial limitations.
Note also that the current implementation leaves it to the DfuSe file packager to make sure one programmed element doesn't get erased by a subsequent element in the same flash erase page.

Sounds reasonable.

[Hendry Kaak]

I have now pushed the changes to git.

One way to test it out is to pack a DfuSe file from multiple binary files with several alternate interfaces, and for the non-flash targets, use a RAM address (one that doesn't clash with bootloader operation!):

./dfuse-pack.py \
-b 0x08001000@0:/tmp/at4k.bin \
-b 0x08002000@0:/tmp/at8k2k.bin \
-b 0x20004000@1:/tmp/at4k.bin \
-b 0x20004400@2:/tmp/at4k.bin \
-b 0x20004800@2:/tmp/at4k.bin \
../testalt.dfu
The resulting DfuSe file can be verified with:
./dfuse-pack.py ../testalt.dfu
Now use the force modifier to ignore the "unknown" RAM addresses since they are not declared by the bootloader. To program all elements of the file:
src/dfu-util -D ../testalt.dfu -s :force

Like before, to only program elements belonging to one target:
src/dfu-util -D ../testalt.dfu -s :force -a 1

Nice @Tormod, I did a quick test to see if I could write multiple images to alt 0 and that seems to be working nicely, great work! :-)