#390 Add new FMF V2 format for easier evolution

[Fredrick Meunier]

One of the outstanding tasks for landing the ULA+ branch to Fuse is to get support for ULA+ modes added to it. The existing format is a bespoke binary format with no built-in support for format evolution, and so rather than spend effort working on that I'd rather move our format to a more widely-supported binary encoding scheme that supports evolving the format over time and provides support for writing the disk formatted data.

I've had a survey of some solutions available online and have settled on using Cap'n Proto which has a GPL v2-friendly MIT license, supports many language implementations (obviously including a C implementation) and pleasingly doesn't require deserialisation to read the saved structures.

My initial implementation is planned to follow this structure:

FMF V2 Files

This only has the same features as the existing FMF format, ULA+ features will be evolved in unless we can release both in one Fuse release.

Files will use the .fm2 extension.

The format will be composed of tagged blocks similar to the SZX format, each block is preceded by a header of two fields:

Field	Type	Notes
Id	char(4)	Indicates which type of block follows this header. It is a 4 byte sequence compressed into a dword. e.g. the FMF header block has a block id of 'F', 'M', 'F', 'H'.
Size	libspectrum_dword	Little-endian format. Specifies the size of the block following this header. It does not include the size of this header.

Cap’n Proto Schema

Shared Data Types

These structures are shared between the two block types:

struct SoundDetails {
  soundEncoding @0 :SoundEncoding;
  freq @1 :UInt32;
# Sound freq in Hz

  soundType @2 :SoundType;

  enum SoundEncoding {
    pcm @0;
    ulaw @1;
    alaw @2;
  }

  enum SoundType {
    stereo @0;
    mono @1;
  }
}

struct FrameDetails {
  frameSkip @0 :UInt8;
# Frame skip ( 1:# )

  frameFreq @1 :UInt32;
# Frame rate in mHz

  screenType @2 :ScreenType;

  enum ScreenType {
    standard @0;
    hires @1;
    hicolor @2;
    standardTimex @3;
# Double canvas size of standard screen
  }
}

FMF Header Block

Id: FMFH

Schema:

struct FMFHeader {
  compression @0 :CompressionType;
  frameDetails @1 :FrameDetails;
  soundDetails @2 :SoundDetails;

  enum CompressionType {
    uncompressed @0;
    zlib @1;
  }
}

FMF Data Block

Id: DATA

Schema:

struct FrameData {
  union {
    newFrame @0 :FrameDetails;
    screenChunk @1 :ScreenArea;
    soundChunk @2 :SoundChunk;
  }

  struct ScreenArea {
    x @0 :UInt8;
    y @1 :UInt8;
    width @2 :UInt8;
    height @3 :UInt8;
    data @4 :BitmapData;

    struct BitmapData {
      bitmap1 @0 :Data;
# bitmap1

      bitmap2 :union {
        lores @1 :Void;
        bitmap @2 :Data;
# bitmap2
      }

      attributes @3 :Data;
# attrib

    }
  }

  struct SoundChunk {
    soundDetails @0:SoundDetails;

    numberOfFrames @1 :UInt16;
# number of frames of sound - 1 (0-65535) -> 1-65536 sound frames

    data @2 :Data;
  }
}

[Sergio Baldoví]

Appears promising. If I understand correctly, Fuse and fmfconv would need Cap’n Proto and c-capnproto compilers as build dependencies and c-capnproto library as runtime dependency (unless included in source tree).

[Fredrick Meunier]

Yes, that's right. It may make sense to roll the format handling into libspectrum and handle it there rather than duplicate it across Fuse and fuse-utils.

[Fredrick Meunier]

This is progressing slowly - I am currently writing a FMF V1 to FMF V2 converter and it won't be done for the next release.