[Biosig-general] GDF v3: timestamps and timezone information

[Alois S. <alo...@is...>]

This email concerns mostly software developers involved accessing GDF
data with their own software. If you are a user of biosig, this mail
should not concern you, because Biosig will provide full support for
past and future versions of the GDF format.

As you might know, the aim of GDF is to combine to the best features of
all data formats in a single format. For some time and especially in the
last few months, it became clear that some additional features that
might be useful in GDF, too. These are:
a) TimeStamps
b) Time zone information
c) and whether the POSition field in the eventtable should be extended
to 64 bit (currently its 32 bit).

(a) Timestamps
GDF supports non-continuous recording through the Event 0x7ffe (start of
a new segment after a break). However, there is no way the know the
absolute starttime of the next segment. In order to store the absolute
starttime, I plan to extend the event table by an additional field for
the timestamp. This will just be an extension of the current format and
will be no problem for backwards compatibility. (probably bit 2 in the
mode field of the Eventtable will be set to indicate the timestamp field).


(b) time zone information.
Currently, there is no way to know whether the starttime (and in future
the timestamps) are stored in localtime or utc, or some other timezone.
The plan is to use the currently unused bytes 254 and 255 for storing
the timezone offset utc in minutes.
e.g. +60 for Berlin/Europe in Winter and +120 for Berlin/Europe summer
time.

The difficult part is to decide whether the starttime and the timestamps
should be stored in localtime on in utc. This is tricky, because
time zone information is not very helpful, but might introduced unwanted
ambiguities if its not clear which time is stored internally.

Arguments in favor of localtime (+) and in favor of utc (-) :
    + most data use local time (is this true ?),
    + data conversion can ignore the time zone information
    + easiest migration path because less need for changes are expected
    + For circadian and sleep studies, the local time is more relevant
    - effort for timezone management is shifted to application space.
    - long term compatibility with other systems
    - the problem of data conversion might become more difficult later on

I've not made up my mind about that, but would like your imput on it.


(c) EVENT.POS and EVENT.DUR are limited to 32 bit.
Currently, event positions are limited to ~4e9 samples. Do we expect
that we'll have data with more then 2^32 (~4e9) samples ? That's
continuous recordings for 11.9 hours sampled with 100 kHz. I'm not
talking about the file size. The file size limits are IMHO far beyond
any current need (65535 channels, 2^63~9e19 records, each record can
hold up to 2^32 ~ 4e9 samples, the number of events is limited to 2^24 ~
16 Mio events). The limitation is only that the event table can not
address samples beyond 4e9. Similarily, sparse samples can be stored
only in datatypes up to 4 bytes (no int64 or double) due to the size of
the duration field. The question is whether there is - in the
foreseeable future
(lets say 5 years) - any need to extend these limitations.


Concerning the timeline, I think (a) can go into GDF relatively soon
(v2.5). For v2.5 I'd suggest to store everything in local
time. And time will tell, whether we can confirm this policy or whether
we need to revise this decision in v3.0.


Are there any objections, concerns, or do you agree? I'm looking forward
to your responses.



Alois