[Gpsbabel-code] [PATCH] Realtime CSV support and fixes, Part of second support in CSV files

[Gerhard Olsson <gerhard.nospam@gm...>]

Two kind of separate patches, but since I have developed and tested
them together. Should not be too difficult to handle, as the only
common file is xcsv.c (reading of options).
No documentation updated (other than this mail), no enhancement bug
submitted, I send them for review first.
Tested in Cygwin/Linux.

Possible impact on existing functionality:
 * Of course, there is a risk I have screwed up existing handling for
reading/writing csv, but this is relative easy to review and test so
the risk is low.
 * Parts of second is new functionality, mark as experimental
 * Realtime is marked as experimental still and not really working in
CVS, so the risk should be low.


----------------------
CSV part of seconds support

Robert added millisecond support to GPX in 1.3.3
However, there is no support in reading/writing parts of a second to CSV files.
This is not an easy match as the time handling is defined to
manipulating complete seconds. I have come up with the following
handling, but someone may have better ideas?

The following is done:
 Fields MILLISECOND and MICROSECOND added. Best written as ".%03" and
".%06" (using option "no_delim_before")
 "no_delim_before" implemented also for IFIELD with slightly differnt
meaning, the complete info is parsed again. Example:
   IFIELD	GMT_TIME,"","%m/%d/%Y %H:%M:%S"
   IFIELD	MILLISECOND,"","%*m/%*d/%*Y %*H:%*M:%*S.%03d","no_delim_before"
     or
   IFIELD	MILLISECOND,"","%*[^.].%03d","no_delim_before,optional"
  ISO_TIME_MS reads and writes time with milliseconds


Possible caveeats:
 * No checks that the user have written time correctly for MILLISECOND
and MICROSECOND. I.e 3.18 is interpreted as "18" us. The reason is
that I know no way to find the length of a match, where sscanf starts
and ends. A solution is to request that the user uses a %[0123456789]
pattern and the resulting string is parsed, but that is clumsy.
   This is not a problem with *_MS field that has integrated
 * YYYMMDD_TIME_MS uses double when parsing that teoretically can give
precision errors
 * Rounding is incorrect at output for MILLISECOND if (us >
999000)(unless for *_MS fields), there is no way to add seconds to a
previously written fields (output set to 999)
 * No rounding for microseconds, always truncated

Some code (commented out) for EXCEL_TIME_MS, but I cannot test it.
Changes in csv_util.h, csv_util.c, xcsv.c

-------------------
Realtime
 * support for CSV as output
 * support for "-" as output (sets the max_position_points to 1 (no
big point for kml, but included))

 * Files were not actually saved, until exit of utility (error in CVS,
likely OK in 1.3.3)
 * input/output files had to be specified with -f, -F (not last on the
command line)
 * kml files where max_position_points were set were garbled

CSV support required some tweaks I wish I was not necessary:
 * Init (parse style etc) done only once. This increases the
efficiency, but done mostly to avoid core when exit (wr_deinit called
twice)
 * Must set datatype to wpt/trk to avoid double output of lines.

Changes in xcsv.c, kml.c, main.c

-- 
Gerhard
The sender address really contains .nospam.

[Gpsbabel-code] [PATCH] Realtime CSV support and fixes, Part of second support in CSV files

[Gerhard Olsson <gerhard.nospam@gm...>]

Two kind of separate patches, but since I have developed and tested
them together. Should not be too difficult to handle, as the only
common file is xcsv.c (reading of options).
No documentation updated (other than this mail), no enhancement bug
submitted, I send them for review first.
Tested in Cygwin/Linux.

Possible impact on existing functionality:
 * Of course, there is a risk I have screwed up existing handling for
reading/writing csv, but this is relative easy to review and test so
the risk is low.
 * Parts of second is new functionality, mark as experimental
 * Realtime is marked as experimental still and not really working in
CVS, so the risk should be low.


----------------------
CSV part of seconds support

Robert added millisecond support to GPX in 1.3.3
However, there is no support in reading/writing parts of a second to CSV files.
This is not an easy match as the time handling is defined to
manipulating complete seconds. I have come up with the following
handling, but someone may have better ideas?

The following is done:
 Fields MILLISECOND and MICROSECOND added. Best written as ".%03" and
".%06" (using option "no_delim_before")
 "no_delim_before" implemented also for IFIELD with slightly differnt
meaning, the complete info is parsed again. Example:
   IFIELD	GMT_TIME,"","%m/%d/%Y %H:%M:%S"
   IFIELD	MILLISECOND,"","%*m/%*d/%*Y %*H:%*M:%*S.%03d","no_delim_before"
     or
   IFIELD	MILLISECOND,"","%*[^.].%03d","no_delim_before,optional"
  ISO_TIME_MS reads and writes time with milliseconds


Possible caveeats:
 * No checks that the user have written time correctly for MILLISECOND
and MICROSECOND. I.e 3.18 is interpreted as "18" us. The reason is
that I know no way to find the length of a match, where sscanf starts
and ends. A solution is to request that the user uses a %[0123456789]
pattern and the resulting string is parsed, but that is clumsy.
   This is not a problem with *_MS field that has integrated
 * YYYMMDD_TIME_MS uses double when parsing that teoretically can give
precision errors
 * Rounding is incorrect at output for MILLISECOND if (us >
999000)(unless for *_MS fields), there is no way to add seconds to a
previously written fields (output set to 999)
 * No rounding for microseconds, always truncated

Some code (commented out) for EXCEL_TIME_MS, but I cannot test it.
Changes in csv_util.h, csv_util.c, xcsv.c

-------------------
Realtime
 * support for CSV as output
 * support for "-" as output (sets the max_position_points to 1 (no
big point for kml, but included))

 * Files were not actually saved, until exit of utility (error in CVS,
likely OK in 1.3.3)
 * input/output files had to be specified with -f, -F (not last on the
command line)
 * kml files where max_position_points were set were garbled

CSV support required some tweaks I wish I was not necessary:
 * Init (parse style etc) done only once. This increases the
efficiency, but done mostly to avoid core when exit (wr_deinit called
twice)
 * Must set datatype to wpt/trk to avoid double output of lines.

Changes in xcsv.c, kml.c, main.c

-- 
Gerhard
The sender address really contains .nospam.

Re: [Gpsbabel-code] [PATCH] Realtime CSV support and fixes, Part of second support in CSV files

[Gerhard Olsson <gerhard.nospam@gm...>]

Submitted patches:

CSV microseconds:
http://sourceforge.net/tracker/index.php?func=detail&aid=1746009&group_id=58972&atid=489477
CSV realtime:
http://sourceforge.net/tracker/index.php?func=detail&aid=1746009&group_id=58972&atid=489477


On 6/30/07, Gerhard Olsson <gerhard.nospam@...> wrote:
> Two kind of separate patches, but since I have developed and tested
> them together. Should not be too difficult to handle, as the only
> common file is xcsv.c (reading of options).
> No documentation updated (other than this mail), no enhancement bug
> submitted, I send them for review first.
> Tested in Cygwin/Linux.
>
> Possible impact on existing functionality:
>  * Of course, there is a risk I have screwed up existing handling for
> reading/writing csv, but this is relative easy to review and test so
> the risk is low.
>  * Parts of second is new functionality, mark as experimental
>  * Realtime is marked as experimental still and not really working in
> CVS, so the risk should be low.
>
>
> ----------------------
> CSV part of seconds support
>
> Robert added millisecond support to GPX in 1.3.3
> However, there is no support in reading/writing parts of a second to CSV files.
> This is not an easy match as the time handling is defined to
> manipulating complete seconds. I have come up with the following
> handling, but someone may have better ideas?
>
> The following is done:
>  Fields MILLISECOND and MICROSECOND added. Best written as ".%03" and
> ".%06" (using option "no_delim_before")
>  "no_delim_before" implemented also for IFIELD with slightly differnt
> meaning, the complete info is parsed again. Example:
>    IFIELD       GMT_TIME,"","%m/%d/%Y %H:%M:%S"
>    IFIELD       MILLISECOND,"","%*m/%*d/%*Y %*H:%*M:%*S.%03d","no_delim_before"
>      or
>    IFIELD       MILLISECOND,"","%*[^.].%03d","no_delim_before,optional"
>   ISO_TIME_MS reads and writes time with milliseconds
>
>
> Possible caveeats:
>  * No checks that the user have written time correctly for MILLISECOND
> and MICROSECOND. I.e 3.18 is interpreted as "18" us. The reason is
> that I know no way to find the length of a match, where sscanf starts
> and ends. A solution is to request that the user uses a %[0123456789]
> pattern and the resulting string is parsed, but that is clumsy.
>    This is not a problem with *_MS field that has integrated
>  * YYYMMDD_TIME_MS uses double when parsing that teoretically can give
> precision errors
>  * Rounding is incorrect at output for MILLISECOND if (us >
> 999000)(unless for *_MS fields), there is no way to add seconds to a
> previously written fields (output set to 999)
>  * No rounding for microseconds, always truncated
>
> Some code (commented out) for EXCEL_TIME_MS, but I cannot test it.
> Changes in csv_util.h, csv_util.c, xcsv.c
>
> -------------------
> Realtime
>  * support for CSV as output
>  * support for "-" as output (sets the max_position_points to 1 (no
> big point for kml, but included))
>
>  * Files were not actually saved, until exit of utility (error in CVS,
> likely OK in 1.3.3)
>  * input/output files had to be specified with -f, -F (not last on the
> command line)
>  * kml files where max_position_points were set were garbled
>
> CSV support required some tweaks I wish I was not necessary:
>  * Init (parse style etc) done only once. This increases the
> efficiency, but done mostly to avoid core when exit (wr_deinit called
> twice)
>  * Must set datatype to wpt/trk to avoid double output of lines.
>
> Changes in xcsv.c, kml.c, main.c
>
> --
> Gerhard
> The sender address really contains .nospam.
>
>


-- 
Gerhard
The sender address really contains .nospam.

Re: [Gpsbabel-code] [PATCH] Realtime CSV support and fixes, Part of second support in CSV files

[Robert Lipe <robertlipe@gp...>]

> Two kind of separate patches, but since I have developed and tested
> them together. Should not be too difficult to handle, as the only

I'm not sure if the intent was to obsolete these with what you did on SF
a day later, but as there's no file attached there, I'll just comment on
this one. Please use unified ('cvs diff -u') or context diffs ('cvs diff
-c' when submitting code to review. It's easier to read and much more
likely to apply without excessive human costs.

> * Of course, there is a risk I have screwed up existing handling for
> reading/writing csv, but this is relative easy to review and test so
> the risk is low.

Our test suite is supposed to be a safety net there.  It's not foolproof
(we manage to implement lots of new problems. :-) but it's a help.

> However, there is no support in reading/writing parts of a second to CSV 
> files.
> This is not an easy match as the time handling is defined to
> manipulating complete seconds. I have come up with the following
> handling, but someone may have better ideas?

Ugh. The number of microsecs is internally in a field that's bolted
onto time_t, but I'd like to not expose the user to that particular
bit of unpleasantness. How about we add creation_time + (microsecs
/ 1000000.0), stuff it into a float, and use that instead of
wpt->creation_time in the conversions? Wouldn't that be more transparent
to the user?

Or is the goal to support some format that has these values actually
disembodied (in a different XCSV field) from the whole number of
seconds?


> that I know no way to find the length of a match, where sscanf starts
> and ends. A solution is to request that the user uses a %[0123456789]

	        sscanf("1.234blah", "%lf%n", &flag, &consumed);

will write the double "1.234" into flag and write the number of chars
consumed (5) into the integer consumed.

> Realtime
> * support for CSV as output

Is there anything that would actually use this mode?

Let's tackle the first half first.

Thanx,
RJL

Re: [Gpsbabel-code] [PATCH] Realtime CSV support and fixes, Part of second support in CSV files

[Gerhard Olsson <gerhard.nospam@gm...>]

On 7/2/07, Robert Lipe <robertlipe@...> wrote:
> > Two kind of separate patches, but since I have developed and tested
> > them together. Should not be too difficult to handle, as the only
>
> I'm not sure if the intent was to obsolete these with what you did on SF
> a day later, but as there's no file attached there, I'll just comment on
> this one. Please use unified ('cvs diff -u') or context diffs ('cvs diff -c

I submitted to mailing list, when I did not see the message appear
on the mailing list, (thought the attachment could be scrubbed)
I added the same patch the day after to SF (what I should have done in
the first place).

Will upload in unified format to SF

> > However, there is no support in reading/writing parts of a second to CSV
> > files.
> > This is not an easy match as the time handling is defined to
> > manipulating complete seconds. I have come up with the following
> > handling, but someone may have better ideas?
>
> Ugh. The number of microsecs is internally in a field that's bolted
> onto time_t, but I'd like to not expose the user to that particular
> bit of unpleasantness. How about we add creation_time + (microsecs
> / 1000000.0), stuff it into a float, and use that instead of
> wpt->creation_time in the conversions? Wouldn't that be more transparent
> to the user?
>

Since seconds is the basis for all time computations, I do not find it
that bad to keep parts of a second separate.
I abandoned seconds as a float for the following reasons:
 * The str*time functions cannot be used easily without some internal
handling that divides the task to parse in two steps
 * Parsing seconds separately will be like parsing time_t as a float,
with potential rounding errors
 * What to base the seconds on? time_t requiring the user to convert
to Unix time first?
   (Actually, I implemented TIME_T_MS but ripped this out due to not
want more time_t exposure and because this was relative easy to do
with MILLISECONDS).

The ugliest part as I see it is requiring separate IFIELD/OFIELD (and
two time fields)

> Or is the goal to support some format that has these values actually
> disembodied (in a different XCSV field) from the whole number of
> seconds?
>
I am happy with one common format with millisecond support, preferably
ISO_TIME_MS.
The MILLISECOND and MICROSECOND fields were added to get a "complete"
functionality, also for other time formats.

>
> > that I know no way to find the length of a match, where sscanf starts
> > and ends. A solution is to request that the user uses a %[0123456789]
>
>                 sscanf("1.234blah", "%lf%n", &flag, &consumed);
>
> will write the double "1.234" into flag and write the number of chars
> consumed (5) into the integer consumed.

Thanks for the lesson. Should add to parsing of MILLISECONDS or MICROSECONDS

There is still be a problem for all fields with user supplied format
strings like HMS*_TIME though, there is no way to find where the
second field is without parsing the line.
The proper solution is to parse the format line like
"%Y-%m-%dT%H:%M:%SZ" for %S and generate two format strings for
parsing like:
  %Y-%m-%dT%H:%M:%SZ
  %*d-%*d-%*dT%*d:%*d:%*d%[.,0123456789]%nZ
and generate a format string with wanted part of second format for output
Parsing could be done automatically, but output should be controlled somehow.

I rather rip out the fields MILLISECONDS and MICROSECONDS than
implementing this.
Another alternative is to simply not documenting the fields.

>
> > Realtime
> > * support for CSV as output
>
> Is there anything that would actually use this mode?
>
That is why I started. I wanted to make a "live updating" plugin to a
non-core GIS application that called GPSBabel as an external program.
I wanted to get some information that is not in kml and not bother to
check when files are updated (stdout instead of files).

Thanks

Gerhard