Thread: [Gpsbabel-misc] Filtering waypoints from inaccurate GPS
Brought to you by:
robertl
From: <gps...@he...> - 2010-03-28 20:49:13
|
Hi. I've got the iGot-U Travel Logger [1] (aka Mr Lee Cat Track [2]), which I attach to my cat. The device is great for its size, but it seems to be logging quite a few wrong waypoints. Unless my cat is able to teleport and fly, of course. Here's an example track: http://www.everytrail.com/view_trip.php?trip_id=237305 Is there a way to make gpsbabel filter out the faulty waypoints? Best regards, Henrik [1] http://www.a-trip.com/download#IGOTU [2] http://www.mr-lee-catcam.de/ct_how_en.htm |
From: Robert L. <rob...@gp...> - 2010-03-29 01:58:07
|
On Sun, Mar 28, 2010 at 2:33 PM, <gps...@he...> wrote: > Hi. > > I've got the iGot-U Travel Logger [1] (aka Mr Lee Cat Track [2]), which > I attach to my cat. The device is great for its size, but it seems to > be logging quite a few wrong waypoints. Unless my cat is able to > teleport and fly, of course. > I've long suspected that cats could do more than they were letting on and have been plotting against us to take over the world. I'm not at all sure I want to be involved in any technology that might reveal their plan before they're ready. When kitty domination is the norm and they're determining our ranks in their servitude, I don't want to be the guy that helped expose their plan. I want a good seat under the new reign. I don't know anything about the iGotu. If it stores some kind of DOP information in whatever format you're using and the GPS is smart enough to know when the points probably shouldn't be truested, you can use the 'discard' filter to toss the zingers. See: http://www.gpsbabel.org/htmldoc-development/filter_discard.html > Here's an example track: > http://www.everytrail.com/view_trip.php?trip_id=237305 > > Is there a way to make gpsbabel filter out the faulty waypoints? > Best regards, > Henrik > > [1] http://www.a-trip.com/download#IGOTU > [2] http://www.mr-lee-catcam.de/ct_how_en.htm > > > ------------------------------------- > |
From: <gps...@he...> - 2010-03-29 17:57:35
|
Den 29.03.10 03.57, skrev Robert Lipe: > I don't know anything about the iGotu. If it stores some kind of DOP > information in whatever format you're using and the GPS is smart > enough to know when the points probably shouldn't be truested, you can > use the 'discard' filter to toss the zingers. See: > http://www.gpsbabel.org/htmldoc-development/filter_discard.html Thanks, I've used the discard filter to remove waypoints with a low satellite count: gpsbabel -t -i gpx -f in.gpx -x discard,sat=7 -o gpx -F out.gpx The result is a bit better, but I still see waypoints where the cat is flying or digging far under ground. Thus I'd like to discard waypoints that are above or below a certain elevation, but I can't find any filters for it. Are there any? Best regards, Henrik |
From: Robert L. <rob...@gp...> - 2010-03-29 18:20:51
|
On Mon, Mar 29, 2010 at 11:57 AM, <gps...@he...> wrote: > Den 29.03.10 03.57, skrev Robert Lipe: > > I don't know anything about the iGotu. If it stores some kind of DOP > > information in whatever format you're using and the GPS is smart > > enough to know when the points probably shouldn't be truested, you can > > use the 'discard' filter to toss the zingers. See: > > http://www.gpsbabel.org/htmldoc-development/filter_discard.html > > Thanks, I've used the discard filter to remove waypoints with a low > satellite count: > > gpsbabel -t -i gpx -f in.gpx -x discard,sat=7 -o gpx -F out.gpx > > The result is a bit better, but I still see waypoints where the cat is > flying or digging far under ground. Thus I'd like to discard waypoints > that are above or below a certain elevation, but I can't find any > filters for it. Are there any? > Don't think so. As the world is, unfortunately, not spherical and GPS altitude isn't really very accurate anyway, doing this accurately would require a digital elevation model of the world and is outside the scope of GPSBabel. As kitteh is presumably earth-bound and your software probably assumes that it's on the ground (sometimes to the annoyance of pilot types....) you may just try removing all the <ele> tags from the GPX and see if that gives more satisfying results. RJL |
From: <gps...@he...> - 2010-03-29 18:41:56
|
Den 29.03.10 20.20, skrev Robert Lipe: > On Mon, Mar 29, 2010 at 11:57 AM, <gps...@he... > <mailto:gps...@he...>> wrote: > > The result is a bit better, but I still see waypoints where the cat is > flying or digging far under ground. Thus I'd like to discard > waypoints > that are above or below a certain elevation, but I can't find any > filters for it. Are there any? > > > Don't think so. As the world is, unfortunately, not spherical and GPS > altitude isn't really very accurate anyway, doing this accurately > would require a digital elevation model of the world and is outside > the scope of GPSBabel. > > As kitteh is presumably earth-bound and your software probably assumes > that it's on the ground (sometimes to the annoyance of pilot > types....) you may just try removing all the <ele> tags from the GPX > and see if that gives more satisfying results. I thought of the way-off <ele> tags as identifiers of faulty waypoints. I wrote a quick and dirty php script to remove those points, and voila! Look at the trail now, it's much better: http://www.everytrail.com/view_trip.php?trip_id=550268 I think I'll have to look at the source code for the discard filter, this is definitly a useful feature :) Henrik PS: Here's the ugly script in case anyone is interested. Run with droptracks.php in.gpx > out.gpx #!/usr/bin/env php <?php $MIN = 0; $MAX = 100; $lines = preg_split("/\n/", file_get_contents($argv[1])); $dropped = 0; $started = false; $buf = ""; $valid = true; foreach ($lines as $line) { if (stripos($line, "<trkpt ") !== false) { $started = true; } else if ($started == false) { print "$line\n"; continue; } if (stripos($line, "</ele>") !== false) { $ele = preg_replace("#^.*<ele>(.*)</ele>.*$#i", "$1", $line); if ($ele < $MIN || $ele > $MAX) { $valid = false; } } $buf .= "$line\n"; if (stripos($line, "</trkpt>") !== false) { if ($valid) { print $buf; } else { $dropped++; } $valid = true; $buf = ""; $started = false; } } fwrite(STDERR, "Dropped $dropped waypoints\n"); |
From: Greg <we...@we...> - 2010-03-29 19:04:35
|
On Mar 29, 2010, at 11:41 AM, gps...@he... wrote: > Den 29.03.10 20.20, skrev Robert Lipe: >> On Mon, Mar 29, 2010 at 11:57 AM, <gps...@he...> wrote: >> The result is a bit better, but I still see waypoints where the cat is >> flying or digging far under ground. Thus I'd like to discard waypoints >> that are above or below a certain elevation, but I can't find any >> filters for it. Are there any? >> >> Don't think so. As the world is, unfortunately, not spherical and GPS altitude isn't really very accurate anyway, doing this accurately would require a digital elevation model of the world and is outside the scope of GPSBabel. >> >> As kitteh is presumably earth-bound and your software probably assumes that it's on the ground (sometimes to the annoyance of pilot types....) you may just try removing all the <ele> tags from the GPX and see if that gives more satisfying results. > > I thought of the way-off <ele> tags as identifiers of faulty waypoints. I wrote a quick and dirty php script to remove those points, and voila! Look at the trail now, it's much better: http://www.everytrail.com/view_trip.php?trip_id=550268 > > I think I'll have to look at the source code for the discard filter, this is definitly a useful feature :) > > Henrik > > PS: Here's the ugly script in case anyone is interested. Run with droptracks.php in.gpx > out.gpx > > #!/usr/bin/env php > <?php > > $MIN = 0; > $MAX = 100; > > $lines = preg_split("/\n/", file_get_contents($argv[1])); > > $dropped = 0; > $started = false; > $buf = ""; > $valid = true; > > foreach ($lines as $line) { > if (stripos($line, "<trkpt ") !== false) { > $started = true; > } else if ($started == false) { > print "$line\n"; > continue; > } > if (stripos($line, "</ele>") !== false) { > $ele = preg_replace("#^.*<ele>(.*)</ele>.*$#i", "$1", $line); > if ($ele < $MIN || $ele > $MAX) { > $valid = false; > } > } > $buf .= "$line\n"; > if (stripos($line, "</trkpt>") !== false) { > if ($valid) { > print $buf; > } else { > $dropped++; > } > $valid = true; > $buf = ""; > $started = false; > } > } > fwrite(STDERR, "Dropped $dropped waypoints\n"); > > Are you sure you want everyone knowing where your kitty is? Have you asked her/him? Cats are very private you know. You're lucky you don't live in the US, you probably have the animal rights groups after you. A follow up to Lipe's original comment. Please don't take this seriously. But GPS has to be about more than elevation and source code. |
From: <gps...@he...> - 2010-03-29 19:21:48
Attachments:
discard.c.patch
|
Den 29.03.10 21.04, skrev Greg: > Are you sure you want everyone knowing where your kitty is? Have you > asked her/him? Cats are very private you know. You're lucky you don't > live in the US, you probably have the animal rights groups after you. Just don't tell her. She's more into layering my keyboard with fur than actually using the keyboard. Or so I'm led to believe. Anyhow, patching the filter was pretty simple, and I've attached the patch to discard.c here. Usage: gpsbabel -t -i gpx -f in.gpx -x discard,elemin=0 -x discard,elemax=99 -o gpx -F out.gpx Henrik |
From: Robert L. <rob...@gp...> - 2010-03-29 19:33:28
|
On Mon, Mar 29, 2010 at 1:21 PM, <gps...@he...> wrote: > Den 29.03.10 21.04, skrev Greg: > > Are you sure you want everyone knowing where your kitty is? Have you > > asked her/him? Cats are very private you know. You're lucky you don't > > live in the US, you probably have the animal rights groups after you. > > Just don't tell her. She's more into layering my keyboard with fur than > actually using the keyboard. Or so I'm led to believe. > That deception is all part of their plan... Anyhow, patching the filter was pretty simple, and I've attached the > patch to discard.c here. Usage: > If you can scratch up a few sentences describing this (units are in meters[1], why you'd ever want to use it, etc.) I'll check it in. I suspect it's too simple for general purpose use, but if we set expectations appropriately low maybe someone will find it useful. Thanx, RJL [1] Yes, there are still backwards corners of the world where SI isn't common and every time we add an option that accepts linear units and parses the letter 'm' I get flamed for that sometimes meaning 'miles' and sometimes 'meters'. |
From: Henrik B. A. <gps...@he...> - 2010-03-29 19:53:24
|
Den 29.03.10 21.33, skrev Robert Lipe: > If you can scratch up a few sentences describing this (units are in > meters[1], why you'd ever want to use it, etc.) I'll check it in. I > suspect it's too simple for general purpose use, but if we set > expectations appropriately low maybe someone will find it useful. Allright, I'll try: *elemin option * Suppress waypoints with lower elevation This option drops waypoints with an altitude lower than the specified value (in meters). Although GPS altitude isn't very accurate, GPS devices might log faulty waypoints from time to time, e.g. near tall buildings. Elevation values that are way off may signify such waypoints. *elemax option* Suppress waypoints with higher elevation This option drops waypoints with an altitude higher than the specified value (in meters), in the same manner as the elemin option. Best regards, Henrik |
From: Jan M. <jan...@di...> - 2010-03-31 23:26:31
|
Hi, I was facing a similar problem. Check out my solution using the "simplify" option: http://www.diy-streetview.org/2010/02/20/geotagging-for-streetviews Jan On Mon, Mar 29, 2010 at 9:53 PM, Henrik Brautaset Aronsen < gps...@he...> wrote: > Den 29.03.10 21.33, skrev Robert Lipe: > > If you can scratch up a few sentences describing this (units are in > meters[1], why you'd ever want to use it, etc.) I'll check it in. I > suspect it's too simple for general purpose use, but if we set expectations > appropriately low maybe someone will find it useful. > > > Allright, I'll try: > > *elemin option > * > Suppress waypoints with lower elevation > > This option drops waypoints with an altitude lower than the specified value > (in meters). Although GPS altitude isn't very accurate, GPS devices might > log faulty waypoints from time to time, e.g. near tall buildings. Elevation > values that are way off may signify such waypoints. > > *elemax option* > > Suppress waypoints with higher elevation > > This option drops waypoints with an altitude higher than the specified > value (in meters), in the same manner as the elemin option. > > Best regards, > Henrik > > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > > -- http://www.DIY-streetview.org |
From: Robert L. <rob...@gp...> - 2010-03-31 23:43:18
|
On Wed, Mar 31, 2010 at 4:59 PM, Jan Martin <jan...@di...>wrote: > Hi, > > I was facing a similar problem. > > Check out my solution using the "simplify" option: > > http://www.diy-streetview.org/2010/02/20/geotagging-for-streetviews > > Simplify works well for points that are just noisy or slightly wrong. It works badly for wildly wrong points, which is what the OP had. In fact, simplify works pretty hard to keep those "very wrong" points because they have great influence on the shape of the resulting polyline. > Jan > > > On Mon, Mar 29, 2010 at 9:53 PM, Henrik Brautaset Aronsen < > gps...@he...> wrote: > >> Den 29.03.10 21.33, skrev Robert Lipe: >> >> If you can scratch up a few sentences describing this (units are in >> meters[1], why you'd ever want to use it, etc.) I'll check it in. I >> suspect it's too simple for general purpose use, but if we set expectations >> appropriately low maybe someone will find it useful. >> >> >> Allright, I'll try: >> >> *elemin option >> * >> Suppress waypoints with lower elevation >> >> This option drops waypoints with an altitude lower than the specified >> value (in meters). Although GPS altitude isn't very accurate, GPS devices >> might log faulty waypoints from time to time, e.g. near tall buildings. >> Elevation values that are way off may signify such waypoints. >> >> *elemax option* >> >> Suppress waypoints with higher elevation >> >> This option drops waypoints with an altitude higher than the specified >> value (in meters), in the same manner as the elemin option. >> >> Best regards, >> Henrik >> >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Gpsbabel-misc mailing list http://www.gpsbabel.org >> Gps...@li... >> To unsubscribe, change list options, or see archives, visit: >> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >> >> > > > -- > http://www.DIY-streetview.org > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > > |
From: Jan M. <jan...@di...> - 2010-04-01 04:28:14
|
Robert, I wonder if it would be possible to have the "reverse" function to this: Remove points that have the largest effect on the shape. Thanks, Jan crosstrack option Use cross-track error (default). This option instructs GPSBabel to remove points that have the *smallest*overall effect on the overall shape of the route. Using this method, the first point to be removed will be the one that is closest to a line drawn between the two points adjacent to it. If neither this option nor the length option is specified, this is the default. On Thu, Apr 1, 2010 at 1:43 AM, Robert Lipe <rob...@gp...> wrote: > > > On Wed, Mar 31, 2010 at 4:59 PM, Jan Martin <jan...@di...>wrote: > >> Hi, >> >> I was facing a similar problem. >> >> Check out my solution using the "simplify" option: >> >> http://www.diy-streetview.org/2010/02/20/geotagging-for-streetviews >> >> > Simplify works well for points that are just noisy or slightly wrong. It > works badly for wildly wrong points, which is what the OP had. In fact, > simplify works pretty hard to keep those "very wrong" points because they > have great influence on the shape of the resulting polyline. > > > >> Jan >> >> >> On Mon, Mar 29, 2010 at 9:53 PM, Henrik Brautaset Aronsen < >> gps...@he...> wrote: >> >>> Den 29.03.10 21.33, skrev Robert Lipe: >>> >>> If you can scratch up a few sentences describing this (units are in >>> meters[1], why you'd ever want to use it, etc.) I'll check it in. I >>> suspect it's too simple for general purpose use, but if we set expectations >>> appropriately low maybe someone will find it useful. >>> >>> >>> Allright, I'll try: >>> >>> *elemin option >>> * >>> Suppress waypoints with lower elevation >>> >>> This option drops waypoints with an altitude lower than the specified >>> value (in meters). Although GPS altitude isn't very accurate, GPS devices >>> might log faulty waypoints from time to time, e.g. near tall buildings. >>> Elevation values that are way off may signify such waypoints. >>> >>> *elemax option* >>> >>> Suppress waypoints with higher elevation >>> >>> This option drops waypoints with an altitude higher than the specified >>> value (in meters), in the same manner as the elemin option. >>> >>> Best regards, >>> Henrik >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> Gpsbabel-misc mailing list http://www.gpsbabel.org >>> Gps...@li... >>> To unsubscribe, change list options, or see archives, visit: >>> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >>> >>> >> >> >> -- >> http://www.DIY-streetview.org >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Gpsbabel-misc mailing list http://www.gpsbabel.org >> Gps...@li... >> To unsubscribe, change list options, or see archives, visit: >> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >> >> > -- http://www.DIY-streetview.org |
From: Robert L. <rob...@gp...> - 2010-04-01 04:35:16
|
On Wed, Mar 31, 2010 at 10:28 PM, Jan Martin <jan...@di...>wrote: > Robert, > > I wonder if it would be possible to have the "reverse" function to this: > Remove points that have the largest effect on the shape. > > Thanks, > > Jan > For non-broken (and the OP already knows his tracks _are_ broken...) wouldn't this be a Very Bad Thing? It would likely remove the endpoints, and the most strategic turnpoints, for example. I really think that for this class of GPSes that insists on recording points even when they basically know they're pullling coords out of their...hats that trusting their DOP is the thing to do. If a GPS maker records made up coords and doesn't associate a DOP value with them, then shame on them. That said, I really would like to see a data set &/or code for a filter that could figure that that kitteh wasn't walking, teleported to the next continent one second, then teleported back the next and could figure out that erroneous point was the bogus one. > crosstrack option > > Use cross-track error (default). > > This option instructs GPSBabel to remove points that have the *smallest*overall effect on the overall shape of the route. Using this method, the > first point to be removed will be the one that is closest to a line drawn > between the two points adjacent to it. > > If neither this option nor the length option is specified, this is the > default. > > > On Thu, Apr 1, 2010 at 1:43 AM, Robert Lipe <rob...@gp...>wrote: > >> >> >> On Wed, Mar 31, 2010 at 4:59 PM, Jan Martin <jan...@di... >> > wrote: >> >>> Hi, >>> >>> I was facing a similar problem. >>> >>> Check out my solution using the "simplify" option: >>> >>> http://www.diy-streetview.org/2010/02/20/geotagging-for-streetviews >>> >>> >> Simplify works well for points that are just noisy or slightly wrong. It >> works badly for wildly wrong points, which is what the OP had. In fact, >> simplify works pretty hard to keep those "very wrong" points because they >> have great influence on the shape of the resulting polyline. >> >> >> >>> Jan >>> >>> >>> On Mon, Mar 29, 2010 at 9:53 PM, Henrik Brautaset Aronsen < >>> gps...@he...> wrote: >>> >>>> Den 29.03.10 21.33, skrev Robert Lipe: >>>> >>>> If you can scratch up a few sentences describing this (units are in >>>> meters[1], why you'd ever want to use it, etc.) I'll check it in. I >>>> suspect it's too simple for general purpose use, but if we set expectations >>>> appropriately low maybe someone will find it useful. >>>> >>>> >>>> Allright, I'll try: >>>> >>>> *elemin option >>>> * >>>> Suppress waypoints with lower elevation >>>> >>>> This option drops waypoints with an altitude lower than the specified >>>> value (in meters). Although GPS altitude isn't very accurate, GPS devices >>>> might log faulty waypoints from time to time, e.g. near tall buildings. >>>> Elevation values that are way off may signify such waypoints. >>>> >>>> *elemax option* >>>> >>>> Suppress waypoints with higher elevation >>>> >>>> This option drops waypoints with an altitude higher than the specified >>>> value (in meters), in the same manner as the elemin option. >>>> >>>> Best regards, >>>> Henrik >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Download Intel® Parallel Studio Eval >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> _______________________________________________ >>>> Gpsbabel-misc mailing list http://www.gpsbabel.org >>>> Gps...@li... >>>> To unsubscribe, change list options, or see archives, visit: >>>> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >>>> >>>> >>> >>> >>> -- >>> http://www.DIY-streetview.org >>> >>> >>> ------------------------------------------------------------------------------ >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> Gpsbabel-misc mailing list http://www.gpsbabel.org >>> Gps...@li... >>> To unsubscribe, change list options, or see archives, visit: >>> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >>> >>> >> > > > -- > http://www.DIY-streetview.org > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > > |
From: Tom P. <tom...@gm...> - 2010-04-01 05:08:02
|
On Thu, Apr 1, 2010 at 3:35 PM, Robert Lipe <rob...@gp...> wrote: > > > On Wed, Mar 31, 2010 at 10:28 PM, Jan Martin <jan...@di...> wrote: >> >> Robert, >> >> I wonder if it would be possible to have the "reverse" function to this: >> Remove points that have the largest effect on the shape. >> >> Thanks, >> >> Jan > > For non-broken (and the OP already knows his tracks _are_ broken...) wouldn't this be a Very Bad Thing? It would likely remove the endpoints, and the most strategic turnpoints, for example. I think this sounds promising, but it will be a bigg challenge to determine when to stop the algorithm: The existing simplify filter is "self limiting" as it runs out of points that don't change the path length significantly. Removing the points with the biggest effect can proceed until there is nothing left... That said, obviously the end-points are a special case and the rule wouldn't be applied there. And assuming it worked, I wouldn't worry too much about corners as most of these units are recording a point every second, and no one (especially kittehs) is likely to be turning fast enough to chop off a significant amount of the log. It's a manually applied filter after all, so not likely to cause any more problems than someone mis-using the current simplify filter. > I really think that for this class of GPSes that insists on recording points even when they basically know they're pullling coords out of their...hats that trusting their DOP is the thing to do. > If a GPS maker records made up coords and doesn't associate a DOP value with them, then shame on them. I'd like to agree, but the potential is that you can buy a cheap logger and "fix" it by applying some really powerful open source and free tools to the problem, rather than handing over hundreds of dollars for a "high quality" unit. > That said, I really would like to see a data set &/or code for a filter that could figure that that kitteh wasn't walking, teleported to the next continent one second, > then teleported back the next and could figure out that erroneous point was the bogus one. I had some luck finding points with unlikely speeds, then removing a few points either side and interpolating between the remaining points again. This was an extreme case with pleasing results: http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file Tom |
From: Jan M. <jan...@di...> - 2010-04-01 05:17:17
|
On Thu, Apr 1, 2010 at 7:07 AM, Tom Paton <tom...@gm...> wrote: > On Thu, Apr 1, 2010 at 3:35 PM, Robert Lipe <rob...@gp...> > wrote: > > > > > > On Wed, Mar 31, 2010 at 10:28 PM, Jan Martin < > jan...@di...> wrote: > >> > >> Robert, > >> > >> I wonder if it would be possible to have the "reverse" function to this: > >> Remove points that have the largest effect on the shape. > >> > >> Thanks, > >> > >> Jan > > > > For non-broken (and the OP already knows his tracks _are_ broken...) > wouldn't this be a Very Bad Thing? It would likely remove the endpoints, > and the most strategic turnpoints, for example. > > I think this sounds promising, but it will be a bigg challenge to > determine when to stop the algorithm: The existing simplify filter is > "self limiting" as it runs out of points that don't change the path > length significantly. Removing the points with the biggest effect can > proceed until there is nothing left... > I thought about having a "percentage" option: E.g. discard 5% of points that change the shape most. Of course one could also do statistics on the influence of all points and then automatically decide what percentage to remove. Jan > That said, obviously the end-points are a special case and the rule > wouldn't be applied there. > > And assuming it worked, I wouldn't worry too much about corners as > most of these units are recording a point every second, and no one > (especially kittehs) is likely to be turning fast enough to chop off a > significant amount of the log. It's a manually applied filter after > all, so not likely to cause any more problems than someone mis-using > the current simplify filter. > > > I really think that for this class of GPSes that insists on recording > points even when they basically know they're pullling coords out of > their...hats that trusting their DOP is the thing to do. > > If a GPS maker records made up coords and doesn't associate a DOP value > with them, then shame on them. > > I'd like to agree, but the potential is that you can buy a cheap > logger and "fix" it by applying some really powerful open source and > free tools to the problem, rather than handing over hundreds of > dollars for a "high quality" unit. > > > That said, I really would like to see a data set &/or code for a filter > that could figure that that kitteh wasn't walking, teleported to the next > continent one second, > > then teleported back the next and could figure out that erroneous point > was the bogus one. > > I had some luck finding points with unlikely speeds, then removing a > few points either side and interpolating between the remaining points > again. This was an extreme case with pleasing results: > http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file > > Tom > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > -- http://www.DIY-streetview.org |
From: Robert L. <rob...@gp...> - 2010-04-01 05:21:41
|
> For non-broken (and the OP already knows his tracks _are_ broken...) wouldn't this be a Very Bad Thing? It would likely remove the endpoints, and the most strategic turnpoints, for example. > > I think this sounds promising, but it will be a bigg challenge to > determine when to stop the algorithm: The existing simplify filter is > "self limiting" as it runs out of points that don't change the path > length significantly. Removing the points with the biggest effect can > proceed until there is nothing left... > > That said, obviously the end-points are a special case and the rule > wouldn't be applied there. > It's a hard problem that's been floated on the list several times. Present a patch (with test cases and analysis as what works well for a backpacker works badly for a pilot and vice versa) and share the love. > I really think that for this class of GPSes that insists on recording > points even when they basically know they're pullling coords out of > their...hats that trusting their DOP is the thing to do. > > If a GPS maker records made up coords and doesn't associate a DOP value > with them, then shame on them. > > I'd like to agree, but the potential is that you can buy a cheap > logger and "fix" it by applying some really powerful open source and > free tools to the problem, rather than handing over hundreds of > dollars for a "high quality" unit. > Right now, we have the "HQ" units that don't record flagrantly made up points and the $50 ones that do. About half of the $50 units (forgive me with the numbers. Substitute $Cheap if you like) don't associate *any* indication that the recorded fixes are bogus. I suspect that given a 4D point (lat, lon, alt, time) it should be possible to determine "zinger" points, but have not yet seen a formula that effectively determines that. > That said, I really would like to see a data set &/or code for a filter > that could figure that that kitteh wasn't walking, teleported to the next > continent one second, > > then teleported back the next and could figure out that erroneous point > was the bogus one. > > I had some luck finding points with unlikely speeds, then removing a > few points either side and interpolating between the remaining points > again. This was an extreme case with pleasing results: > http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file > > Humans tend to be good at figuring out which members of a set don't belong. Humans that can describe that process to a computer are rather more rare. This general process is probably the oldest unsolved problem on this list. A math nerd with a collection of broken data sets and an algorithm/implementation for "fixing" them has a chance to be a hero. |
From: Jan M. <jan...@di...> - 2010-04-01 05:28:27
|
> > Humans tend to be good at figuring out which members of a set don't > belong. Humans that can describe that process to a computer are rather more > rare. > > This general process is probably the oldest unsolved problem on this > list. A math nerd with a collection of broken data sets and an > algorithm/implementation for "fixing" them has a chance to be a hero. > I am not the type to "just wait for a hero". We need to do our part first. Lets start with collecting the broken data sets. Any suggestion where to collect them? With visualization on a map? Jan -- http://www.DIY-streetview.org |
From: <gps...@he...> - 2010-04-01 09:37:28
|
Den 01.04.10 07.28, skrev Jan Martin: > Lets start with collecting the broken data sets. > Any suggestion where to collect them? With visualization on a map? I can provide a broken data set from my i-gotU Travel Logger. Should I just post it here? Henrik |
From: <gps...@he...> - 2010-04-01 09:55:49
|
Den 01.04.10 07.07, skrev Tom Paton: > I had some luck finding points with unlikely speeds, then removing a > few points either side and interpolating between the remaining points > again. This was an extreme case with pleasing results: > http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file > Thanks for the tip. I tried uploading a track with faulty points to gpsloglabs, but it discarded a lot of valid points as it imported my file. I'm still better off with gpsbabel and the discard:sat/elemin/elemax options. Henrik |
From: John R. P. <jr...@gm...> - 2010-04-01 21:46:56
|
I had a thought a while ago trying to solve the same issue that if calculus were applied, we could filter on velocity, acceleration, rate of change of acceleration (ie v; v'; v'') and so on, and I recon that good results could be had. This would be best handled as vector data to give direction, and accurately handle negatives etc. The idea being that if kitteh is walking along a fence, then teleports 10 meters, then returns for the next second, his velocity jumps from a nice 0.3ms^-1 to 10ms^-1 for 2 seconds, then back to a cool 0.3ms^-1. His acceleration hovers around a cool 0ms^-2 as he is walking at a constant speed, then jumps to 10ms^-2 then 20ms^-2 (because vector data maintains direction) then 10ms^-2, then 0ms^-2. rate of change of acceleration goes equally haywire. It seems to tend that the more levels of rate of change you go into, the more smooth it gets for real life data, and the more haywire it goes with false data. this could be extended to filtering altitude; a'; a'';a''' and then bearing and a set of it's differentials. This filter could either be applied telling the user to select one variable, or it could use some additive system where a measure of how ridiculous each term was was added up. I'm pretty sure the zinger points would light up like a Christmas tree lights using this. Unfortunately the only tool I had handy for filtering was excel, and it just wasn't cutting it, so I gave up. Please also note that by the same mechanism above (thought experiment) some bright spark had the whole scientific community convinced for years that rocks fall faster than feathers. Now the rate of change of... JR On 1 April 2010 06:07, Tom Paton <tom...@gm...> wrote: > On Thu, Apr 1, 2010 at 3:35 PM, Robert Lipe <rob...@gp...> > wrote: > > > > > > On Wed, Mar 31, 2010 at 10:28 PM, Jan Martin < > jan...@di...> wrote: > >> > >> Robert, > >> > >> I wonder if it would be possible to have the "reverse" function to this: > >> Remove points that have the largest effect on the shape. > >> > >> Thanks, > >> > >> Jan > > > > For non-broken (and the OP already knows his tracks _are_ broken...) > wouldn't this be a Very Bad Thing? It would likely remove the endpoints, > and the most strategic turnpoints, for example. > > I think this sounds promising, but it will be a bigg challenge to > determine when to stop the algorithm: The existing simplify filter is > "self limiting" as it runs out of points that don't change the path > length significantly. Removing the points with the biggest effect can > proceed until there is nothing left... > > That said, obviously the end-points are a special case and the rule > wouldn't be applied there. > > And assuming it worked, I wouldn't worry too much about corners as > most of these units are recording a point every second, and no one > (especially kittehs) is likely to be turning fast enough to chop off a > significant amount of the log. It's a manually applied filter after > all, so not likely to cause any more problems than someone mis-using > the current simplify filter. > > > I really think that for this class of GPSes that insists on recording > points even when they basically know they're pullling coords out of > their...hats that trusting their DOP is the thing to do. > > If a GPS maker records made up coords and doesn't associate a DOP value > with them, then shame on them. > > I'd like to agree, but the potential is that you can buy a cheap > logger and "fix" it by applying some really powerful open source and > free tools to the problem, rather than handing over hundreds of > dollars for a "high quality" unit. > > > That said, I really would like to see a data set &/or code for a filter > that could figure that that kitteh wasn't walking, teleported to the next > continent one second, > > then teleported back the next and could figure out that erroneous point > was the bogus one. > > I had some luck finding points with unlikely speeds, then removing a > few points either side and interpolating between the remaining points > again. This was an extreme case with pleasing results: > http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file > > Tom > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > |
From: Dave P. <da...@dp...> - 2010-04-02 07:04:44
|
On Thu, 1 Apr 2010 22:46:28 +0100 John Robert Peterson <jr...@gm...> wrote: > I had a thought a while ago trying to solve the same issue that if > calculus were applied, we could filter on velocity, acceleration, > rate of change of acceleration (ie v; v'; v'') and so on, and I recon > that good results could be had. > Unfortunately the only tool I had handy for filtering was excel, and > it just wasn't cutting it, so I gave up. > > Please also note that by the same mechanism above (thought > experiment) some bright spark had the whole scientific community > convinced for years that rocks fall faster than feathers. Now the > rate of change of... Using an xml format, there is a library that has all the trig functions, for use with xslt 1.0 and 2.0, see http://fxsl.sourceforge.net/ -- regards -- Dave Pawson XSLT XSL-FO FAQ. http://www.dpawson.co.uk |
From: Robert L. <rob...@gp...> - 2010-04-04 19:13:32
|
On Thu, Apr 1, 2010 at 4:46 PM, John Robert Peterson <jr...@gm...>wrote: > I had a thought a while ago trying to solve the same issue that if calculus > were applied, we could filter on velocity, acceleration, rate of change of > acceleration (ie v; v'; v'') and so on, and I recon that good results could > be had. > > This would be best handled as vector data to give direction, and accurately > handle negatives etc. > > The idea being that if kitteh is walking along a fence, then teleports 10 > meters, then returns for the next second, his velocity jumps from a nice > 0.3ms^-1 to 10ms^-1 for 2 seconds, then back to a cool 0.3ms^-1. His > acceleration hovers around a cool 0ms^-2 as he is walking at a constant > speed, then jumps to 10ms^-2 then 20ms^-2 (because vector data maintains > direction) then 10ms^-2, then 0ms^-2. rate of change of acceleration goes > equally haywire. > > It seems to tend that the more levels of rate of change you go into, the > more smooth it gets for real life data, and the more haywire it goes with > false data. > > this could be extended to filtering altitude; a'; a'';a''' and then bearing > and a set of it's differentials. > > This filter could either be applied telling the user to select one > variable, or it could use some additive system where a measure of how > ridiculous each term was was added up. > > I'm pretty sure the zinger points would light up like a Christmas tree > lights using this. > > Unfortunately the only tool I had handy for filtering was excel, and it > just wasn't cutting it, so I gave up. > Yes, it's probably a mere matter of math. I've looked at several implementations and they all tend to do bad things with real-world examples like cars that stop and go or flights that start or end (Yes, the plane did average 200mph, but is really was going 4mph during taxi.) or joggers that really do make sharp switchbacks and such. Kalman filtering makes my brain bleed. > Please also note that by the same mechanism above (thought experiment) some > bright spark had the whole scientific community convinced for years that > rocks fall faster than feathers. Now the rate of change of... > > JR > > > On 1 April 2010 06:07, Tom Paton <tom...@gm...> wrote: > >> On Thu, Apr 1, 2010 at 3:35 PM, Robert Lipe <rob...@gp...> >> wrote: >> > >> > >> > On Wed, Mar 31, 2010 at 10:28 PM, Jan Martin < >> jan...@di...> wrote: >> >> >> >> Robert, >> >> >> >> I wonder if it would be possible to have the "reverse" function to >> this: >> >> Remove points that have the largest effect on the shape. >> >> >> >> Thanks, >> >> >> >> Jan >> > >> > For non-broken (and the OP already knows his tracks _are_ broken...) >> wouldn't this be a Very Bad Thing? It would likely remove the endpoints, >> and the most strategic turnpoints, for example. >> >> I think this sounds promising, but it will be a bigg challenge to >> determine when to stop the algorithm: The existing simplify filter is >> "self limiting" as it runs out of points that don't change the path >> length significantly. Removing the points with the biggest effect can >> proceed until there is nothing left... >> >> That said, obviously the end-points are a special case and the rule >> wouldn't be applied there. >> >> And assuming it worked, I wouldn't worry too much about corners as >> most of these units are recording a point every second, and no one >> (especially kittehs) is likely to be turning fast enough to chop off a >> significant amount of the log. It's a manually applied filter after >> all, so not likely to cause any more problems than someone mis-using >> the current simplify filter. >> >> > I really think that for this class of GPSes that insists on recording >> points even when they basically know they're pullling coords out of >> their...hats that trusting their DOP is the thing to do. >> > If a GPS maker records made up coords and doesn't associate a DOP value >> with them, then shame on them. >> >> I'd like to agree, but the potential is that you can buy a cheap >> logger and "fix" it by applying some really powerful open source and >> free tools to the problem, rather than handing over hundreds of >> dollars for a "high quality" unit. >> >> > That said, I really would like to see a data set &/or code for a filter >> that could figure that that kitteh wasn't walking, teleported to the next >> continent one second, >> > then teleported back the next and could figure out that erroneous point >> was the bogus one. >> >> I had some luck finding points with unlikely speeds, then removing a >> few points either side and interpolating between the remaining points >> again. This was an extreme case with pleasing results: >> http://blog.gpsloglabs.com/cleaning-up-a-bad-gps-log-file >> >> Tom >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Gpsbabel-misc mailing list http://www.gpsbabel.org >> Gps...@li... >> To unsubscribe, change list options, or see archives, visit: >> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc >> > > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc > > |