Thread: [Gpsbabel-misc] position,distance=10m problem
Brought to you by:
robertl
From: Dr. J. W. G. <gle...@dr...> - 2011-06-24 05:02:05
Attachments:
gpsbabel_filtering_distance=10m_problem.gpx
|
Running the latest Linux version (compiled from source today), using position,distance=10m to filter a file produces erroneous (so far as I understand what the filter is supposed to do based on the documentation) results, notably with all 46 points in one 1264 ft segment being deleted. Don't know if attachments get through to this list, but am attaching gpx file with original and filtered versions. (Got same result using distance=30f). |
From: Roger <ar...@ma...> - 2011-06-24 11:34:41
|
In http://wiki.openstreetmap.org/wiki/Using_filters_with_GPSBabel I found this comment: Remove duplicates and nearby points filter To remove duplicates points use a filter of -x duplicate,location To remove points that are closer than 5 meters to the proceding point use a filter of Ambox warning pn.svg <http://wiki.openstreetmap.org/wiki/File:Ambox_warning_pn.svg> All current gpsbabel versions seem to suffer from a data loss bug in /position/ filter (http://sourceforge.net/mailarchive/message.php?msg_id=27158444) -x position,distance=5m On 11-06-24 12:48 AM, Dr. John W. Glendening wrote: > Running the latest Linux version (compiled from source today), using > position,distance=10m to filter a file produces erroneous (so far as I > understand what the filter is supposed to do based on the documentation) > results, notably with all 46 points in one 1264 ft segment being deleted. > Don't know if attachments get through to this list, but am attaching gpx file > with original and filtered versions. (Got same result using distance=30f). > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense.. > http://p.sf.net/sfu/splunk-d2d-c1 > > > _______________________________________________ > Gpsbabel-misc mailing list http://www.gpsbabel.org > Gps...@li... > To unsubscribe, change list options, or see archives, visit: > https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc |
From: Robert L. <rob...@gp...> - 2011-06-24 14:10:35
|
On Thu, Jun 23, 2011 at 11:48 PM, Dr. John W. Glendening < gle...@dr...> wrote: > Running the latest Linux version (compiled from source today), using > position,distance=10m to filter a file produces erroneous (so far as I > understand what the filter is supposed to do based on the documentation) > results, notably with all 46 points in one 1264 ft segment being deleted. > Don't know if attachments get through to this list, but am attaching gpx > file > with original and filtered versions. (Got same result using distance=30f). You didn't include the command you issued, but I note that this is a track and the position filter is thus the wrong tool for this task. Use the simplify filter to reduce the number of points in a track. http://www.gpsbabel.org/htmldoc-development/filter_simplify.html RJL |
From: Dr. J. W. G. <gle...@dr...> - 2011-06-28 16:45:01
Attachments:
FilterComparison-LauntzCreekHike.gpx
|
Thanks for that response. Don't know what you meant by "you didn't include the command you used" since the filtering was the only non-input/output command supplied to gpsbabel, but in any case you understand the filtering I was trying to use. Reading the documentation I did not understand that the "position" filter was intended to be used only with single points (not tracks), especially since the documentation page includes the statement "This is useful if you have multiple tracks of the same course and you'd like the filter to consider the tracks the same". I was trying to simplify my track, but my intent was to try to do it via a multiple filter process. My track was for a very slow hike (not a clear path, with vegetation/downfall which needed to be negotiation) and using a GPS subject to "spidering", so I hoped to get a better result by first filtering out many nearby points (i.e. filtering out the spidering) and then perform a second filtering using a different criterion, and was hoping to use gpsbabel for both steps. Finding that was not possible, I went ahead and created my own filter to do such (sorting on distance between points, combining the lowest-distance pair into a single point, then re-calculating distances and re-sorting and re- iterating) for a filter distance of 5 m, then ran the result through the gpsbabel's crosstrack filter. I'm attaching a gpx file with the original track, the track multiple-filtered as described (green in MapSource), and the original track single-filtered using gpsbabel's crosstrack filter, i.e. without the pre-filtering, using a parameter value yielding the same number of points as for the multiple-filter case (red in MapSource) so a valid comparison could be made. Though beauty may be in the eye of the beholder, to me the multiple-filter result is superior to the single-filter one, notably in not having as many apparent track reversals as appear in the crosstrack-only result, and hence is a better treatment of such cases. Jack On Friday, June 24, 2011 07:10:26 am Robert Lipe wrote: > > Running the latest Linux version (compiled from source today), using > > position,distance=10m to filter a file produces erroneous (so far as I > > understand what the filter is supposed to do based on the documentation) > > results, notably with all 46 points in one 1264 ft segment being deleted. > > Don't know if attachments get through to this list, but am attaching gpx > > file > > with original and filtered versions. (Got same result using > > distance=30f). > > You didn't include the command you issued, but I note that this is a track > and the position filter is thus the wrong tool for this task. Use the > simplify filter to reduce the number of points in a track. > > http://www.gpsbabel.org/htmldoc-development/filter_simplify.html > > RJL -- John W. (Jack) Glendening 831-484-6929 25953 Deer Run Lane Salinas, CA 93908 |
From: Rich <ri...@na...> - 2011-06-25 15:37:37
|
On 06/24/11 07:48, Dr. John W. Glendening wrote: > Running the latest Linux version (compiled from source today), using > position,distance=10m to filter a file produces erroneous (so far as I > understand what the filter is supposed to do based on the documentation) > results, notably with all 46 points in one 1264 ft segment being deleted. > Don't know if attachments get through to this list, but am attaching gpx file > with original and filtered versions. (Got same result using distance=30f). that seems to be my experience as well - distance filter removes more points than it should. i like gpsbabel and surely it's developer(s ?) are good at coding - unfortunately, there seems to be lack of interest in this problem :) as you've noticed, there was a claim that another mode should be used, but, unfortunately, that other mode (simplify) does not allow to only remove points based on distance, not nuking points on straight segments - at least i did not find a way to do so. i don't use gpsbabel in a commercial project but only to clean up traces before uploading them to www.openstreetmap.org, thus i don't have a budget to pay somebody to fix this. i can promise to buy some beers to the person who will do that, but that would require some chance of a physical meetup :) -- Rich |
From: Kacper P. <kac...@gm...> - 2011-06-25 16:26:48
|
Rich, On 25 June 2011 17:37, Rich <ri...@na...> wrote: > [...] I'm an OSM participant. http://www.openstreetmap.org/user/Kacper%20Perschke I'd suggest upload the traces untouched. JOSM enables the editor to convert gpx to the data layer, cut to a particular segments, delete really sick points and simplify the rest of the segment. It's simplifying algorithm is rather simple and bases on some heuristics but is quite good on the task was designed for. I swear. :) Greets KAcper Perschke |
From: Rich <ri...@na...> - 2011-06-25 19:39:43
|
On 06/25/11 19:26, Kacper Perschke wrote: > Rich, > > On 25 June 2011 17:37, Rich<ri...@na...> wrote: >> [...] > > I'm an OSM participant. http://www.openstreetmap.org/user/Kacper%20Perschke > > I'd suggest upload the traces untouched. JOSM enables the editor to > convert gpx to the data layer, cut to a particular segments, delete > really sick points and simplify the rest of the segment. It's > simplifying algorithm is rather simple and bases on some heuristics > but is quite good on the task was designed for. I swear. :) oh, that was part of my normal workflow - distance filter with gpsbabel, manual cleanup in josm. but with lots of traces it is very time consuming to clean traces up properly - all blobs when not moving around etc... i'm being a bit pedantic regarding what i upload to osm, thus i'd like to clean those traces up (i log every second so the trace can end up with lots of useless points). but that's a bit of an offtopic here ;) > Greets > KAcper Perschke -- Rich |
From: Rich <ri...@na...> - 2011-06-28 17:00:22
|
On 06/28/11 19:31, Dr. John W. Glendening wrote: > Thanks for that response. Don't know what you meant by "you didn't include > the command you used" since the filtering was the only non-input/output > command supplied to gpsbabel, but in any case you understand the filtering I > was trying to use. Reading the documentation I did not understand that the > "position" filter was intended to be used only with single points (not > tracks), especially since the documentation page includes the statement "This > is useful if you have multiple tracks of the same course and you'd like the > filter to consider the tracks the same". I was trying to simplify my track, > but my intent was to try to do it via a multiple filter process. My track was > for a very slow hike (not a clear path, with vegetation/downfall which needed > to be negotiation) and using a GPS subject to "spidering", so I hoped to get a > better result by first filtering out many nearby points (i.e. filtering out > the spidering) and then perform a second filtering using a different > criterion, and was hoping to use gpsbabel for both steps. > > Finding that was not possible, I went ahead and created my own filter to do > such (sorting on distance between points, combining the lowest-distance pair > into a single point, then re-calculating distances and re-sorting and re- > iterating) for a filter distance of 5 m, then ran the result through the how exactly did you implement the filter ? i'm still quite interested in a working version of such a filter :) > gpsbabel's crosstrack filter. I'm attaching a gpx file with the original > track, the track multiple-filtered as described (green in MapSource), and the > original track single-filtered using gpsbabel's crosstrack filter, i.e. > without the pre-filtering, using a parameter value yielding the same number of > points as for the multiple-filter case (red in MapSource) so a valid > comparison could be made. Though beauty may be in the eye of the beholder, > to me the multiple-filter result is superior to the single-filter one, > notably in not having as many apparent track reversals as appear in the > crosstrack-only result, and hence is a better treatment of such cases. > > Jack > > > On Friday, June 24, 2011 07:10:26 am Robert Lipe wrote: >>> Running the latest Linux version (compiled from source today), using >>> position,distance=10m to filter a file produces erroneous (so far as I >>> understand what the filter is supposed to do based on the documentation) >>> results, notably with all 46 points in one 1264 ft segment being deleted. >>> Don't know if attachments get through to this list, but am attaching gpx >>> file >>> with original and filtered versions. (Got same result using >>> distance=30f). >> >> You didn't include the command you issued, but I note that this is a track >> and the position filter is thus the wrong tool for this task. Use the >> simplify filter to reduce the number of points in a track. >> >> http://www.gpsbabel.org/htmldoc-development/filter_simplify.html >> >> RJL -- Rich |
From: Dr. J. W. G. <gle...@dr...> - 2011-06-29 04:58:13
|
On Tuesday, June 28, 2011 10:00:11 am Rich wrote: > how exactly did you implement the filter ? i'm still quite interested in > a working version of such a filter :) If you mean software-wise, I wrote a script to read/filter/write a file using Perl, since I have the Perl program installed on my PC. If you mean methodology, I outlined that in my last post though I did not go into my method of reducing a pair point to a single point - I first took the simplest approach of simply discarding one point (the one having the nearest non-pair neighbor), then went to averaging them, finally ended up doing a weighed average with the weight being the number of points previously combined into an existing point, so the location of a point resulting from multiple combinations will be weighted more than an original point (perhaps gilding the lilly). Jack |
From: Rich <ri...@na...> - 2011-06-29 21:17:41
|
On 06/29/11 07:58, Dr. John W. Glendening wrote: > On Tuesday, June 28, 2011 10:00:11 am Rich wrote: >> how exactly did you implement the filter ? i'm still quite interested in >> a working version of such a filter :) > > If you mean software-wise, I wrote a script to read/filter/write a file using > Perl, since I have the Perl program installed on my PC. If you mean > methodology, I outlined that in my last post though I did not go into my > method of reducing a pair point to a single point - I first took the simplest > approach of simply discarding one point (the one having the nearest non-pair > neighbor), then went to averaging them, finally ended up doing a weighed > average with the weight being the number of points previously combined into an > existing point, so the location of a point resulting from multiple > combinations will be weighted more than an original point (perhaps gilding the > lilly). that sounds fairly advanced to me :) do you mind sharing the script, or maybe publishing it under some opensource licence or so ? > Jack -- Rich |
From: Dr. J. W. G. <gle...@dr...> - 2011-07-01 14:32:20
|
You can view and download the script at http://www.drjack.info/PUBLIC/MISC/gpx_filter.pl and are welcome to use it, but the programming is rough - I don't know how much I'll be using the script so just wanted to quickly get something that worked. The script must be taken "as is" - I do not want to be bothered with further questions about it, requests for support, etc.! FYI while it can do 3 different types of filtering, the "crosstrack" filter simply uses the gpsbabel algorithm via an external call to an installed gpsbabel executable - that would likely need some changes if run on a Windows system. For that matter, I've only run the script on a Linux machine so can't really say anything about it's use on a Windows system with Perl installed, other that in theory that it should work for the "point distance" filtering since that does not rely on any external programs. Jack On Wednesday, June 29, 2011 02:17:30 pm Rich wrote: > On 06/29/11 07:58, Dr. John W. Glendening wrote: > > On Tuesday, June 28, 2011 10:00:11 am Rich wrote: > >> how exactly did you implement the filter ? i'm still quite interested in > >> a working version of such a filter :) > > > > If you mean software-wise, I wrote a script to read/filter/write a file > > using Perl, since I have the Perl program installed on my PC. If you > > mean methodology, I outlined that in my last post though I did not go > > into my method of reducing a pair point to a single point - I first took > > the simplest approach of simply discarding one point (the one having the > > nearest non-pair neighbor), then went to averaging them, finally ended > > up doing a weighed average with the weight being the number of points > > previously combined into an existing point, so the location of a point > > resulting from multiple combinations will be weighted more than an > > original point (perhaps gilding the lilly). > > that sounds fairly advanced to me :) > do you mind sharing the script, or maybe publishing it under some > opensource licence or so ? > > > Jack -- John W. (Jack) Glendening 831-484-6929 25953 Deer Run Lane Salinas, CA 93908 |
From: Rich <ri...@na...> - 2011-07-03 09:57:47
|
On 07/01/11 17:31, Dr. John W. Glendening wrote: > You can view and download the script at > > http://www.drjack.info/PUBLIC/MISC/gpx_filter.pl > > and are welcome to use it, but the programming is rough - I don't know > how much I'll be using the script so just wanted to quickly get > something that worked. The script must be taken "as is" - I do not > want to be bothered with further questions about it, requests for > support, etc.! FYI while it can do 3 different types of filtering, > the "crosstrack" filter simply uses the gpsbabel algorithm via an > external call to an installed gpsbabel executable - that would likely > need some changes if run on a Windows system. For that matter, I've > only run the script on a Linux machine so can't really say anything > about it's use on a Windows system with Perl installed, other that in > theory that it should work for the "point distance" filtering since > that does not rely on any external programs. i hope feedback is not prohibited ;) awesome, it does seem to work and does not drop any points it should not, at least as far as i have tested. i did have to make a few modifications, though (and as i don't code, they're even more rough than the code itself :) ). hoping this might help somebody else : 1. i made sure to set these options once ;) (as they were the only ones different from the defaults i was interested at the time) : $pointdistance_filter_m = 2. ; $crosstrack_filter_m = 0. ; i also changed $LMETHOD_POINTDISTANCE = 1 ; to be able to visually verify the result 2. it failed on my input file with a fairly cryptic error message, which i eventually traced down to my file containing no ele information, thus i commented out the following : # if( $nfilept != $nele ). # {. # print "ERROR EXIT for track $ntrk - nfilept = $nfilept != nele = $nele \n" ; # sleep 10;. # exit ; # } i could have run gpx through gpsbabel to add fake ele values, but that seemed a bit too awful of a solution 3. after that it seemed to run, but the resulting track didn't have time entries... but there was 'ele' entry that was the first part of the timestamp ;) so just to get quickly around this i modified "if( $line =~ m| lon=\"([0-9\.\-]+)\"| )" block and right after "$values[$nfilept] .= ',' . $1 ;" i added "$values[$nfilept] .= ',0' ;" that seemed to do the trick (of inserting '0' ele values). --------------------------- i'd still want to remove fake ele values and looks like i'll have to remove '\r' in "sub write_trkpt" to get rid of those pesky windows newlines in the output file, but in the worst case that's a sed run on the output file anyway :) thanks again. > Jack > > > On Wednesday, June 29, 2011 02:17:30 pm Rich wrote: >> On 06/29/11 07:58, Dr. John W. Glendening wrote: >>> On Tuesday, June 28, 2011 10:00:11 am Rich wrote: >>>> how exactly did you implement the filter ? i'm still quite interested in >>>> a working version of such a filter :) >>> >>> If you mean software-wise, I wrote a script to read/filter/write a file >>> using Perl, since I have the Perl program installed on my PC. If you >>> mean methodology, I outlined that in my last post though I did not go >>> into my method of reducing a pair point to a single point - I first took >>> the simplest approach of simply discarding one point (the one having the >>> nearest non-pair neighbor), then went to averaging them, finally ended >>> up doing a weighed average with the weight being the number of points >>> previously combined into an existing point, so the location of a point >>> resulting from multiple combinations will be weighted more than an >>> original point (perhaps gilding the lilly). >> >> that sounds fairly advanced to me :) >> do you mind sharing the script, or maybe publishing it under some >> opensource licence or so ? >> >>> Jack -- Rich |
From: Rich <ri...@na...> - 2011-07-04 06:36:46
|
On 07/03/11 12:57, Rich wrote: > On 07/01/11 17:31, Dr. John W. Glendening wrote: >> You can view and download the script at >> >> http://www.drjack.info/PUBLIC/MISC/gpx_filter.pl >> >> and are welcome to use it, but the programming is rough - I don't know >> how much I'll be using the script so just wanted to quickly get >> something that worked. The script must be taken "as is" - I do not >> want to be bothered with further questions about it, requests for >> support, etc.! FYI while it can do 3 different types of filtering, >> the "crosstrack" filter simply uses the gpsbabel algorithm via an >> external call to an installed gpsbabel executable - that would likely >> need some changes if run on a Windows system. For that matter, I've >> only run the script on a Linux machine so can't really say anything >> about it's use on a Windows system with Perl installed, other that in >> theory that it should work for the "point distance" filtering since >> that does not rely on any external programs. > > i hope feedback is not prohibited ;) > awesome, it does seem to work and does not drop any points it should > not, at least as far as i have tested. i did have to make a few > modifications, though (and as i don't code, they're even more rough than > the code itself :) ). > > hoping this might help somebody else : > > 1. i made sure to set these options once ;) (as they were the only ones > different from the defaults i was interested at the time) : > > $pointdistance_filter_m = 2. ; > $crosstrack_filter_m = 0. ; > > i also changed $LMETHOD_POINTDISTANCE = 1 ; to be able to visually > verify the result > > 2. it failed on my input file with a fairly cryptic error message, which > i eventually traced down to my file containing no ele information, thus > i commented out the following : > > # if( $nfilept != $nele ). > # {. > # print "ERROR EXIT for track $ntrk - nfilept = $nfilept != nele = $nele > \n" ; > # sleep 10;. > # exit ; > # } > > i could have run gpx through gpsbabel to add fake ele values, but that > seemed a bit too awful of a solution > > 3. after that it seemed to run, but the resulting track didn't have time > entries... but there was 'ele' entry that was the first part of the > timestamp ;) > so just to get quickly around this i modified "if( $line =~ m| > lon=\"([0-9\.\-]+)\"| )" block and right after "$values[$nfilept] .= ',' > . $1 ;" i added "$values[$nfilept] .= ',0' ;" > > that seemed to do the trick (of inserting '0' ele values). > > --------------------------- > > i'd still want to remove fake ele values and looks like i'll have to > remove '\r' in "sub write_trkpt" to get rid of those pesky windows > newlines in the output file, but in the worst case that's a sed run on > the output file anyway :) just more feedback, not asking for any fixes ;) 4. it seemed to fail on a larger file with one of the tracks containing one point : Illegal division by zero at gpx_filter.pl line 198, <INFILE> line 44210. so in line 198 i just removed the "-1" part... hopefully that won't have any negative impact. 5. it also seemed to print out values from the previous track if nothing was done on the current one (probably should reset noutpt somewhere, but it's a bit too late for me). seemed to be purely cosmetic problem (hopefully i didn't break it myself before :) ) 20110619-2.gpx : trk= 2 pts= 2015 len= 13090 m avg.incr= 6.5 m PointDistanceFilter1=2m : trk= 2 pts= 1198 len= 13034 m avg.incr= 10.9 m 20110619-2.gpx : trk= 3 pts= 1 len= 0 m avg.incr= 0.0 m PointDistanceFilter1=2m : trk= 3 pts= 1198 len= 20530 m avg.incr= 17.2 m > thanks again. > >> Jack >> >> >> On Wednesday, June 29, 2011 02:17:30 pm Rich wrote: >>> On 06/29/11 07:58, Dr. John W. Glendening wrote: >>>> On Tuesday, June 28, 2011 10:00:11 am Rich wrote: >>>>> how exactly did you implement the filter ? i'm still quite >>>>> interested in >>>>> a working version of such a filter :) >>>> >>>> If you mean software-wise, I wrote a script to read/filter/write a file >>>> using Perl, since I have the Perl program installed on my PC. If you >>>> mean methodology, I outlined that in my last post though I did not go >>>> into my method of reducing a pair point to a single point - I first >>>> took >>>> the simplest approach of simply discarding one point (the one having >>>> the >>>> nearest non-pair neighbor), then went to averaging them, finally ended >>>> up doing a weighed average with the weight being the number of points >>>> previously combined into an existing point, so the location of a point >>>> resulting from multiple combinations will be weighted more than an >>>> original point (perhaps gilding the lilly). >>> >>> that sounds fairly advanced to me :) >>> do you mind sharing the script, or maybe publishing it under some >>> opensource licence or so ? >>> >>>> Jack -- Rich |
From: Dr. J. W. G. <gle...@dr...> - 2011-07-04 04:35:37
|
On Sunday, July 03, 2011 02:57:29 am you wrote: > ... thanks again. Glad to hear the script is being used. I don't have any gpx files without <ele> data so the script did not anticipate that case. I made a few modifications so the script currently there does allow for files without <ele> and/or <time> data. Jack |
From: Rich <ri...@na...> - 2011-07-07 10:17:44
|
On 07/04/11 07:35, Dr. John W. Glendening wrote: > On Sunday, July 03, 2011 02:57:29 am you wrote: >> ... thanks again. > > Glad to hear the script is being used. I don't have any gpx files without > <ele> data so the script did not anticipate that case. I made a few > modifications so the script currently there does allow for files without<ele> > and/or<time> data. great. looks like what i thought was just an output problem is an actual problem with data being written out as well, though : 20110305.gpx : trk= 1 pts= 31 len= 0 m avg.incr= 0.0 m PointDistanceFilter1=2m : trk= 1 pts= 3 len= 0 m avg.incr= 0.0 m 20110305.gpx : trk= 2 pts= 1414 len= 27313 m avg.incr= 19.3 m PointDistanceFilter1=2m : trk= 2 pts= 1291 len= 27298 m avg.incr= 21.2 m 20110305.gpx : trk= 3 pts= 8 len= 138 m avg.incr= 17.3 m PointDistanceFilter1=2m : trk= 3 pts= 1297 len= 61312 m avg.incr= 47.3 m in the end, resulting gpx has multiple tracks overlapping (in one testcase i got sections with 5 overlapping tracks :) ) not that i understand what's going on, but i'll try to look into this some... > Jack -- Rich |