As indicated here (http://www.openstreetmap.org/user/Henri%20Willox/diary/13244 ) there is already a project called osm2pgsql. So this new project will lead to confusion. Your script is obviously based on osmosis. So please rename your project.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Thank you for your comment. I understand your concern. I have modified the page description to have: " It is NOT the same project as osm2pgsql" in the sourceforge summary page (this takes 40 characters of the 250 characters I am entitled to use). This is a good advertisement for osm2pgsql since this short sourceforge summary is syndicated/reproduced on many website. Similarly, I created obvious links to osm2pgsql and similar projects here: http://wiki.openstreetmap.org/wiki/Osm2postgresql
Still, I do not see why it should be confusing for people. There is a similar (or maybe higher) risk of confusion between Osm2pgsql and OSM2Postgres than between osm2postgresql and osm2pgsl.
Before creating the project, when searching for something like this, I searched for osm2postgresql then OSM2Postgres. For me osm2pgsql is not something I would search for. Today, if someone searches for osm2postgresql on Google, they will immediately become aware of osm2pgsql.
About your comment: "Your script is obviously based on osmosis" I do not see why it is relevant here. As a side note, the call to osmosis is only one line of my code, while my full code is more than 1000 lines. I very much like osmosis; still, I needed a bit more on top of it, and I'm just sharing it.
I am using Kubuntu and if you use Ubuntu yourself I guess you understand the difference between Kubuntu, Ubuntu and Xubuntu, even if there is only one letter more. All statisticians know the difference between the following software/languages: S, S+ and R. I hope this answers your concerns.
May I ask if you are somehow linked to the osm2pgsql project?
Regards,
Mayeul
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm trying a spin on your project. My goal is to plot whole brazil geodata avaiable on openstreemap and use it on qgis with a custom layer of mine. To be complete on the description of my problem i must stat that:
- all i have is a 2 lists of lat/long points
- I want to check how much these too lists intersect in a 50km radius area around them
- I know i could use standard postgis functions to give a numerical result on this but i need more, I need that plotted as a custom layer on top of brazil layer so the results are more easily checked.
I have a few suggestions and questions for you since i have already struggled with many issues and still haven't achieved my goal:
- we should separate sql code from bash code.
- how much time does it take to run the full sql code to create polygons from osm data? I have a vm running ubuntuserver + latest postgresql 9.1beta + postgis 1.5.2 + geos 3.2.2 + gdal 1.8. The script is running for more than 10 hours now and doesnt seem likely that it'll finish anytime soon. Is this huge delay expected from a brazil.osm which uncompressed takes 1.5gb?
- how can I effectively can use that data on qgis? Any special queries should be done on top of the tables avaiable?
So far I should thank you ALOT for making your work public, and huge advanced thanks for your time answering these questions.
Best Regards,
Israel Rodrigo Faria
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Im checking other sources and have to admit that i have underestimated the processing power needed for this. My VM have only 1GB to work with. Funny that swap partition isnt used..
Other sources show, as stated above the osm2pgsql way. it is avaiable as a package on ubuntu, how does your solution compare to theirs, thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi israelfaria,
Thanks a lot for your encouragements!
I'm doing this on my spare time (to customize my maps), so seeing that it is useful to others delights me!
On entire country it will take a while, but you will be satisfied with the result! Details below:
I tested it on entire italy.osm, took about one night. Memory is not necessarily the issue especially if you stay with the basic postgresql install, see: http://wiki.postgresql.org/wiki/Performance_Optimization http://www.revsys.com/writings/postgresql-performance.html
I encourage you to look at the tablespace option; I think disk access is a major issue and the VirtualMachine makes it probably worse.
However, if you have to do the processing only rarely and you can wait a bit, I'm not sure spending to much time to customize the postgresql parameters is worth it.
Some of the advantages of my solution are:
1- it gives an efficient data backed for GIS packages, especially if you want to customize the rendering. See here for details with qgis: http://www.qgis.org/wiki/Using_OpenStreetMap_data
I made some rendering which I found much nicer than Osmarender, without writing a line a xml stylesheet, see: http://www.qgis.org/wiki/OpenStreetMap_data_rendered_with_QGIS
2- It gives advance processing of complex polygons and relations
3- The bash is quite straightforward and the rest is pure sql: you can easily customize the script for your needs. If you do so, I suggest you test on small osm files first (I did my first tests with monaco.osm)
I am currently using one file only (bash and sql) as this is the most portable structure and the easiest to cstomize. WOuld be great to have a pure sql file, I agree, but then you will have to do all to other steps manually.
For the qgis rendering, when it is finished:
- in qgis, press the "add postgis layer" icon
- connect to the database, select the following layers: nodes_with_tags, ways, polygons
You can use any symbology based on the attributes. The most powerful is to use wildcard matching of the "tags" field: you have access to all tags! I will try to make a screencast of this.
Do not hesitate to keep asking or make comments.
Hope this helps,
Mayeul
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
this script is not very useful for an existing postgresql database with postgis.
The script wants to create the database in every case and the Parameter -U is a farce. It becomes overwritten by whoami and the called pgsql command does not get recognized by -U.
There is no support for osmosis 0.39 …
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi all
The way to use the QGIS style files is clearly shown on this youtube video.
For now, those styles are (C) Mayeul Kauffmann CC-by-SA, as well as derived works (similar to OSM maps and data): so if you use them simply mention it somewhere on an accessible part of your website (1 click away from map page is OK).
I am aware that providing password several times is cumbersome. Some of the tools launched from my script allow to specify the password on the command line, but this is not always safe (ps will show it to anybody), so I preferred to avoid it.
I was not aware of the -U bug and the "always try to create db" and I will try to fix this soon. As well as migrating to osmosis 0.39 (but they changed some parameter names, grrrr).
PS: Please, open new discussions for you topics!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Thanks all for your comments.
Version 05rc2 should solve most (all?) of the issues mentioned here and brings other major improvements. I have tested it on full continents, successfully. Please have a try and give your feedback!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Welcome to Open Discussion
As indicated here (http://www.openstreetmap.org/user/Henri%20Willox/diary/13244 ) there is already a project called osm2pgsql. So this new project will lead to confusion. Your script is obviously based on osmosis. So please rename your project.
Hi,
Thank you for your comment. I understand your concern. I have modified the page description to have: " It is NOT the same project as osm2pgsql" in the sourceforge summary page (this takes 40 characters of the 250 characters I am entitled to use). This is a good advertisement for osm2pgsql since this short sourceforge summary is syndicated/reproduced on many website. Similarly, I created obvious links to osm2pgsql and similar projects here:
http://wiki.openstreetmap.org/wiki/Osm2postgresql
Still, I do not see why it should be confusing for people. There is a similar (or maybe higher) risk of confusion between Osm2pgsql and OSM2Postgres than between osm2postgresql and osm2pgsl.
Before creating the project, when searching for something like this, I searched for osm2postgresql then OSM2Postgres. For me osm2pgsql is not something I would search for. Today, if someone searches for osm2postgresql on Google, they will immediately become aware of osm2pgsql.
About your comment: "Your script is obviously based on osmosis" I do not see why it is relevant here. As a side note, the call to osmosis is only one line of my code, while my full code is more than 1000 lines. I very much like osmosis; still, I needed a bit more on top of it, and I'm just sharing it.
I am using Kubuntu and if you use Ubuntu yourself I guess you understand the difference between Kubuntu, Ubuntu and Xubuntu, even if there is only one letter more. All statisticians know the difference between the following software/languages: S, S+ and R. I hope this answers your concerns.
May I ask if you are somehow linked to the osm2pgsql project?
Regards,
Mayeul
Hi,
I'm trying a spin on your project. My goal is to plot whole brazil geodata avaiable on openstreemap and use it on qgis with a custom layer of mine. To be complete on the description of my problem i must stat that:
- all i have is a 2 lists of lat/long points
- I want to check how much these too lists intersect in a 50km radius area around them
- I know i could use standard postgis functions to give a numerical result on this but i need more, I need that plotted as a custom layer on top of brazil layer so the results are more easily checked.
I have a few suggestions and questions for you since i have already struggled with many issues and still haven't achieved my goal:
- we should separate sql code from bash code.
- how much time does it take to run the full sql code to create polygons from osm data? I have a vm running ubuntuserver + latest postgresql 9.1beta + postgis 1.5.2 + geos 3.2.2 + gdal 1.8. The script is running for more than 10 hours now and doesnt seem likely that it'll finish anytime soon. Is this huge delay expected from a brazil.osm which uncompressed takes 1.5gb?
- how can I effectively can use that data on qgis? Any special queries should be done on top of the tables avaiable?
So far I should thank you ALOT for making your work public, and huge advanced thanks for your time answering these questions.
Best Regards,
Israel Rodrigo Faria
Im checking other sources and have to admit that i have underestimated the processing power needed for this. My VM have only 1GB to work with. Funny that swap partition isnt used..
Other sources show, as stated above the osm2pgsql way. it is avaiable as a package on ubuntu, how does your solution compare to theirs, thanks.
Hi israelfaria,
Thanks a lot for your encouragements!
I'm doing this on my spare time (to customize my maps), so seeing that it is useful to others delights me!
On entire country it will take a while, but you will be satisfied with the result! Details below:
I tested it on entire italy.osm, took about one night. Memory is not necessarily the issue especially if you stay with the basic postgresql install, see:
http://wiki.postgresql.org/wiki/Performance_Optimization
http://www.revsys.com/writings/postgresql-performance.html
I encourage you to look at the tablespace option; I think disk access is a major issue and the VirtualMachine makes it probably worse.
However, if you have to do the processing only rarely and you can wait a bit, I'm not sure spending to much time to customize the postgresql parameters is worth it.
Some of the advantages of my solution are:
1- it gives an efficient data backed for GIS packages, especially if you want to customize the rendering. See here for details with qgis:
http://www.qgis.org/wiki/Using_OpenStreetMap_data
I made some rendering which I found much nicer than Osmarender, without writing a line a xml stylesheet, see:
http://www.qgis.org/wiki/OpenStreetMap_data_rendered_with_QGIS
2- It gives advance processing of complex polygons and relations
3- The bash is quite straightforward and the rest is pure sql: you can easily customize the script for your needs. If you do so, I suggest you test on small osm files first (I did my first tests with monaco.osm)
I am currently using one file only (bash and sql) as this is the most portable structure and the easiest to cstomize. WOuld be great to have a pure sql file, I agree, but then you will have to do all to other steps manually.
For the qgis rendering, when it is finished:
- in qgis, press the "add postgis layer" icon
- connect to the database, select the following layers: nodes_with_tags, ways, polygons
You can use any symbology based on the attributes. The most powerful is to use wildcard matching of the "tags" field: you have access to all tags! I will try to make a screencast of this.
Do not hesitate to keep asking or make comments.
Hope this helps,
Mayeul
I have put a video on how to use osm2postgresql and QGIS together here:
http://www.youtube.com/watch?v=NBBYtH2svw0
this script is not very useful for an existing postgresql database with postgis.
The script wants to create the database in every case and the Parameter -U is a farce. It becomes overwritten by whoami and the called pgsql command does not get recognized by -U.
There is no support for osmosis 0.39 …
Hi, I tried out the script and found it quite helpful, thany for the good work!!
The only issues (version 4) are:
-can't change user name with -U option
-have to provide password several times during the process, this is very annoying
Questions I have:
You are also providing QGIS style files , could you exactly document how to use them with the layers from Postgis…
Under which license are the style files?
greetings!
Hi all
The way to use the QGIS style files is clearly shown on this youtube video.
For now, those styles are (C) Mayeul Kauffmann CC-by-SA, as well as derived works (similar to OSM maps and data): so if you use them simply mention it somewhere on an accessible part of your website (1 click away from map page is OK).
I am aware that providing password several times is cumbersome. Some of the tools launched from my script allow to specify the password on the command line, but this is not always safe (ps will show it to anybody), so I preferred to avoid it.
I was not aware of the -U bug and the "always try to create db" and I will try to fix this soon. As well as migrating to osmosis 0.39 (but they changed some parameter names, grrrr).
PS: Please, open new discussions for you topics!
Hi,
Thanks all for your comments.
Version 05rc2 should solve most (all?) of the issues mentioned here and brings other major improvements. I have tested it on full continents, successfully. Please have a try and give your feedback!