#6 osmosis might fail if nodeLocationStoreType="TempFile"

[Mayeul]

(An OSM contributer (kfj) reported this to me):
I ran osm2postgresql, and at the point when the call to osmosis happens, the process would fail with a lot of lengthy error messages (in the log file). I tried the isolated osmosis call with modified parameters, and found that leaving out nodeLocationStoreType="TempFile" made the problem go away. So without this parameter, the osmosis default is used, and that runs just fine.

Any reason you use this parameter? When I tried it, my system would produce heavy disk activity for about a minute and then fail saying that the read-xml task had failed.

I'm using Kubuntu 11.4 on a 32bit intel system.

I also think it might be a nice addition to your script if you let it output some messages as well - even if it's only a little message saying if it has succeeded or failed.

So now I can finally put OSM data as a postgis layer on top of the contours I've made

[Mayeul]

The nodeLocationStoreType="TempFile" is aboslutely required for large files (europe.osm...) and may be necessary for systems with low memory. If you put on memory with low memory system, it will go to cache anyway but with lower performance.
Since the bug is only related to osmosis, it may be better to search for help in the osmosis community. Still reporting the bug here may help someone else. Could you post your "lot of lengthy error messages (in the log file)" here? (or the ones you identify as relevant).

Anyway, I'm glad I use bash shells: it helps users tweaks my code for their own needs or to solve their own problems, and that's why I choose that solution over compiled code.

(for the output of messages, this is hard as I redirect everything to log - depending on version of osm2postgresql - proposals welcome, maybe in another thread).

Mayeul

[Anonymous]

according to osmosis 0.40 documentation, the default for the parameter in question is "CompactTempFile" , which sounds like a better option than "TempFile" (at least it does to me ;-) - and not "InMemory", which would indeed be a problem for large data sets.

I just managed to import piemonte.pbf into postgres using this sequence (I have database superuser privileges attached to my database user account, which is in the same name as my system account):

# first set up the database:

dbname=piemonte

osmfile=$dbname.osm
pbffile=$dbname.pbf

createdb $dbname
createlang plpgsql $dbname

psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d $dbname
psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql -d $dbname
psql -d $dbname -f /usr/share/postgresql/8.4/contrib/hstore-new.sql
psql -d $dbname -f /usr/share/postgresql/8.4/contrib/adminpack.sql

# now let osm2postgresql (with the modified parameter) do the rest :

./osm2postgresql_05rc4.sh --dbname $dbname --pbf --file $pbffile

I suppose letting osm2postgresql set up the database would also work, since the code is practically the same in your script.

Import took roughly a quarter of an hour.

The log file with the original parameter looks just fine up to the point where all the osmosis tasks are launched:

...
30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager execute
FEIN: Launching task 2-buffer in a new thread.
30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager execute
FEIN: Task 3-write-pgsql-dump is passive, no execution required.
30.04.2012 09:15:20 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
FEIN: Waiting for task 1-read-xml to complete.

It remains like this for several minutes (with a test file containing only a few hundred nodes!), while there is heavy dis use, I can't tell why or where, but the only process seemingly doing anything is java. Then the process stops, and the log file continues like this:

SCHWERWIEGEND: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error has occurred, aborting.
at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:78)
at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:134)
at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:184)
at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.process(EntityBuffer.java:38)
at org.openstreetmap.osmosis.xml.v0_6.impl.NodeElementProcessor.end(NodeElementProcessor.java:118)
at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:108)
at java.lang.Thread.run(Thread.java:679)
30.04.2012 09:21:11 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
FEIN: Waiting for task 2-buffer to complete.
30.04.2012 09:21:14 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SCHWERWIEGEND: Thread for task 2-buffer failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to write node location data to node storage file /tmp/nodelatlon3486110848548710927.tmp.
at org.openstreetmap.osmosis.pgsnapshot.common.PersistentNodeLocationStore.addLocation(PersistentNodeLocationStore.java:170)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.WayGeometryBuilder.addNodeLocation(WayGeometryBuilder.java:64)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetBuilder.process(CopyFilesetBuilder.java:157)
at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetBuilder.process(CopyFilesetBuilder.java:115)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDumpWriter.process(PostgreSqlDumpWriter.java:58)
at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.run(EntityBuffer.java:74)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.openstreetmap.osmosis.pgsnapshot.common.PersistentNodeLocationStore.addLocation(PersistentNodeLocationStore.java:156)
... 7 more
30.04.2012 09:21:14 org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager waitForCompletion
FEIN: Task 3-write-pgsql-dump is passive, no completion wait required.
30.04.2012 09:21:14 org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Error 1. The command which triggered the error was:
./$osmosis_version/bin/osmosis -v --read-$filetype file="$osmfile" --buffer --write-pgsql-dump enableBboxBuilder=yes enableLinestringBuilder=yes nodeLocationStoreType="TempFile" directory="$temporarydir"
on line 438 of the osm2postgresql script.
## To delete the imported or temporary data, you might need to
## run something similar to this (use with care the lines with the '*'):
# To remove osmosis temporary data:
rm osmosis-0.38.tgz osmosis-0.40.tgz
rm -r osmosis-0.38/ osmosis-0.40/
rm -r /home/kfj/OSM/tempxx
# To delete the imported data, you may need to run:
dropdb -p 5432 xx
# If you created a tablespace:
echo "DROP TABLESPACE ;" | psql -p 5432
# If you created a tablespace or used the --install option:
sudo rm -r /home/kfj/PostgresPlus/data

########### End of log file

I hope this helps you track down what might have gone wrong. Still I think using the osmosis defaul won't hurt, even with larger datasets. But of course I'm not an expert...

Kay

[Mayeul]

kfj wrote: "according to osmosis 0.40 documentation, the default for the parameter in
question is "CompactTempFile" , which sounds like a better option than
"TempFile" (at least it does to me ;-)"

Well, the wiki says: "you must choose between the "TempFile" option which is much slower but still faster than relying on the default database geometry building implementation, or the "CompactTempFile" option which is more efficient for smaller datasets. "
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.40

[Mayeul]

Reading again your comment, the strange thing is "Thread for task 1-read-xml failed". So maybe the issue is with reading on your setup, it does not seem related to temp files (not sure). As I cannot reproduce this, I suggest to ask in osmosis support channels. I've processed a few hundred GB of .pbf (a few weeks of processing) and I confirm what the wiki says about "TempFile" preffered for huge files (maybe not true if you have a huge SSD; this might be hardware dependant).