#6 osmosis might fail if nodeLocationStoreType="TempFile"

open
nobody
None
5
2012-04-29
2012-04-29
Mayeul
No

(An OSM contributer (kfj) reported this to me):
I ran osm2postgresql, and at the point when the call to osmosis happens, the process would fail with a lot of lengthy error messages (in the log file). I tried the isolated osmosis call with modified parameters, and found that leaving out nodeLocationStoreType="TempFile" made the problem go away. So without this parameter, the osmosis default is used, and that runs just fine.

Any reason you use this parameter? When I tried it, my system would produce heavy disk activity for about a minute and then fail saying that the read-xml task had failed.

I'm using Kubuntu 11.4 on a 32bit intel system.

I also think it might be a nice addition to your script if you let it output some messages as well - even if it's only a little message saying if it has succeeded or failed.

So now I can finally put OSM data as a postgis layer on top of the contours I've made

Discussion

  • Mayeul
    Mayeul
    2012-04-29

    The nodeLocationStoreType="TempFile" is aboslutely required for large files (europe.osm...) and may be necessary for systems with low memory. If you put on memory with low memory system, it will go to cache anyway but with lower performance.
    Since the bug is only related to osmosis, it may be better to search for help in the osmosis community. Still reporting the bug here may help someone else. Could you post your "lot of lengthy error messages (in the log file)" here? (or the ones you identify as relevant).

    Anyway, I'm glad I use bash shells: it helps users tweaks my code for their own needs or to solve their own problems, and that's why I choose that solution over compiled code.

    (for the output of messages, this is hard as I redirect everything to log - depending on version of osm2postgresql - proposals welcome, maybe in another thread).

    Mayeul

     
  • Mayeul
    Mayeul
    2012-04-29

    • summary: osmosis fails because of nodeLocationStoreType="TempFile" --> osmosis might fail if nodeLocationStoreType="TempFile"
     

  • Anonymous
    2012-04-30

    according to osmosis 0.40 documentation, the default for the parameter in question is "CompactTempFile" , which sounds like a better option than "TempFile" (at least it does to me ;-) - and not "InMemory", which would indeed be a problem for large data sets.

    I just managed to import piemonte.pbf into postgres using this sequence (I have database superuser privileges attached to my database user account, which is in the same name as my system account):

    # first set up the database:

    dbname=piemonte

    osmfile=$dbname.osm
    pbffile=$dbname.pbf

    createdb $dbname
    createlang plpgsql $dbname

    psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d $dbname
    psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql -d $dbname
    psql -d $dbname -f /usr/share/postgresql/8.4/contrib/hstore-new.sql
    psql -d $dbname -f /usr/share/postgresql/8.4/contrib/adminpack.sql

    # now let osm2postgresql (with the modified parameter) do the rest :

    ./osm2postgresql_05rc4.sh --dbname $dbname --pbf --file $pbffile

    I suppose letting osm2postgresql set up the database would also work, since the code is practically the same in your script.

    Import took roughly a quarter of an hour.

    The log file with the original parameter looks just fine up to the point where all the osmosis tasks are launched:

    ...
    30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager execute
    FEIN: Launching task 2-buffer in a new thread.
    30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager execute
    FEIN: Task 3-write-pgsql-dump is passive, no execution required.
    30.04.2012 09:15:20 org.openstreetmap.osmosis.core.Osmosis run
    INFO: Pipeline executing, waiting for completion.
    30.04.2012 09:15:20 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
    FEIN: Waiting for task 1-read-xml to complete.

    It remains like this for several minutes (with a test file containing only a few hundred nodes!), while there is heavy dis use, I can't tell why or where, but the only process seemingly doing anything is java. Then the process stops, and the log file continues like this:

    SCHWERWIEGEND: Thread for task 1-read-xml failed
    org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error has occurred, aborting.
    at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:78)
    at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:134)
    at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:184)
    at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.process(EntityBuffer.java:38)
    at org.openstreetmap.osmosis.xml.v0_6.impl.NodeElementProcessor.end(NodeElementProcessor.java:118)
    at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
    at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
    at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:108)
    at java.lang.Thread.run(Thread.java:679)
    30.04.2012 09:21:11 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
    FEIN: Waiting for task 2-buffer to complete.
    30.04.2012 09:21:14 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
    SCHWERWIEGEND: Thread for task 2-buffer failed
    org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to write node location data to node storage file /tmp/nodelatlon3486110848548710927.tmp.
    at org.openstreetmap.osmosis.pgsnapshot.common.PersistentNodeLocationStore.addLocation(PersistentNodeLocationStore.java:170)
    at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.WayGeometryBuilder.addNodeLocation(WayGeometryBuilder.java:64)
    at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetBuilder.process(CopyFilesetBuilder.java:157)
    at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
    at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetBuilder.process(CopyFilesetBuilder.java:115)
    at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDumpWriter.process(PostgreSqlDumpWriter.java:58)
    at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.run(EntityBuffer.java:74)
    at java.lang.Thread.run(Thread.java:679)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:297)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at org.openstreetmap.osmosis.pgsnapshot.common.PersistentNodeLocationStore.addLocation(PersistentNodeLocationStore.java:156)
    ... 7 more
    30.04.2012 09:21:14 org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager waitForCompletion
    FEIN: Task 3-write-pgsql-dump is passive, no completion wait required.
    30.04.2012 09:21:14 org.openstreetmap.osmosis.core.Osmosis main
    SCHWERWIEGEND: Execution aborted.
    org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
    at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
    at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
    at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
    Error 1. The command which triggered the error was:
    ./$osmosis_version/bin/osmosis -v --read-$filetype file="$osmfile" --buffer --write-pgsql-dump enableBboxBuilder=yes enableLinestringBuilder=yes nodeLocationStoreType="TempFile" directory="$temporarydir"
    on line 438 of the osm2postgresql script.
    ## To delete the imported or temporary data, you might need to
    ## run something similar to this (use with care the lines with the '*'):
    # To remove osmosis temporary data:
    rm osmosis-0.38.tgz osmosis-0.40.tgz
    rm -r osmosis-0.38/ osmosis-0.40/
    rm -r /home/kfj/OSM/tempxx
    # To delete the imported data, you may need to run:
    dropdb -p 5432 xx
    # If you created a tablespace:
    echo "DROP TABLESPACE ;" | psql -p 5432
    # If you created a tablespace or used the --install option:
    sudo rm -r /home/kfj/PostgresPlus/data

    ########### End of log file

    I hope this helps you track down what might have gone wrong. Still I think using the osmosis defaul won't hurt, even with larger datasets. But of course I'm not an expert...

    Kay

     
    Last edit: Anonymous 2014-11-09
  • Mayeul
    Mayeul
    2012-04-30

    kfj wrote: "according to osmosis 0.40 documentation, the default for the parameter in
    question is "CompactTempFile" , which sounds like a better option than
    "TempFile" (at least it does to me ;-)"

    Well, the wiki says: "you must choose between the "TempFile" option which is much slower but still faster than relying on the default database geometry building implementation, or the "CompactTempFile" option which is more efficient for smaller datasets. "
    http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.40

     
  • Mayeul
    Mayeul
    2012-04-30

    Reading again your comment, the strange thing is "Thread for task 1-read-xml failed". So maybe the issue is with reading on your setup, it does not seem related to temp files (not sure). As I cannot reproduce this, I suggest to ask in osmosis support channels. I've processed a few hundred GB of .pbf (a few weeks of processing) and I confirm what the wiki says about "TempFile" preffered for huge files (maybe not true if you have a huge SSD; this might be hardware dependant).