Re: [Libosmscout-development] Import Problems, step #20

[Tim T. <ti...@fr...>]

Hello Augusto,

> I am pretty sure you ran out of memory, as I have seen that error  
> before on a system with 8 gigs. I would use a minimum of 16 gigs if  
> you are doing imports. I have never had issues with 16.

Yes, he run out of memory. However he said, that he build for 32bit.  
In this case the maximum address space of the process is 4GB. Having  
more main memory will not help, around 3,7GB memory will be full.

The file format of the resulting database is independend of the  
processor architectur. So you could build a 64 bit version of the  
library and the import to use the full address space and still use a  
32 bit compiled library to access the database form the client.

Else you will always struggle with the 5GB limit and the rather worse  
performance in result of working around it.

For 32 bit version that best you can do is to disable mmap access for  
all files. If this does not help, you currently need a 64 bit version  
since in this import step there is no block processing that limits the  
amount of required memory. Building the location tree (currently)  
requires to hold the complete tree in memory.

8GB main memory is OK in most cases. The limit for 8GB is importing  
germany or france. I recently switch from 8 to 16 GB. The importer was  
around 20% faster with the same configuration (for a smaller importer  
file) as for 8GB just because (I assume) OS file and memory handling  
was better (reading files from OS memory cache instead from disk).

-- 
Gruß...
    Tim