Re: [Bigdata-developers] "No space left on device" importing Wikidata

[Kevin F. <kf...@ar...>]

What's in the tmp directory?

Look, too, for a growing log file.

Those two seem like logical places to start, so apologies if you've 
already covered these locales.

Yours,
Kevin


On 12/12/17 02:33, Rune Stilling wrote:
> If someone takes a look this at some point it now seems that the import process has stalled. Since yesterday the invisible disk usage has gone up, but the datafile hasn’t been touched.
> 
> Yesterday:
> -rw-r--r-- 1 root     root     186G 11 dec 11:46 blazegraph.jnl
> /dev/xvda1      2,0T  1,7T  344G   83% /
> 
> Today:
> -rw-r--r-- 1 root     root     186G 11 dec 11:46 blazegraph.jnl
> /dev/xvda1      2,0T  1,9T   73G   97% /
> 
> Soon the importer will probably stop with an out of disk space exception.
> 
> /Rune
> 
>> Den 11. dec. 2017 kl. 10.24 skrev Rune Stilling <rs...@un...>:
>>
>> Hi list
>>
>> An update on my question. It seems that the blazegraph importer makes use of disk space besides the datafile. Currently I’m running the import on a server with 2 TB of disk space. Even though the blazegraph.jnl file is only around 186 GB (and still running the import) the disk reports a use of 1.7 TB. I can’t find any files that make up for this use though.
>>
>> If the import doesn’t finish soon the import will end with the same exception again.
>>
>> Is this normal behavior and what (hidden) files is the blazegraph importer creating during the import?
>>
>> Reagrds,
>> Rune
>>
>> ~]$ ls -la ~/blazegraph --block-size=G
>> totalt 171G
>> drwxrwxr-x 2 ec2-user ec2-user   1G  4 dec 12:25 .
>> drwx------ 6 ec2-user ec2-user   1G 29 nov 10:52 ..
>> -rw-r--r-- 1 root     root     186G 11 dec 09:21 blazegraph.jnl
>>
>> [ ~]$ df -h
>> Filsystem      Størr Brugt  Tilb Brug% Monteret på
>> devtmpfs         15G   60K   15G    1% /dev
>> tmpfs            15G     0   15G    0% /dev/shm
>> /dev/xvda1      2,0T  1,7T  363G   83% /
>>
>>> Den 4. dec. 2017 kl. 07.45 skrev Rune Stilling <rs...@un...>:
>>>
>>> Hi list
>>>
>>> I have setup an AWS instance using the blazegraph-ami-2.1.5. I have done the setup using a one disc configuration with 1 terabyte of space. After this I use the biodata.jar to import the full Wikidata dataset:
>>>
>>>> sudo java -cp bigdata.jar com.bigdata.rdf.store.DataLoader -namespace wikidata -defaultGraph http://www.wikidata.org fullfeat
>>>> ure.properties latest-all.ttl
>>>
>>>
>>> After running 4 days the import terminates with an exception saying “no more space left on device”. There’s plenty of space left so what can I do to make this work?
>>>
>>>> [ec2-user@ip-10-10-0-189 ~]$ df -h
>>>> Filsystem      Størr Brugt  Tilb Brug% Monteret på
>>>> devtmpfs         15G   60K   15G    1% /dev
>>>> tmpfs            15G     0   15G    0% /dev/shm
>>>> /dev/xvda1     1008G  305G  703G   31% /
>>>
>>>
>>>> [ec2-user@ip-10-10-0-189 ~]$ ls -la ~/blazegraph --block-size=GB
>>>> totalt 119GB
>>>> drwxrwxr-x 2 ec2-user ec2-user   1GB 29 nov 10:53 .
>>>> drwx------ 6 ec2-user ec2-user   1GB 29 nov 10:52 ..
>>>> -rw-r--r-- 1 root     root     124GB  2 dec 10:14 blazegraph.jnl
>>>
>>>
>>> /Rune Stilling
>>>
>>
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
>