Re: [Bigdata-developers] "No space left on device" importing Wikidata

[Rune S. <rs...@un...>]

If someone takes a look this at some point it now seems that the import process has stalled. Since yesterday the invisible disk usage has gone up, but the datafile hasn’t been touched.

Yesterday:
-rw-r--r-- 1 root     root     186G 11 dec 11:46 blazegraph.jnl
/dev/xvda1      2,0T  1,7T  344G   83% /

Today:
-rw-r--r-- 1 root     root     186G 11 dec 11:46 blazegraph.jnl
/dev/xvda1      2,0T  1,9T   73G   97% /

Soon the importer will probably stop with an out of disk space exception.

/Rune

> Den 11. dec. 2017 kl. 10.24 skrev Rune Stilling <rs...@un...>:
> 
> Hi list
> 
> An update on my question. It seems that the blazegraph importer makes use of disk space besides the datafile. Currently I’m running the import on a server with 2 TB of disk space. Even though the blazegraph.jnl file is only around 186 GB (and still running the import) the disk reports a use of 1.7 TB. I can’t find any files that make up for this use though.
> 
> If the import doesn’t finish soon the import will end with the same exception again.
> 
> Is this normal behavior and what (hidden) files is the blazegraph importer creating during the import?
> 
> Reagrds,
> Rune
> 
> ~]$ ls -la ~/blazegraph --block-size=G
> totalt 171G
> drwxrwxr-x 2 ec2-user ec2-user   1G  4 dec 12:25 .
> drwx------ 6 ec2-user ec2-user   1G 29 nov 10:52 ..
> -rw-r--r-- 1 root     root     186G 11 dec 09:21 blazegraph.jnl
> 
> [ ~]$ df -h
> Filsystem      Størr Brugt  Tilb Brug% Monteret på
> devtmpfs         15G   60K   15G    1% /dev
> tmpfs            15G     0   15G    0% /dev/shm
> /dev/xvda1      2,0T  1,7T  363G   83% /
> 
>> Den 4. dec. 2017 kl. 07.45 skrev Rune Stilling <rs...@un...>:
>> 
>> Hi list
>> 
>> I have setup an AWS instance using the blazegraph-ami-2.1.5. I have done the setup using a one disc configuration with 1 terabyte of space. After this I use the biodata.jar to import the full Wikidata dataset:
>> 
>>> sudo java -cp bigdata.jar com.bigdata.rdf.store.DataLoader -namespace wikidata -defaultGraph http://www.wikidata.org fullfeat
>>> ure.properties latest-all.ttl
>> 
>> 
>> After running 4 days the import terminates with an exception saying “no more space left on device”. There’s plenty of space left so what can I do to make this work?
>> 
>>> [ec2-user@ip-10-10-0-189 ~]$ df -h
>>> Filsystem      Størr Brugt  Tilb Brug% Monteret på
>>> devtmpfs         15G   60K   15G    1% /dev
>>> tmpfs            15G     0   15G    0% /dev/shm
>>> /dev/xvda1     1008G  305G  703G   31% /
>> 
>> 
>>> [ec2-user@ip-10-10-0-189 ~]$ ls -la ~/blazegraph --block-size=GB
>>> totalt 119GB
>>> drwxrwxr-x 2 ec2-user ec2-user   1GB 29 nov 10:53 .
>>> drwx------ 6 ec2-user ec2-user   1GB 29 nov 10:52 ..
>>> -rw-r--r-- 1 root     root     124GB  2 dec 10:14 blazegraph.jnl
>> 
>> 
>> /Rune Stilling
>> 
>