Re: [Geoserver-users] About the plugin ImageMosaic with very large data and versions of Geoserver

[<ced...@ge...>]

Simone Giannecchini a écrit :
> Ciao Cedric,
> sorry to get back to you with such a delay.
>
> First of all a quick question, which geoserver version are you running 
> now?
>
Ciao Simone.
I have two different versions. On my computer I have a 1.5.0 version, 
which lasts around one month or two. On this version, the ImageMosaic 
plugin works well for my data, using around 500Mb or ram for my jvm on 
tomcat (5.5.23).
In an other server, I have a Geoserver trunk version compiled yesterday. 
But I think of using an 1.5.x version, because it is a stable server.
>
> Aside from this, there is a flaw in your approach to handling this
> dataset (btw I am using the same thing as background :-) ).
Sorry, I haven't completely explained the whole process I've done on 
these data :)
I have 8 tiff files, which weight around 1Gb each one.
At this time, I used some gdal functions, because I didn't know the 
OverviewsEmbedder you are speaking about.

So I did these commands :
"gdal_translate -of GTiff -co "COMPRESS=JPEG" -co "TILED=YES" a1.tif 
a1_tiled.tif"
"gdaladdo a1_tiled.tif 2 4 8 16 32 64 128 256 512"
The resolution of these raster is 21600*21600.

The first one compress with a Jpeg compression, the second one adds 
overviews, as you explain in the following of your mail.
So each of my tiff files, after this process, weight around 100Mb, and 
contains overviews.
> It is not enough to compress the tiff as jpeg, You need to add
> overviews to it. My approach, which works as fast as hell btw, is to
> tile and compress jpeg each piece of the mosaic and then to add
> overviews to it. After that I build the mosaic. If you don't do this,
> you'll end up consuming a huge amount of memory (ranging from a lot to
> the size of the universe depending on the tile size of each single
> piece :-) ) and a lot of processor time to do the jpeg decompression
> and the on-the-fly subsampling.
>
> If you want to have good performance check the coverage tools inside
> the geotools 2.3.x branch. There is a nice tool called
> OverviewsEmbedder. You may want to run it on each ov your tiles this
> way:
>
> OverviewsEmbedder -s "/usr/home/tmp" -w *.tiff -t "512,512" -f 2 -n 8
> -a nn -c 512 -p 5 -z JPEG -r 0.75
>
> parameters are:
>
> <<< -s >>>
> source dir
>
>
> <<< -w >>>
> wildcard to pick up files
>
>
> <<< -t >>>
> tile size
>
>
> <<< -f >>>
> scale factors of each overview level
>
> <<< -n >>>
> number of levels of overv iews added
>
>
> <<< -a >>>
> algorithm to use for subsampling
>
>
> <<< -c >>>
> cahce isze used during resampling
>
> <<< -p >>>
> thread priority
>
> <<< -z >>>
> compression type, use JPEG
>
> <<< -r >>>
> compression ratio, use 0.75
>
> THe app will open up the original file and will write it down, tiled,
> compressed jpeg with overviews compressed jpeg themsevles. It will
> take some time to do it though.
>
I think I can try that, because I 've kept the original version of these 
tiff files somewhere on my pc.
You spoke about time that it will take, the 2 lines I've mentionned take 
around 3 minutes each one (so it is quite repetitve on my 8 files :) ).

So for these data, I use the class in Geotools 2.3.1 that manage the 
creation of a shape file from the information found in the tiff files (I 
think ImageMosaicIndexBuilder, something like that).
For the same data, with the same shape file, on my computer with 1Gb for 
the tomcat's jvm memory, I do not crash, even when I do lots of request 
on it. In fact, the blue marble I get from my geoserver is a background 
for a demo in mapbuilder I've done, so I've all the tools from 
mapbuilder to stress it :).
The same wms request on the same data and same shapefile defined on a 
Geoserver trunk crash, using more than the 4Gb of memory I've allocated 
to the tomcat's jvm ...

Hoping that this will help you :)

Cheers,
Cédric Briançon.
> Let me know if you need more help!
>
>
> Ciao,
> Simone.
>
> cache size to use.
> number of levels of overv iews added
> On 6/28/07, Cédric Briançon <ced...@ge...> wrote:
>> Hi everyone,
>>
>> last month, I've used a Geoserver version 1.5.0  (gt2.3.1 tags), and
>> I've configured a mosaic of 8 Geotiff files, which represents the whole
>> world (blue marble data).
>> Each files weights between around 1Gb.
>> I've used a gdal function to compress them with the JPEG type
>> compression, and I've obtained 8 files of around 100Mb.
>>
>> On this version, the ImageMosaic plugin works well without crashing in
>> memory. It takes a shape file that I've created with the
>> ImageMosaicIndexBuilder(if I remember well) which is a kind of grid for
>> these 8 geotiff files to put them together in order to obtain a unique
>> raster.
>> At this moment, it works on my office computer, with 2Gb of ram (I've
>> given 1Gb to the jvm of tomcat). It takes around 800Mb of that memory at
>> this moment...
>>
>> Today I've tested the same data (with the same shapefile) on a Geoserver
>> built two hours ago, on a new computer that has allows me to give to the
>> jvm 4Gb of memory (it has 8Gb of Ram embarqued :) ) and when I do the
>> same WMS request on this mosaic, I get an Out of Memory problem... it
>> seems that it has reached the 4Gb allocated :(
>>
>> So I wonder why such a difference between trunk and 2.3.1 in tags
>> version for Geotools...
>> Have you experiment the same problem with large data using ImageMosaic ?
>>
>> I have to add that a WMS request with group of layers (the 8 tiff files
>> separetely defined in my Geoserver) works. It takes around 1Gb of memory
>> for the jvm.
>>
>> Regards,
>> Cédric Briançon.
>>
>>
>> ------------------------------------------------------------------------- 
>>
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Geoserver-users mailing list
>> Geo...@li...
>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>
>
>