Re: [Geotools-gt2-users] Proper way to do parallel raster processing?

[Jody G. <jod...@gm...>]

Tobias you may need to go and ask on the ImageIO-Ext project which provides
the data format sink pulling all the raster content through the tile cache.

I am not sure if parallel processing is used when writing out a single
image (it would depend on the format). In geoserver there end up being
multiple requests each one writing out a different area of the raster
content.
--
Jody Garnett


On Mon, 23 May 2022 at 08:16, Tobias Gerdin <tob...@ha...>
wrote:

> Hello!
>
>
>
> I brought up this question before but basically I am still unsure of how
> multithreading fits into Geotools when it comes to raster processing
> (making it a JAI question, I guess). JAI being a lazy "pull”-based API I
> would expect the terminal operations (such as encoding a raster into a
> format for export, such as GeoTIFF) having some sort of multiprocessing
> ability, but that does not seem to be the case?
>
>
>
> Last time I posed this question I was pointed to this code:
>
>
> https://github.com/geosolutions-it/soil_sealing/blob/96a8c86e9ac891a273e7bc61b910416a0dbe1582/src/extension/wps-soil-sealing/wps-changematrix/src/main/java/org/geoserver/wps/gs/soilsealing/ChangeMatrixProcess.java#L669
>
>
>
> Similar to the above I currently I do this:
>
>
>
> private void triggerActualCalculation(RenderedImage renderedImage) {
>     final int numTileY = renderedImage.getNumYTiles(),
>             numTileX = renderedImage.getNumXTiles(),
>             minTileX = renderedImage.getMinTileX(),
>             minTileY = renderedImage.getMinTileY();
>
>     final List<Point> tiles = new ArrayList<Point>(numTileX * numTileY);
>     for (int i = minTileX; i < minTileX + numTileX; i++) {
>         for (int j = minTileY; j < minTileY + numTileY; j++) {
>             tiles.add(new Point(i, j));
>         }
>     }
>
>     tiles.stream()
>             .parallel()
>             .forEach(tileIndex -> renderedImage.getTile(tileIndex.x,
> tileIndex.y));
> }
>
>
>
> This seems to cause the rendering chain to be performed in parallel, but
> now it looks like I need to make sure that all computed tiles fit into the
> tile cache, or else I guess tiles will be thrown out and recomputed again
> when invoking the GeoTiffWriter to actually serialize the raster. So it
> seems like I need to choose between doing things in parallel or having
> enough memory for all the tiles to fit into the tile cache. While the
> resulting image is just a single band (as opposed to the 100 something
> bands used as inputs), it seems like there must a better way to do this,
> having tiles computed in parallel but all of them not necessarily fitting
> into the tile cache (the raster grid can potentially be very large and we
> would like to be able to deploy this system on machines without huge
> amounts of RAM). As I understand it Geotools was designed to cope with
> these constraints, but how is it to be done?
>
>
>
> *Regards,*
>
> *Tobias Gerdin*
> Havs- och vattenmyndigheten behandlar dina personuppgifter i enlighet med
> dataskyddsförordningen och myndighetens dataskyddspolicy, läs mer på
> www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
> SwAM processes your personal data in accordance with the General Data
> Protection Regulation (GDPR) and our Data Protection Policy, see
> www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
> _______________________________________________
> GeoTools-GT2-Users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>