Thread: [Geotools-gt2-users] embedding GeoTools is always a pain

Toolkit for working with and mapping geospatial data

Brought to you by: aaime, afabiani, cholmesny, cory2070, and 14 others

geotools-gt2-users

[Geotools-gt2-users] embedding GeoTools is always a pain

From: teeschke <dan...@we...> - 2015-11-24 22:44:20

Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Mark P. <mc....@gm...> - 2015-11-26 09:05:07

On 24-11-15 23:35, teeschke wrote:
> Dear GeoTools users and developers,
>
> after embedding GeoTools in my n-th project I am very annoyed about the
> complexity and pitfalls of this process. Two Scenarios to describe the
> problem:
>
> First: I wanted to parse a KML file and try to follow the Geometry Guide on
> http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
> 1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
> because I learned my lessons from projects before
> 2. trying to resolve the KMLConfiguration
> a) adding gt-xml and gt-geometry modules as dependencies which is not
> documented but seems to be meaningful because the how-to is about this
> b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
> the xsd folder and it sounds meaningful as well
> --> lucky me: it was the right one
> Sidenote: I needed to scroll through the boundless repository manually to
> scan for this modules
>
> Second Scenario: whenever I want to do something with grid files - the JAI
> library is required but not hosted in the maven central or boundless repo.
> So I must use another repository (osgeo/webda) which seems to be the last
> one which is hosting the JAI library :)

see item 7 in the quickstart: 
http://docs.geotools.org/latest/userguide/tutorial/quickstart/maven.html 
basically you should use the osgeo repo unless you want to live on the edge

In my experience any IDE with Maven support (eclipse, netbeans) also has 
support for Maven indexes making it possible to do local lookups.

It would be helpful if of osgeo was running a proper maven repository so 
it could provide an index, but I guess the boundless repo can serve that 
purpose. I would agree that hosting in maven central is the optimal 
solution, but that also requires building javadoc for each artifact.

-M

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Jody G. <jod...@gm...> - 2015-11-29 18:41:17

We are starting to look at replacing JAI
<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing
design and fundraising over the course of 2016. JAI is not open source so
we are unable to place it on maven central. The second part is service
provider interface is a pain in some environments, and the "service
registry" we use looks to be unsupported in Java 9.

If you have any suggestions on parties interested in the above tasks we
would love to know.


--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:

> Dear GeoTools users and developers,
>
> after embedding GeoTools in my n-th project I am very annoyed about the
> complexity and pitfalls of this process. Two Scenarios to describe the
> problem:
>
> First: I wanted to parse a KML file and try to follow the Geometry Guide on
>
> http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
> 1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
> because I learned my lessons from projects before
> 2. trying to resolve the KMLConfiguration
> a) adding gt-xml and gt-geometry modules as dependencies which is not
> documented but seems to be meaningful because the how-to is about this
> b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
> the xsd folder and it sounds meaningful as well
> --> lucky me: it was the right one
> Sidenote: I needed to scroll through the boundless repository manually to
> scan for this modules
>
> Second Scenario: whenever I want to do something with grid files - the JAI
> library is required but not hosted in the maven central or boundless repo.
> So I must use another repository (osgeo/webda) which seems to be the last
> one which is hosting the JAI library :)
>
> At the end of the day all this searching and trying needs a lot of time, is
> not a pleasure and from my point of view not state-of-the-art because all
> other libraries are available at maven central and are able to manage their
> own dependencies. But in case of GeoTools I always need to look into
> reference projects to find out how I was getting it the last time.
>
> What are your experience with scenarios like this? Are you adding all
> modules into the pom to not run into problems I always got or is there a
> how-to I missed? :)
>
> Thanks and kind regards,
> Tee
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
> Sent from the geotools-gt2-users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple
> OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> _______________________________________________
> GeoTools-GT2-Users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Andy T. <A.G...@le...> - 2015-11-29 19:28:26

Hi Jody, Hi everyone,


Many moons ago I started developing a raster data calculator based on the Raster class of GeoTools. I developed this so that it tiles/chunks 2D square celled rasters into chunks of the same size, but configurable rows and columns. I called the library Grids. Every Grid in Grids currently either holds ints or doubles, but I had envisaged developing ones for boolean and BigDecimal too. Every Grid shares a Grids_Environment object for memory handling, so if when processing the system runs low on memory, various data/tiles/chunks can be swapped to file space. There can be quite a lot involved in being clever about swapping data to and from different memory stores. Anyway, the library can handle multiple large grids on modest resources for many operations. Perhaps this can be evolved to be the JAI replacement.


In Grids there are various ways of storing chunks. They can be stored as HashMaps or Arrays or in other ways depending on their size. At present these can be stored as JAI images too, so currently there is a dependency on JAI for these, but it is not necessary at all.


Here is a link to the latest Grids on GitHub:

https://github.com/agdturner/grids


There is a bit more history of grids here:

http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/grids/


Any questions I'll be happy to try to answer them and provide examples etc.


Cheers,


Andy


________________________________
From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

We are starting to look at replacing JAI<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
[https://avatars2.githubusercontent.com/u/1187031?v=3&s=400]<https://github.com/geotools/geotools/wiki/Replace-JAI>

Replace JAI · geotools/geotools Wiki · GitHub
geotools - Official GeoTools repository ... You can clone with HTTPS
Read more...<https://github.com/geotools/geotools/wiki/Replace-JAI>



If you have any suggestions on parties interested in the above tasks we would love to know.


--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...<mailto:dan...@we...>> wrote:
Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Andy T. <A.G...@le...> - 2015-11-29 19:57:51

One thing I think it is worth pointing out about Grids (as it is fairly fundamental) is that the underlying coordinate system is based on BigDecimal. The calculations for where a grid value is located are based on BigDecimal arithmetic. This is because otherwise there can be problems when dealing with rasters with very large numbers of rows and columns. Although Grids will handle multiple rasters with very large numbers of rows and columns, there are non memory/disk resource restrictions on how many rows and columns any grid can have and this will vary depending on how the definition of the chunks (data model used to store these).


I have not parallelised anything much in grids, but many grid operations can be parallelised and I have thought about this in respect of memory handling.


Some of the memory handling catches OutOfMemoryErrors to deal with problems if they do come up. It is much better though to test available memory to judge if the grids processing component is running low. In the Generic library upon which Grids depends there are useful functions for this.


________________________________
From: Andy Turner <A.G...@le...>
Sent: 29 November 2015 19:27
To: Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain


Hi Jody, Hi everyone,


Many moons ago I started developing a raster data calculator based on the Raster class of GeoTools. I developed this so that it tiles/chunks 2D square celled rasters into chunks of the same size, but configurable rows and columns. I called the library Grids. Every Grid in Grids currently either holds ints or doubles, but I had envisaged developing ones for boolean and BigDecimal too. Every Grid shares a Grids_Environment object for memory handling, so if when processing the system runs low on memory, various data/tiles/chunks can be swapped to file space. There can be quite a lot involved in being clever about swapping data to and from different memory stores. Anyway, the library can handle multiple large grids on modest resources for many operations. Perhaps this can be evolved to be the JAI replacement.


In Grids there are various ways of storing chunks. They can be stored as HashMaps or Arrays or in other ways depending on their size. At present these can be stored as JAI images too, so currently there is a dependency on JAI for these, but it is not necessary at all.


Here is a link to the latest Grids on GitHub:

https://github.com/agdturner/grids


There is a bit more history of grids here:

http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/grids/


Any questions I'll be happy to try to answer them and provide examples etc.


Cheers,


Andy


________________________________
From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

We are starting to look at replacing JAI<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
[https://avatars2.githubusercontent.com/u/1187031?v=3&s=400]<https://github.com/geotools/geotools/wiki/Replace-JAI>

Replace JAI · geotools/geotools Wiki · GitHub
geotools - Official GeoTools repository ... You can clone with HTTPS
Read more...<https://github.com/geotools/geotools/wiki/Replace-JAI>



If you have any suggestions on parties interested in the above tasks we would love to know.


--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...<mailto:dan...@we...>> wrote:
Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Peter B. <pet...@ya...> - 2015-12-03 10:58:51

I can definitely sympathize with Tee's situation. In the latest release of geotools (14.1) there are 223 different jar files. Chances are you don't need all of them for your project. In fact, if you're a noob and load all the jar files into a project without maven you'll get all sorts of errors.
Personally, I like my projects to have as small a footprint as possible with a minimal set of jars/dependencies. In the case of geotools, that means removing as many jars as necessary. I don't use maven so I have to prune the jars manually. 

First, I separate out all the jars into categories (raster/image jars, database jars, vector jars, ogc/web related jars, xml/schema, commons, etc). Then I start adding the jars I think I need and add more jars as needed. This is an time consuming process but it keeps my final deployment size to a minimum. 

Again, I don't use Maven and I don't know if it would help with this process. However, it would definitely help folks like me if the jars could be binned in some way. I can draft a list of bins if the packaging folks are willing to entertain this idea.

Thanks,
Peter



PS. Love geotools! Thanks for everyone's hard work.


From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain We are starting to look at replacing JAI, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
|  | Replace JAI · geotools/geotools Wiki · GitHubgeotools - Official GeoTools repository ... You can clone with HTTPSRead more... |



If you have any suggestions on parties interested in the above tasks we would love to know.

--Jody Garnett
On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:

Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users



------------------------------------------------------------------------------

_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Jes W. N. <je...@no...> - 2015-12-03 12:33:38

You really want an automated dependency manager for that, so you can just point at the functionality you want, and get the needed jars assembled automatically.

If you don’t want Maven, you could take a look at the leaner Ivy, which uses the same definitions and repositories as Maven, but only handles the dependency management.
(In fairness, if you only use Maven for dependency management, it’s not bad. Most of its bad reputation comes from people obsessively trying to use it as its own programming language).

JWN

From: Peter Borissow [mailto:pet...@ya...]
Sent: 3. december 2015 11:59
To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...>
Cc: GeoTools Users <geo...@li...>
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

I can definitely sympathize with Tee's situation. In the latest release of geotools (14.1) there are 223 different jar files. Chances are you don't need all of them for your project. In fact, if you're a noob and load all the jar files into a project without maven you'll get all sorts of errors.

Personally, I like my projects to have as small a footprint as possible with a minimal set of jars/dependencies. In the case of geotools, that means removing as many jars as necessary. I don't use maven so I have to prune the jars manually.

First, I separate out all the jars into categories (raster/image jars, database jars, vector jars, ogc/web related jars, xml/schema, commons, etc). Then I start adding the jars I think I need and add more jars as needed. This is an time consuming process but it keeps my final deployment size to a minimum.

Again, I don't use Maven and I don't know if it would help with this process. However, it would definitely help folks like me if the jars could be binned in some way. I can draft a list of bins if the packaging folks are willing to entertain this idea.

Thanks,
Peter

PS. Love geotools! Thanks for everyone's hard work.

________________________________
From: Jody Garnett <jod...@gm...<mailto:jod...@gm...>>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

We are starting to look at replacing JAI<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
[https://avatars2.githubusercontent.com/u/1187031?v=3&s=400]<https://github.com/geotools/geotools/wiki/Replace-JAI>

Replace JAI · geotools/geotools Wiki · GitHub
geotools - Official GeoTools repository ... You can clone with HTTPS
Read more...<https://github.com/geotools/geotools/wiki/Replace-JAI>

If you have any suggestions on parties interested in the above tasks we would love to know.

--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...<mailto:dan...@we...>> wrote:

Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

------------------------------------------------------------------------------

_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Peter B. <pet...@ya...> - 2015-12-03 16:07:29

Yes, an automated dependency manager would be great! My IDE (Netbeans) will flag missing dependencies in my source code without Maven or Ivy. Problem is that it doesn't catch various inter-dependencies within the jar files. Unfortunately, those inter-dependencies only become obvious at runtime. 

In the absence of an automated dependency manager, it would be *really nice* to break up the the binary distribution of GeoTools into sub-directories. That would help users pick and choose which jars/libs they need for their specific needs. So instead of having 200+ different jars in your project, you might only need a dozen or so. 

For example, I tend to organize the jars into the following directory structure:

/core - gt-api-14.1.jar - gt-opengis-14.1.jar - gt-metadata-14.1.jar - gt-referencing-14.1.jar - gt-geometry-14.1.jar
 - jsr-275-1.0-beta-2.jar - jts-1.13.jar
 ...

/epsg - gt-epsg-extension-14.1.jar - gt-epsg-wkt-14.1.jar
 ...
/epsg/driver
 - gt-epsg-hsql-14.1.jar - gt-epsg-postgresql-14.1.jar
 ...

/raster
 - gt-coverage-14.1.jar
 - gt-grid-14.1.jar
 - jai_core-1.1.3.jar ...
/raster/processing - gt-image-14.1.jar
 - gt-imagemosaic-14.1.jar - gt-imagepyramid-14.1.jar

/raster/drivers - gt-geotiff-14.1.jar
 - gdal-1.8.1.jar
 - gt-gtopo30-14.1.jar
 - gt-imageio-ext-.jar ...
/vector/drivers - gt-geojson-14.1.jar
 - gt-shapefile-14.1.jar - gt-vpf-14.1.jar
 ...
/database/drivers - gt-jdbc-14.1.jar ...
/database/drivers/h2 - gt-jdbc-h2-14.1.jar - h2-1.1.119.jar
/database/drivers/sqlite - gt-jdbc-spatialite-14.1.jar
 - sqlite-jdbc-3.8.6.jar
/database/drivers/postgis - gt-jdbc-postgis-14.1.jar
 - postgresql-9.4-1201-jdbc41.jar
/web - gt-wfs-14.1.jar 
 - net.opengis.wfs-14.1.jar - gt-wms-14.1.jar


Something like that...
Then, when I need to create a new app, I pick which jars I think I need based on their functionality. I only include the database drivers that I need, I might not need to do anything with rasters so I can exclude that directory wholesale, etc. 

I realize that the original thread was maven-specific so I apologize if my comments are steering the conversation off course. Just throwing out an idea to help make GeoTools more accessible to new users and casual users like me :-)

Thanks,
Peter






      From: Jes Wulfsberg Nielsen <je...@no...>
 To: GeoTools Users <geo...@li...> 
 Sent: Thursday, December 3, 2015 7:30 AM
 Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain
   
#yiv3288042049 #yiv3288042049 -- _filtered #yiv3288042049 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv3288042049 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv3288042049 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv3288042049 {panose-1:2 11 5 2 4 2 4 2 2 3;} _filtered #yiv3288042049 {panose-1:2 11 4 2 4 2 4 2 2 3;}#yiv3288042049 #yiv3288042049 p.yiv3288042049MsoNormal, #yiv3288042049 li.yiv3288042049MsoNormal, #yiv3288042049 div.yiv3288042049MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv3288042049 a:link, #yiv3288042049 span.yiv3288042049MsoHyperlink {color:blue;text-decoration:underline;}#yiv3288042049 a:visited, #yiv3288042049 span.yiv3288042049MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv3288042049 span.yiv3288042049EmailStyle17 {color:#1F497D;}#yiv3288042049 .yiv3288042049MsoChpDefault {font-size:10.0pt;} _filtered #yiv3288042049 {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv3288042049 div.yiv3288042049WordSection1 {}#yiv3288042049 You really want an automated dependency manager for that, so you can just point at the functionality you want, and get the needed jars assembled automatically.  If you don’t want Maven, you could take a look at the leaner Ivy, which uses the same definitions and repositories as Maven, but only handles the dependency management.(In fairness, if you only use Maven for dependency management, it’s not bad. Most of its bad reputation comes from people obsessively trying to use it as its own programming language).  JWN  

From: Peter Borissow [mailto:pet...@ya...] 
Sent: 3. december 2015 11:59
To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...>
Cc: GeoTools Users <geo...@li...>
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain  I can definitely sympathize with Tee's situation. In the latest release of geotools (14.1) there are 223 different jar files. Chances are you don't need all of them for your project. In fact, if you're a noob and load all the jar files into a project without maven you'll get all sorts of errors.  Personally, I like my projects to have as small a footprint as possible with a minimal set of jars/dependencies. In the case of geotools, that means removing as many jars as necessary. I don't use maven so I have to prune the jars manually.   First, I separate out all the jars into categories (raster/image jars, database jars, vector jars, ogc/web related jars, xml/schema, commons, etc). Then I start adding the jars I think I need and add more jars as needed. This is an time consuming process but it keeps my final deployment size to a minimum.   Again, I don't use Maven and I don't know if it would help with this process. However, it would definitely help folks like me if the jars could be binned in some way. I can draft a list of bins if the packaging folks are willing to entertain this idea.    Thanks,Peter    PS. Love geotools! Thanks for everyone's hard work.  From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain  We are starting to look at replacing JAI, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9. 
|  | Replace JAI · geotools/geotools Wiki · GitHubgeotools - Official GeoTools repository ... You can clone with HTTPSRead more... |

    If you have any suggestions on parties interested in the above tasks we would love to know.  
--Jody Garnett  On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:


Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
    ------------------------------------------------------------------------------  _______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users  
------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Jody G. <jod...@gm...> - 2016-01-18 07:17:09

Drafting bins would be fine, I think we can control the assembly step
(where the final zip is produced) but I am not quite sure how.

It is too bad you do not use maven, or ant+ivy, as it makes managing java
dependencies easier and is integrated in most IDEs now.

--
Jody Garnett

On 3 December 2015 at 02:58, Peter Borissow <pet...@ya...>
wrote:

> I can definitely sympathize with Tee's situation. In the latest release of
> geotools (14.1) there are 223 different jar files. Chances are you don't
> need all of them for your project. In fact, if you're a noob and load all
> the jar files into a project without maven you'll get all sorts of errors.
>
> Personally, I like my projects to have as small a footprint as possible
> with a minimal set of jars/dependencies. In the case of geotools, that
> means removing as many jars as necessary. I don't use maven so I have to
> prune the jars manually.
>
> First, I separate out all the jars into categories (raster/image jars,
> database jars, vector jars, ogc/web related jars, xml/schema, commons,
> etc). Then I start adding the jars I think I need and add more jars as
> needed. This is an time consuming process but it keeps my final deployment
> size to a minimum.
>
> Again, I don't use Maven and I don't know if it would help with this
> process. However, it would definitely help folks like me if the jars could
> be binned in some way. I can draft a list of bins if the packaging folks
> are willing to entertain this idea.
>
>
> Thanks,
> Peter
>
>
>
> PS. Love geotools! Thanks for everyone's hard work.
>
>
> ------------------------------
> *From:* Jody Garnett <jod...@gm...>
> *Sent:* 29 November 2015 18:40
> *To:* teeschke
> *Cc:* GeoTools Users
> *Subject:* Re: [Geotools-gt2-users] embedding GeoTools is always a pain
>
> We are starting to look at replacing JAI
> <https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be
> doing design and fundraising over the course of 2016. JAI is not open
> source so we are unable to place it on maven central. The second part is
> service provider interface is a pain in some environments, and the "service
> registry" we use looks to be unsupported in Java 9.
> <https://github.com/geotools/geotools/wiki/Replace-JAI>
> Replace JAI · geotools/geotools Wiki · GitHub
> geotools - Official GeoTools repository ... You can clone with HTTPS
> Read more... <https://github.com/geotools/geotools/wiki/Replace-JAI>
>
>
> If you have any suggestions on parties interested in the above tasks we
> would love to know.
>
>
> --
> Jody Garnett
>
> On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:
>
> Dear GeoTools users and developers,
>
> after embedding GeoTools in my n-th project I am very annoyed about the
> complexity and pitfalls of this process. Two Scenarios to describe the
> problem:
>
> First: I wanted to parse a KML file and try to follow the Geometry Guide on
>
> http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
> 1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
> because I learned my lessons from projects before
> 2. trying to resolve the KMLConfiguration
> a) adding gt-xml and gt-geometry modules as dependencies which is not
> documented but seems to be meaningful because the how-to is about this
> b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
> the xsd folder and it sounds meaningful as well
> --> lucky me: it was the right one
> Sidenote: I needed to scroll through the boundless repository manually to
> scan for this modules
>
> Second Scenario: whenever I want to do something with grid files - the JAI
> library is required but not hosted in the maven central or boundless repo.
> So I must use another repository (osgeo/webda) which seems to be the last
> one which is hosting the JAI library :)
>
> At the end of the day all this searching and trying needs a lot of time, is
> not a pleasure and from my point of view not state-of-the-art because all
> other libraries are available at maven central and are able to manage their
> own dependencies. But in case of GeoTools I always need to look into
> reference projects to find out how I was getting it the last time.
>
> What are your experience with scenarios like this? Are you adding all
> modules into the pom to not run into problems I always got or is there a
> how-to I missed? :)
>
> Thanks and kind regards,
> Tee
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
> Sent from the geotools-gt2-users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple
> OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> _______________________________________________
> GeoTools-GT2-Users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> GeoTools-GT2-Users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>
>
>

Re: [Geotools-gt2-users] Grids

From: Peter B. <pet...@ya...> - 2015-12-03 14:53:16

Hi Andy-    Sounds interesting. Does the Grids library support no_data and random access to a region of interest? Does the input raster need to be tiled? Is JAI required? Any other dependencies? Any known limitations?

Also, it would be interesting to have a couple simple examples on your github page for how to load in a raster and access pixel values at a given point on the image. 

Thanks,
Peter

      From: Andy Turner <A.G...@le...>
 To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...> 
Cc: GeoTools Users <geo...@li...>
 Sent: Sunday, November 29, 2015 2:55 PM
 Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

#yiv2002285508 #yiv2002285508 -- P {margin-top:0;margin-bottom:0;}#yiv2002285508 One thing I think it is worth pointing out about Grids (as it is fairly fundamental) is that the underlying coordinate system is based on BigDecimal. The calculations for where a grid value is located are based on BigDecimal arithmetic. This is because otherwise there can be problems when dealing with rasters with very large numbers of rows and columns. Although Grids will handle multiple rasters with very large numbers of rows and columns, there are non memory/disk resource restrictions on how many rows and columns any grid can have and this will vary depending on how the definition of the chunks (data model used to store these).
I have not parallelised anything much in grids, but many grid operations can be parallelised and I have thought about this in respect of memory handling.
Some of the memory handling catches OutOfMemoryErrors to deal with problems if they do come up. It is much better though to test available memory to judge if the grids processing component is running low. In the Generic library upon which Grids depends there are useful functions for this.

From: Andy Turner <A.G...@le...>
Sent: 29 November 2015 19:27
To: Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain Hi Jody, Hi everyone,
Many moons ago I started developing a raster data calculator based on the Raster class of GeoTools. I developed this so that it tiles/chunks 2D square celled rasters into chunks of the same size, but configurable rows and columns. I called the library Grids. Every Grid in Grids currently either holds ints or doubles, but I had envisaged developing ones for boolean and BigDecimal too. Every Grid shares a Grids_Environment object for memory handling, so if when processing the system runs low on memory, various data/tiles/chunks can be swapped to file space. There can be quite a lot involved in being clever about swapping data to and from different memory stores. Anyway, the library can handle multiple large grids on modest resources for many operations. Perhaps this can be evolved to be the JAI replacement.
In Grids there are various ways of storing chunks. They can be stored as HashMaps or Arrays or in other ways depending on their size. At present these can be stored as JAI images too, so currently there is a dependency on JAI for these, but it is not necessary at all.
Here is a link to the latest Grids on GitHub:https://github.com/agdturner/grids

There is a bit more history of grids here:http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/grids/

Any questions I'll be happy to try to answer them and provide examples etc.
Cheers,
Andy

From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain We are starting to look at replacing JAI, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
|  | Replace JAI · geotools/geotools Wiki · GitHubgeotools - Official GeoTools repository ... You can clone with HTTPSRead more... |

If you have any suggestions on parties interested in the above tasks we would love to know.

--Jody Garnett
On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:

Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

------------------------------------------------------------------------------

_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] Grids

From: Andy T. <A.G...@le...> - 2015-12-03 15:18:47

Yes, no_data values can be set. In calculations, no_data values are treated as I have thought was appropriate, for example in generalisations that calculate averages etc.



You can request the value of the Grid at a point and there are utility type methods that get all values within a set distance in coordinate space.


The input data does not need to be tiled. I load input data into Grids using an ESRI ASCII grid format. Typically this tiles the data the number of rows and the number of columns in each tile (chunk) can be set and don't have to be the same.


You can do everything in Grids without JAI, so it is not really required, but more of an optional for storage of data in chunks. However, not all the Grids source will compile without it. I did want to separate this out completely and have the JAI data store as a sort of plugin, but have not got around to this - and from the sounds of it, it might be best to factor JAI out completely anyway.


Grids depends on my Generic library (https://github.com/agdturner/generic) which helps handle the swapping of data (amongst other things, in particular calculations using BigDecimal arithmetic). The Generic library itself depends on Apache Commons Math. Again, that is not much of a dependency and could be factored out.


I don't have a lot of time right now, but I'll work on some examples and reply in due course.


Cheers,


Andy


________________________________
From: Peter Borissow <pet...@ya...>
Sent: 03 December 2015 14:50
To: Andy Turner; Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: Grids

Hi Andy-
    Sounds interesting. Does the Grids library support no_data and random access to a region of interest? Does the input raster need to be tiled? Is JAI required? Any other dependencies? Any known limitations?

Also, it would be interesting to have a couple simple examples on your github page for how to load in a raster and access pixel values at a given point on the image.


Thanks,
Peter

________________________________
From: Andy Turner <A.G...@le...>
To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...>
Cc: GeoTools Users <geo...@li...>
Sent: Sunday, November 29, 2015 2:55 PM
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

One thing I think it is worth pointing out about Grids (as it is fairly fundamental) is that the underlying coordinate system is based on BigDecimal. The calculations for where a grid value is located are based on BigDecimal arithmetic. This is because otherwise there can be problems when dealing with rasters with very large numbers of rows and columns. Although Grids will handle multiple rasters with very large numbers of rows and columns, there are non memory/disk resource restrictions on how many rows and columns any grid can have and this will vary depending on how the definition of the chunks (data model used to store these).

I have not parallelised anything much in grids, but many grid operations can be parallelised and I have thought about this in respect of memory handling.

Some of the memory handling catches OutOfMemoryErrors to deal with problems if they do come up. It is much better though to test available memory to judge if the grids processing component is running low. In the Generic library upon which Grids depends there are useful functions for this.




________________________________
From: Andy Turner <A.G...@le...>
Sent: 29 November 2015 19:27
To: Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

Hi Jody, Hi everyone,

Many moons ago I started developing a raster data calculator based on the Raster class of GeoTools. I developed this so that it tiles/chunks 2D square celled rasters into chunks of the same size, but configurable rows and columns. I called the library Grids. Every Grid in Grids currently either holds ints or doubles, but I had envisaged developing ones for boolean and BigDecimal too. Every Grid shares a Grids_Environment object for memory handling, so if when processing the system runs low on memory, various data/tiles/chunks can be swapped to file space. There can be quite a lot involved in being clever about swapping data to and from different memory stores. Anyway, the library can handle multiple large grids on modest resources for many operations. Perhaps this can be evolved to be the JAI replacement.

In Grids there are various ways of storing chunks. They can be stored as HashMaps or Arrays or in other ways depending on their size. At present these can be stored as JAI images too, so currently there is a dependency on JAI for these, but it is not necessary at all.

Here is a link to the latest Grids on GitHub:
https://github.com/agdturner/grids
[https://avatars0.githubusercontent.com/u/5878594?v=3&s=400]<https://github.com/agdturner/grids>

agdturner/grids
grids - A Java spatial raster data handling library.
Read more...<https://github.com/agdturner/grids>



There is a bit more history of grids here:
http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/grids/

Any questions I'll be happy to try to answer them and provide examples etc.

Cheers,

Andy


________________________________
From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

We are starting to look at replacing JAI<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
[https://avatars2.githubusercontent.com/u/1187031?v=3&s=400]<https://github.com/geotools/geotools/wiki/Replace-JAI>

Replace JAI · geotools/geotools Wiki · GitHub
geotools - Official GeoTools repository ... You can clone with HTTPS
Read more...<https://github.com/geotools/geotools/wiki/Replace-JAI>



If you have any suggestions on parties interested in the above tasks we would love to know.


--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...<mailto:dan...@we...>> wrote:
Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users


------------------------------------------------------------------------------

_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] Grids

From: Peter B. <pet...@ya...> - 2015-12-03 16:14:37

Thanks Andy-    I appreciate the info. Something to consider in the future. Maybe the dev team should take a look at it as they investigate replacing JAI? 

Looking forward to some examples/docs whenever you get to it.
Take Care,Peter

      From: Andy Turner <A.G...@le...>
 To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...>; Peter Borissow <pet...@ya...> 
Cc: GeoTools Users <geo...@li...>
 Sent: Thursday, December 3, 2015 10:18 AM
 Subject: Re: Grids

#yiv1662499447 #yiv1662499447 -- P {margin-top:0;margin-bottom:0;}#yiv1662499447 Yes, no_data values can be set. In calculations, no_data values are treated as I have thought was appropriate, for example in generalisations that calculate averages etc. You can request the value of the Grid at a point and there are utility type methods that get all values within a set distance in coordinate space.
The input data does not need to be tiled. I load input data into Grids using an ESRI ASCII grid format. Typically this tiles the data the number of rows and the number of columns in each tile (chunk) can be set and don't have to be the same.
You can do everything in Grids without JAI, so it is not really required, but more of an optional for storage of data in chunks. However, not all the Grids source will compile without it. I did want to separate this out completely and have the JAI data store as a sort of plugin, but have not got around to this - and from the sounds of it, it might be best to factor JAI out completely anyway.
Grids depends on my Generic library (https://github.com/agdturner/generic) which helps handle the swapping of data (amongst other things, in particular calculations using BigDecimal arithmetic). The Generic library itself depends on Apache Commons Math. Again, that is not much of a dependency and could be factored out.
I don't have a lot of time right now, but I'll work on some examples and reply in due course.
Cheers,
Andy

From: Peter Borissow <pet...@ya...>
Sent: 03 December 2015 14:50
To: Andy Turner; Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: Grids Hi Andy-    Sounds interesting. Does the Grids library support no_data and random access to a region of interest? Does the input raster need to be tiled? Is JAI required? Any other dependencies? Any known limitations?

Also, it would be interesting to have a couple simple examples on your github page for how to load in a raster and access pixel values at a given point on the image. 

Thanks,
Peter

From: Andy Turner <A.G...@le...>
To: Jody Garnett <jod...@gm...>; teeschke <dan...@we...>
Cc: GeoTools Users <geo...@li...>
Sent: Sunday, November 29, 2015 2:55 PM
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

One thing I think it is worth pointing out about Grids (as it is fairly fundamental) is that the underlying coordinate system is based on BigDecimal. The calculations for where a grid value is located are based on BigDecimal arithmetic. This is because otherwise there can be problems when dealing with rasters with very large numbers of rows and columns. Although Grids will handle multiple rasters with very large numbers of rows and columns, there are non memory/disk resource restrictions on how many rows and columns any grid can have and this will vary depending on how the definition of the chunks (data model used to store these).
I have not parallelised anything much in grids, but many grid operations can be parallelised and I have thought about this in respect of memory handling.
Some of the memory handling catches OutOfMemoryErrors to deal with problems if they do come up. It is much better though to test available memory to judge if the grids processing component is running low. In the Generic library upon which Grids depends there are useful functions for this.

From: Andy Turner <A.G...@le...>
Sent: 29 November 2015 19:27
To: Jody Garnett; teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain Hi Jody, Hi everyone,
Many moons ago I started developing a raster data calculator based on the Raster class of GeoTools. I developed this so that it tiles/chunks 2D square celled rasters into chunks of the same size, but configurable rows and columns. I called the library Grids. Every Grid in Grids currently either holds ints or doubles, but I had envisaged developing ones for boolean and BigDecimal too. Every Grid shares a Grids_Environment object for memory handling, so if when processing the system runs low on memory, various data/tiles/chunks can be swapped to file space. There can be quite a lot involved in being clever about swapping data to and from different memory stores. Anyway, the library can handle multiple large grids on modest resources for many operations. Perhaps this can be evolved to be the JAI replacement.
In Grids there are various ways of storing chunks. They can be stored as HashMaps or Arrays or in other ways depending on their size. At present these can be stored as JAI images too, so currently there is a dependency on JAI for these, but it is not necessary at all.
Here is a link to the latest Grids on GitHub:https://github.com/agdturner/grids
|  | agdturner/gridsgrids - A Java spatial raster data handling library.Read more... |

There is a bit more history of grids here:http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/grids/

Any questions I'll be happy to try to answer them and provide examples etc.
Cheers,
Andy

From: Jody Garnett <jod...@gm...>
Sent: 29 November 2015 18:40
To: teeschke
Cc: GeoTools Users
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain We are starting to look at replacing JAI, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.
|  | Replace JAI · geotools/geotools Wiki · GitHubgeotools - Official GeoTools repository ... You can clone with HTTPSRead more... |

If you have any suggestions on parties interested in the above tasks we would love to know.

--Jody Garnett
On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:

Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

------------------------------------------------------------------------------

_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Neal D. <ndi...@al...> - 2015-11-29 20:26:53

Hello Jody,

I would suggest it might be worth talking to the folks at IDR Solutions.  They have a replacement that could work if an agreement could be worked out.  Information here: https://www.idrsolutions.com/jdeli

Regards,
Neal

PS:  If this works out perhaps that will get the GeoTiff issue moved up ;)

From: Jody Garnett [mailto:jod...@gm...]
Sent: Sunday, November 29, 2015 10:41 AM
To: teeschke <dan...@we...>
Cc: GeoTools Users <geo...@li...>
Subject: Re: [Geotools-gt2-users] embedding GeoTools is always a pain

We are starting to look at replacing JAI<https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be doing design and fundraising over the course of 2016. JAI is not open source so we are unable to place it on maven central. The second part is service provider interface is a pain in some environments, and the "service registry" we use looks to be unsupported in Java 9.

If you have any suggestions on parties interested in the above tasks we would love to know.


--
Jody Garnett

On 24 November 2015 at 14:35, teeschke <dan...@we...<mailto:dan...@we...>> wrote:
Dear GeoTools users and developers,

after embedding GeoTools in my n-th project I am very annoyed about the
complexity and pitfalls of this process. Two Scenarios to describe the
problem:

First: I wanted to parse a KML file and try to follow the Geometry Guide on
http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
because I learned my lessons from projects before
2. trying to resolve the KMLConfiguration
a) adding gt-xml and gt-geometry modules as dependencies which is not
documented but seems to be meaningful because the how-to is about this
b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
the xsd folder and it sounds meaningful as well
--> lucky me: it was the right one
Sidenote: I needed to scroll through the boundless repository manually to
scan for this modules

Second Scenario: whenever I want to do something with grid files - the JAI
library is required but not hosted in the maven central or boundless repo.
So I must use another repository (osgeo/webda) which seems to be the last
one which is hosting the JAI library :)

At the end of the day all this searching and trying needs a lot of time, is
not a pleasure and from my point of view not state-of-the-art because all
other libraries are available at maven central and are able to manage their
own dependencies. But in case of GeoTools I always need to look into
reference projects to find out how I was getting it the last time.

What are your experience with scenarios like this? Are you adding all
modules into the pom to not run into problems I always got or is there a
how-to I missed? :)

Thanks and kind regards,
Tee




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
Sent from the geotools-gt2-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
GeoTools-GT2-Users mailing list
Geo...@li...<mailto:Geo...@li...>
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

Re: [Geotools-gt2-users] embedding GeoTools is always a pain

From: Jody G. <jod...@gm...> - 2016-01-18 07:15:19

That is interesting, JDeli does not appear to be open source - which is the
same trouble we have with JAI.

Of note the ImageIO code does not seem to have such a difficult license, it
is mostly the JAI interfaces that are a problem.

--
Jody Garnett

On 29 November 2015 at 12:26, Neal Dillman <ndi...@al...>
wrote:

> Hello Jody,
>
>
>
> I would suggest it might be worth talking to the folks at IDR Solutions.
> They have a replacement that could work if an agreement could be worked
> out.  Information here: https://www.idrsolutions.com/jdeli
>
>
>
> Regards,
>
> Neal
>
>
>
> PS:  If this works out perhaps that will get the GeoTiff issue moved up ;)
>
>
>
> *From:* Jody Garnett [mailto:jod...@gm...]
> *Sent:* Sunday, November 29, 2015 10:41 AM
> *To:* teeschke <dan...@we...>
> *Cc:* GeoTools Users <geo...@li...>
> *Subject:* Re: [Geotools-gt2-users] embedding GeoTools is always a pain
>
>
>
> We are starting to look at replacing JAI
> <https://github.com/geotools/geotools/wiki/Replace-JAI>, and will be
> doing design and fundraising over the course of 2016. JAI is not open
> source so we are unable to place it on maven central. The second part is
> service provider interface is a pain in some environments, and the "service
> registry" we use looks to be unsupported in Java 9.
>
>
>
> If you have any suggestions on parties interested in the above tasks we
> would love to know.
>
>
>
>
>
> --
>
> Jody Garnett
>
>
>
> On 24 November 2015 at 14:35, teeschke <dan...@we...> wrote:
>
> Dear GeoTools users and developers,
>
> after embedding GeoTools in my n-th project I am very annoyed about the
> complexity and pitfalls of this process. Two Scenarios to describe the
> problem:
>
> First: I wanted to parse a KML file and try to follow the Geometry Guide on
>
> http://docs.geotools.org/stable/userguide/library/xml/geometry.html#kml-parser
> 1. add boundless repository and include gt-api, gt-main and gt-epsg-hsql
> because I learned my lessons from projects before
> 2. trying to resolve the KMLConfiguration
> a) adding gt-xml and gt-geometry modules as dependencies which is not
> documented but seems to be meaningful because the how-to is about this
> b) adding the xsd/gt-xsd-kml module as dependencies because I stumbled into
> the xsd folder and it sounds meaningful as well
> --> lucky me: it was the right one
> Sidenote: I needed to scroll through the boundless repository manually to
> scan for this modules
>
> Second Scenario: whenever I want to do something with grid files - the JAI
> library is required but not hosted in the maven central or boundless repo.
> So I must use another repository (osgeo/webda) which seems to be the last
> one which is hosting the JAI library :)
>
> At the end of the day all this searching and trying needs a lot of time, is
> not a pleasure and from my point of view not state-of-the-art because all
> other libraries are available at maven central and are able to manage their
> own dependencies. But in case of GeoTools I always need to look into
> reference projects to find out how I was getting it the last time.
>
> What are your experience with scenarios like this? Are you adding all
> modules into the pom to not run into problems I always got or is there a
> how-to I missed? :)
>
> Thanks and kind regards,
> Tee
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/embedding-GeoTools-is-always-a-pain-tp5238366.html
> Sent from the geotools-gt2-users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple
> OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> _______________________________________________
> GeoTools-GT2-Users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>
>
>