geotools-gt2-users

Re: [Geotools-gt2-users] Performance writing SHP files (for the record)

[César M. I. <ces...@gm...>]

Hello again Gabriel,

I finally had some time to had a look to this and implemented it using
FeatureWriter, as you suggested.
The same process takes now about 1 second, which is a pretty good result.

Best regards,

César


El día 11 de enero de 2010 10:10, César Martínez Izquierdo
<ces...@gm...> escribió:
> Thanks for your replies.
>
> Gabriel, I've attached the full class now, so that you can get a
> better idea of the code.
>
> The "create" is called just once by Sextante framework on object
> initialization. This method defines one transaction, which will be
> used all the time.
>
> The "addFeature" method is called from a loop, as much times as
> features are going to be added to the file.
> I'd prefer to don't build a feature collection containing all the
> features as this will create the full layer in memory, which is a bad
> idea for really big layers (as it is the case). I guess I'll need to
> use the FeatureWriter, as you suggest.
>
> Thanks for your time.
>
>
> César
>
>
>
>
>
>
> El día 8 de enero de 2010 20:42, Gabriel Roldan <gr...@op...> escribió:
>> I don't see the loop in the sample code. It certainly looks like if all that
>> code is inside a control loop executed 14k times it might perform badly.
>>
>> Also, it doesn't look like you're using a transaction. Though I'm not
>> completely sure about shapefile datastore performances it could be that
>> working in auto commit mode is a performance waste.
>>
>> You can try two things:
>> - build the feature collection with all the features to add first and then
>> make a single call to store.addFeatures
>> - Acquire a FeatureWriter out of the DataStore instead.
>>
>> in either case try to set a transaction before adding all the features and
>> commit it when done.
>>
>> Transaction tr = new DefaultTransaction();
>> FeatureStore store = ...
>> store.setTransaction(tr);
>> try{
>> ... add features...
>>  tr.commit();
>> }catch(Exception e){
>>  tr.rollback();
>> }
>>
>> sort of..
>>
>> Cheers,
>> Gabriel
>>
>> César Martínez Izquierdo wrote:
>>>
>>> Some additional notes on this topic. The reported time (~63 minutes)
>>> happens when the FeatureStore is a ShapefileDataStore.
>>> When using a MemoryDataStore the process is much faster (~5 minutes).
>>>
>>> However, I still think it is a lot of time, as the same algorithm
>>> execution takes about 3 seconds when using the gvSIG-Sextante
>>> bindings. Moreover, the MemoryDataStore is not a suitable solution for
>>> big layers.
>>>
>>> What's wrong with the code bellow?
>>>
>>> Regards,
>>>
>>> César
>>>
>>> El día 21 de diciembre de 2009 16:19, César Martínez Izquierdo
>>> <ces...@gm...> escribió:
>>>>
>>>> Hello list,
>>>>
>>>> I'm using Sextante through the GeoTools-Sextante bindings to create a
>>>> Graticule (using CreateGraticuleBuilder algorithm form Sextante).
>>>> This creates a shapefile containing a graticule of rectangles.
>>>>
>>>> However, performance is really poor: ~ 63 minutes to just write around
>>>> 14000 features.
>>>> The relevant part of code is:
>>>>                               FeatureStore<SimpleFeatureType,
>>>> SimpleFeature> store =
>>>> ((FeatureStore<SimpleFeatureType, SimpleFeature>) getFeatureSource());
>>>>                               List<Object> attributes = new
>>>> ArrayList<Object>();
>>>>                               attributes.add(geom);
>>>>                               attributes.addAll(Arrays.asList(values));
>>>>                               SimpleFeatureType ft = store.getSchema();
>>>>                               FeatureCollection<SimpleFeatureType,
>>>> SimpleFeature> collection =
>>>> FeatureCollections
>>>>                                               .newCollection();
>>>>
>>>>                               SimpleFeature feature =
>>>> SimpleFeatureBuilder.build(ft, attributes,
>>>> SimpleFeatureBuilder.createDefaultFeatureId());
>>>>                               collection.add(feature);
>>>>                               store.addFeatures(collection);
>>>> Note that "geom" is a JTS geometry, and "values" is an Object[].
>>>>
>>>> I've been measuring times, and 99% of the time is spent in the last line:
>>>>
>>>>   store.addFeatures(collection);
>>>>
>>>> Is there a better way to achieve this with GeoTools or should I bypass
>>>>        the FeatureStore layer in GeoTools and go directly to
>>>> ShapefileWriter?
>>>> I would really appreciate any clue on this.
>>>>
>>>> Regards,
>>>>
>>>> César.
>>>>
>>>>
>>>>
>>>> --
>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>  César Martínez Izquierdo
>>>>  GIS developer
>>>>  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>>>  ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>>>  Universitat Autònoma de Barcelona (SPAIN)
>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Gabriel Roldan
>> OpenGeo - http://opengeo.org
>> Expert service straight from the developers.
>>
>
>
>
> --
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   César Martínez Izquierdo
>   GIS developer
>   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>   Universitat Autònoma de Barcelona (SPAIN)
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>



-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   César Martínez Izquierdo
   GIS developer
   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
   Universitat Autònoma de Barcelona (SPAIN)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Re: [Geotools-gt2-users] Performance writing SHP files (for the record)

[Gabriel R. <gr...@op...>]

Hi Cesar,

that's good news, glad you sorted it out.

Gabriel
On 4/13/10 9:16 AM, César Martínez Izquierdo wrote:
> Hello again Gabriel,
>
> I finally had some time to had a look to this and implemented it using
> FeatureWriter, as you suggested.
> The same process takes now about 1 second, which is a pretty good result.
>
> Best regards,
>
> César
>
>
> El día 11 de enero de 2010 10:10, César Martínez Izquierdo
> <ces...@gm...>  escribió:
>> Thanks for your replies.
>>
>> Gabriel, I've attached the full class now, so that you can get a
>> better idea of the code.
>>
>> The "create" is called just once by Sextante framework on object
>> initialization. This method defines one transaction, which will be
>> used all the time.
>>
>> The "addFeature" method is called from a loop, as much times as
>> features are going to be added to the file.
>> I'd prefer to don't build a feature collection containing all the
>> features as this will create the full layer in memory, which is a bad
>> idea for really big layers (as it is the case). I guess I'll need to
>> use the FeatureWriter, as you suggest.
>>
>> Thanks for your time.
>>
>>
>> César
>>
>>
>>
>>
>>
>>
>> El día 8 de enero de 2010 20:42, Gabriel Roldan<gr...@op...>  escribió:
>>> I don't see the loop in the sample code. It certainly looks like if all that
>>> code is inside a control loop executed 14k times it might perform badly.
>>>
>>> Also, it doesn't look like you're using a transaction. Though I'm not
>>> completely sure about shapefile datastore performances it could be that
>>> working in auto commit mode is a performance waste.
>>>
>>> You can try two things:
>>> - build the feature collection with all the features to add first and then
>>> make a single call to store.addFeatures
>>> - Acquire a FeatureWriter out of the DataStore instead.
>>>
>>> in either case try to set a transaction before adding all the features and
>>> commit it when done.
>>>
>>> Transaction tr = new DefaultTransaction();
>>> FeatureStore store = ...
>>> store.setTransaction(tr);
>>> try{
>>> ... add features...
>>>   tr.commit();
>>> }catch(Exception e){
>>>   tr.rollback();
>>> }
>>>
>>> sort of..
>>>
>>> Cheers,
>>> Gabriel
>>>
>>> César Martínez Izquierdo wrote:
>>>>
>>>> Some additional notes on this topic. The reported time (~63 minutes)
>>>> happens when the FeatureStore is a ShapefileDataStore.
>>>> When using a MemoryDataStore the process is much faster (~5 minutes).
>>>>
>>>> However, I still think it is a lot of time, as the same algorithm
>>>> execution takes about 3 seconds when using the gvSIG-Sextante
>>>> bindings. Moreover, the MemoryDataStore is not a suitable solution for
>>>> big layers.
>>>>
>>>> What's wrong with the code bellow?
>>>>
>>>> Regards,
>>>>
>>>> César
>>>>
>>>> El día 21 de diciembre de 2009 16:19, César Martínez Izquierdo
>>>> <ces...@gm...>  escribió:
>>>>>
>>>>> Hello list,
>>>>>
>>>>> I'm using Sextante through the GeoTools-Sextante bindings to create a
>>>>> Graticule (using CreateGraticuleBuilder algorithm form Sextante).
>>>>> This creates a shapefile containing a graticule of rectangles.
>>>>>
>>>>> However, performance is really poor: ~ 63 minutes to just write around
>>>>> 14000 features.
>>>>> The relevant part of code is:
>>>>>                                FeatureStore<SimpleFeatureType,
>>>>> SimpleFeature>  store =
>>>>> ((FeatureStore<SimpleFeatureType, SimpleFeature>) getFeatureSource());
>>>>>                                List<Object>  attributes = new
>>>>> ArrayList<Object>();
>>>>>                                attributes.add(geom);
>>>>>                                attributes.addAll(Arrays.asList(values));
>>>>>                                SimpleFeatureType ft = store.getSchema();
>>>>>                                FeatureCollection<SimpleFeatureType,
>>>>> SimpleFeature>  collection =
>>>>> FeatureCollections
>>>>>                                                .newCollection();
>>>>>
>>>>>                                SimpleFeature feature =
>>>>> SimpleFeatureBuilder.build(ft, attributes,
>>>>> SimpleFeatureBuilder.createDefaultFeatureId());
>>>>>                                collection.add(feature);
>>>>>                                store.addFeatures(collection);
>>>>> Note that "geom" is a JTS geometry, and "values" is an Object[].
>>>>>
>>>>> I've been measuring times, and 99% of the time is spent in the last line:
>>>>>
>>>>>    store.addFeatures(collection);
>>>>>
>>>>> Is there a better way to achieve this with GeoTools or should I bypass
>>>>>         the FeatureStore layer in GeoTools and go directly to
>>>>> ShapefileWriter?
>>>>> I would really appreciate any clue on this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> César.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>   César Martínez Izquierdo
>>>>>   GIS developer
>>>>>   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>>>>   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>>>>   Universitat Autònoma de Barcelona (SPAIN)
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gabriel Roldan
>>> OpenGeo - http://opengeo.org
>>> Expert service straight from the developers.
>>>
>>
>>
>>
>> --
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>    César Martínez Izquierdo
>>    GIS developer
>>    -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>    ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>    Universitat Autònoma de Barcelona (SPAIN)
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>
>
>
>


-- 
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.