Re: [Geotools-gt2-users] Performance writing SHP files (for the record)

[César M. I. <ces...@gm...>]

Hello again Gabriel,

I finally had some time to had a look to this and implemented it using
FeatureWriter, as you suggested.
The same process takes now about 1 second, which is a pretty good result.

Best regards,

César


El día 11 de enero de 2010 10:10, César Martínez Izquierdo
<ces...@gm...> escribió:
> Thanks for your replies.
>
> Gabriel, I've attached the full class now, so that you can get a
> better idea of the code.
>
> The "create" is called just once by Sextante framework on object
> initialization. This method defines one transaction, which will be
> used all the time.
>
> The "addFeature" method is called from a loop, as much times as
> features are going to be added to the file.
> I'd prefer to don't build a feature collection containing all the
> features as this will create the full layer in memory, which is a bad
> idea for really big layers (as it is the case). I guess I'll need to
> use the FeatureWriter, as you suggest.
>
> Thanks for your time.
>
>
> César
>
>
>
>
>
>
> El día 8 de enero de 2010 20:42, Gabriel Roldan <gr...@op...> escribió:
>> I don't see the loop in the sample code. It certainly looks like if all that
>> code is inside a control loop executed 14k times it might perform badly.
>>
>> Also, it doesn't look like you're using a transaction. Though I'm not
>> completely sure about shapefile datastore performances it could be that
>> working in auto commit mode is a performance waste.
>>
>> You can try two things:
>> - build the feature collection with all the features to add first and then
>> make a single call to store.addFeatures
>> - Acquire a FeatureWriter out of the DataStore instead.
>>
>> in either case try to set a transaction before adding all the features and
>> commit it when done.
>>
>> Transaction tr = new DefaultTransaction();
>> FeatureStore store = ...
>> store.setTransaction(tr);
>> try{
>> ... add features...
>>  tr.commit();
>> }catch(Exception e){
>>  tr.rollback();
>> }
>>
>> sort of..
>>
>> Cheers,
>> Gabriel
>>
>> César Martínez Izquierdo wrote:
>>>
>>> Some additional notes on this topic. The reported time (~63 minutes)
>>> happens when the FeatureStore is a ShapefileDataStore.
>>> When using a MemoryDataStore the process is much faster (~5 minutes).
>>>
>>> However, I still think it is a lot of time, as the same algorithm
>>> execution takes about 3 seconds when using the gvSIG-Sextante
>>> bindings. Moreover, the MemoryDataStore is not a suitable solution for
>>> big layers.
>>>
>>> What's wrong with the code bellow?
>>>
>>> Regards,
>>>
>>> César
>>>
>>> El día 21 de diciembre de 2009 16:19, César Martínez Izquierdo
>>> <ces...@gm...> escribió:
>>>>
>>>> Hello list,
>>>>
>>>> I'm using Sextante through the GeoTools-Sextante bindings to create a
>>>> Graticule (using CreateGraticuleBuilder algorithm form Sextante).
>>>> This creates a shapefile containing a graticule of rectangles.
>>>>
>>>> However, performance is really poor: ~ 63 minutes to just write around
>>>> 14000 features.
>>>> The relevant part of code is:
>>>>                               FeatureStore<SimpleFeatureType,
>>>> SimpleFeature> store =
>>>> ((FeatureStore<SimpleFeatureType, SimpleFeature>) getFeatureSource());
>>>>                               List<Object> attributes = new
>>>> ArrayList<Object>();
>>>>                               attributes.add(geom);
>>>>                               attributes.addAll(Arrays.asList(values));
>>>>                               SimpleFeatureType ft = store.getSchema();
>>>>                               FeatureCollection<SimpleFeatureType,
>>>> SimpleFeature> collection =
>>>> FeatureCollections
>>>>                                               .newCollection();
>>>>
>>>>                               SimpleFeature feature =
>>>> SimpleFeatureBuilder.build(ft, attributes,
>>>> SimpleFeatureBuilder.createDefaultFeatureId());
>>>>                               collection.add(feature);
>>>>                               store.addFeatures(collection);
>>>> Note that "geom" is a JTS geometry, and "values" is an Object[].
>>>>
>>>> I've been measuring times, and 99% of the time is spent in the last line:
>>>>
>>>>   store.addFeatures(collection);
>>>>
>>>> Is there a better way to achieve this with GeoTools or should I bypass
>>>>        the FeatureStore layer in GeoTools and go directly to
>>>> ShapefileWriter?
>>>> I would really appreciate any clue on this.
>>>>
>>>> Regards,
>>>>
>>>> César.
>>>>
>>>>
>>>>
>>>> --
>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>  César Martínez Izquierdo
>>>>  GIS developer
>>>>  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>>>  ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>>>  Universitat Autònoma de Barcelona (SPAIN)
>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Gabriel Roldan
>> OpenGeo - http://opengeo.org
>> Expert service straight from the developers.
>>
>
>
>
> --
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   César Martínez Izquierdo
>   GIS developer
>   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>   Universitat Autònoma de Barcelona (SPAIN)
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>



-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   César Martínez Izquierdo
   GIS developer
   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
   Universitat Autònoma de Barcelona (SPAIN)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -