Re: [Geotools-gt2-users] Performance writing SHP files (for the record)

[Gabriel R. <gr...@op...>]

Hi Cesar,

that's good news, glad you sorted it out.

Gabriel
On 4/13/10 9:16 AM, César Martínez Izquierdo wrote:
> Hello again Gabriel,
>
> I finally had some time to had a look to this and implemented it using
> FeatureWriter, as you suggested.
> The same process takes now about 1 second, which is a pretty good result.
>
> Best regards,
>
> César
>
>
> El día 11 de enero de 2010 10:10, César Martínez Izquierdo
> <ces...@gm...>  escribió:
>> Thanks for your replies.
>>
>> Gabriel, I've attached the full class now, so that you can get a
>> better idea of the code.
>>
>> The "create" is called just once by Sextante framework on object
>> initialization. This method defines one transaction, which will be
>> used all the time.
>>
>> The "addFeature" method is called from a loop, as much times as
>> features are going to be added to the file.
>> I'd prefer to don't build a feature collection containing all the
>> features as this will create the full layer in memory, which is a bad
>> idea for really big layers (as it is the case). I guess I'll need to
>> use the FeatureWriter, as you suggest.
>>
>> Thanks for your time.
>>
>>
>> César
>>
>>
>>
>>
>>
>>
>> El día 8 de enero de 2010 20:42, Gabriel Roldan<gr...@op...>  escribió:
>>> I don't see the loop in the sample code. It certainly looks like if all that
>>> code is inside a control loop executed 14k times it might perform badly.
>>>
>>> Also, it doesn't look like you're using a transaction. Though I'm not
>>> completely sure about shapefile datastore performances it could be that
>>> working in auto commit mode is a performance waste.
>>>
>>> You can try two things:
>>> - build the feature collection with all the features to add first and then
>>> make a single call to store.addFeatures
>>> - Acquire a FeatureWriter out of the DataStore instead.
>>>
>>> in either case try to set a transaction before adding all the features and
>>> commit it when done.
>>>
>>> Transaction tr = new DefaultTransaction();
>>> FeatureStore store = ...
>>> store.setTransaction(tr);
>>> try{
>>> ... add features...
>>>   tr.commit();
>>> }catch(Exception e){
>>>   tr.rollback();
>>> }
>>>
>>> sort of..
>>>
>>> Cheers,
>>> Gabriel
>>>
>>> César Martínez Izquierdo wrote:
>>>>
>>>> Some additional notes on this topic. The reported time (~63 minutes)
>>>> happens when the FeatureStore is a ShapefileDataStore.
>>>> When using a MemoryDataStore the process is much faster (~5 minutes).
>>>>
>>>> However, I still think it is a lot of time, as the same algorithm
>>>> execution takes about 3 seconds when using the gvSIG-Sextante
>>>> bindings. Moreover, the MemoryDataStore is not a suitable solution for
>>>> big layers.
>>>>
>>>> What's wrong with the code bellow?
>>>>
>>>> Regards,
>>>>
>>>> César
>>>>
>>>> El día 21 de diciembre de 2009 16:19, César Martínez Izquierdo
>>>> <ces...@gm...>  escribió:
>>>>>
>>>>> Hello list,
>>>>>
>>>>> I'm using Sextante through the GeoTools-Sextante bindings to create a
>>>>> Graticule (using CreateGraticuleBuilder algorithm form Sextante).
>>>>> This creates a shapefile containing a graticule of rectangles.
>>>>>
>>>>> However, performance is really poor: ~ 63 minutes to just write around
>>>>> 14000 features.
>>>>> The relevant part of code is:
>>>>>                                FeatureStore<SimpleFeatureType,
>>>>> SimpleFeature>  store =
>>>>> ((FeatureStore<SimpleFeatureType, SimpleFeature>) getFeatureSource());
>>>>>                                List<Object>  attributes = new
>>>>> ArrayList<Object>();
>>>>>                                attributes.add(geom);
>>>>>                                attributes.addAll(Arrays.asList(values));
>>>>>                                SimpleFeatureType ft = store.getSchema();
>>>>>                                FeatureCollection<SimpleFeatureType,
>>>>> SimpleFeature>  collection =
>>>>> FeatureCollections
>>>>>                                                .newCollection();
>>>>>
>>>>>                                SimpleFeature feature =
>>>>> SimpleFeatureBuilder.build(ft, attributes,
>>>>> SimpleFeatureBuilder.createDefaultFeatureId());
>>>>>                                collection.add(feature);
>>>>>                                store.addFeatures(collection);
>>>>> Note that "geom" is a JTS geometry, and "values" is an Object[].
>>>>>
>>>>> I've been measuring times, and 99% of the time is spent in the last line:
>>>>>
>>>>>    store.addFeatures(collection);
>>>>>
>>>>> Is there a better way to achieve this with GeoTools or should I bypass
>>>>>         the FeatureStore layer in GeoTools and go directly to
>>>>> ShapefileWriter?
>>>>> I would really appreciate any clue on this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> César.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>   César Martínez Izquierdo
>>>>>   GIS developer
>>>>>   -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>>>>   ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>>>>   Universitat Autònoma de Barcelona (SPAIN)
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gabriel Roldan
>>> OpenGeo - http://opengeo.org
>>> Expert service straight from the developers.
>>>
>>
>>
>>
>> --
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>    César Martínez Izquierdo
>>    GIS developer
>>    -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>>    ETC-LUSI: http://etc-lusi.eionet.europa.eu/
>>    Universitat Autònoma de Barcelona (SPAIN)
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>
>
>
>


-- 
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.