Re: [GeoNetwork-devel] Lucene: Faster indexing

[David N. <Dav...@no...>]

Hi Francois,

I started out using geotools OracleDataStore, but had some issues (it's 
an unsupported branch) and reverted to working directly with the Oracle 
Spatial libraries.  The PostGISDatastore should go smoothly.

The option you describe below sounds great. 

Dave

Francois Prunayre wrote:
> Hello David,
>
> 2009/10/26 David Neufeld <Dav...@no...>:
>   
>> We've had some concerns about the scalability of shapefiles and have
>> done some successful prototyping of inserting the metadata extents into
>> Oracle Spatial.  Is anyone else working or thinking along these lines?
>>     
> Maybe an option could be to use shapefile by default and add an option
> to use db resource when spatial option is available (ie. Oracle and
> PostGIS) ?
> Moving from a ShapefileDataStore to an other type of Datastore should
> not be so difficult. Did you use the OracleDataStore ?
>
> Cheers.
>
> Francois
>
>
>
>   
>> Thanks, Dave
>>
>> David Neufeld
>> Enterprise Data Systems Group
>> NOAA, NGDC, CIRES
>> (303) 497-6507
>> Dav...@no...
>>
>>
>>
>> James Wilson wrote:
>>     
>>> Is the problem within Lucene, or within the shapefile that GeoNetwork creates
>>> using geotools? In my (limited) experience with shapefiles / geotools,
>>> adding to a shapefile using a transaction gets progressively slower as the
>>> number of features in the shapefile grows. I believe geotools copies the
>>> file to a temporary location, then reindexes whole file. This should scale
>>> worse than linearly, but I'm guessing not much worse than  N log N.
>>>
>>> It might be worth trying to insert some logging info into SearchManager to
>>> push out timings for different bits of the indexing processing.
>>>
>>> Interested to see how you get on with that volume of records.
>>>
>>> James
>>>
>>>
>>> Juan Carlos Méndez wrote:
>>>
>>>       
>>>> Hi.  I'm having some issues related to metadata indexing speed:
>>>> It takes more than 20 seconds to index a metadata in a collection of
>>>> 33.000
>>>> records...
>>>> It seems that indexing time degrades over time... and I need to index
>>>> 180.000 records ...
>>>>
>>>> Is there any way to improve speed of metadata indexing?
>>>>
>>>> thanks for your help
>>>>
>>>> Juan Carlos Méndez
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>>>> is the only developer event you need to attend this year. Jumpstart your
>>>> developing skills, take BlackBerry mobile applications to market and stay
>>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>>>> http://p.sf.net/sfu/devconference
>>>> _______________________________________________
>>>> GeoNetwork-devel mailing list
>>>> Geo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>>>> GeoNetwork OpenSource is maintained at
>>>> http://sourceforge.net/projects/geonetwork
>>>>
>>>>
>>>>         
>>>       
>> ------------------------------------------------------------------------------
>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>> is the only developer event you need to attend this year. Jumpstart your
>> developing skills, take BlackBerry mobile applications to market and stay
>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>> http://p.sf.net/sfu/devconference
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> Geo...@li...
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
>>