Re: [gentle] Processing large amounts of objects

[nntp://news.gmane.org/gmane.comp.windows.dotnet.gentle.devel <jam...@em...>]

I could do that to try and put off the point at which an out of virtual 
memory exception occurs. However what I would like to do is mimic the 
behaviour of a data reader and sequentially read through the result set and 
update each object one at a time, rather than as now happens where all 
objects are returned.

James


"Drazen Penic" <Dra...@ck...> wrote in message 
news:001e01c69844$e3b47390$0401a8c0@DH22YS0J...
> You can try something like this:
> 1. can you create proxy object that will not read entire database row, 
> just
> fields needed for indexing, so you can decrease amount of memory needed.
> 2. partition your data source in some way, so you can read and index 
> blocks
> of data:
>    - Create integer primary key, count  number of records and then create
> index ranges. I you read one by one it will probably be very slow.
> Experiment with block size.
>    - Create proxy object that reads only primary key field in memory, read
> list of all keys in memory, scan through that list and  create queries 
> that
> will fetch blocks of data for indexing.
>
> drazen
>
>
> ----- Original Message ----- 
> From: "nntp://news.gmane.org/gmane.comp.windows.dotnet.gentle.devel"
> <jam...@em...>
> To: <gop...@li...>
> Sent: Sunday, June 25, 2006 11:24 AM
> Subject: Re: [gentle] Processing large amounts of objects
>
>
>> In the database there are around 750k rows which I need to iterate over
>> so that I can generate an index for searching.
>>
>> If I use SQLStatement and SQLResult to iterate then Gentle reads all the
>> objects into memory before I get the chance to process each object. What
>> I want to be able to do is update the index and then discard each object
>> as soon as I have finished with it so that the application does not run
>> out of memory.
>
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job 
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642