Menu
▾
▴
Re: [Hsqldb-user] Long Time taken to read data: is it normal?/ Error
|
From: indianscorpio <coo...@gm...> - 2009-03-30 13:53:16
|
By a new index i just was giving a info :)
I just wish to know vaguely with DB of such a size (20,00,000 in one &
approx 5 time of this in second rows & fields(varchar) ~ 8+8 in both) and
both RS open how much heap is required. (I have increased it to 1500m which
was maximum available and again running it
(approx size of .data file is 2-25 GB without indexing.)
Again when I am using just 'Select * from table2' (1,00,00,000 rows) it
gives outOfMemory and seeing
the docs of hsqldb i dont think too large java heap is required for it or is
it...??
fredt wrote:
>
> When you get out of memory errors, the only solution is to increase the
> heap
> memory.
>
> Adding a new index should not reduce the number of rows you can select.
>
> In your program you are keeping a result set open and then creating a
> second
> result set. This means memory for both result sets is needed.
>
> Fred
>
> ----- Original Message -----
> From: "indianscorpio" <coo...@gm...>
> To: <hsq...@li...>
> Sent: 30 March 2009 13:02
> Subject: Re: [Hsqldb-user] Long Time taken to read data: is it normal?/
> Error
>
>
>
> I added one more index and after that .data file now reaches ~5Gb.
>
> Now I am trying to read the table2 just using DB Manager and I am getting
> OutOfMemory Error (have tried it 4-5 times S1000 error). this is weird
> because earlier also I have tried same thing using DB manager and it
> returned answer
>
> I have earlier executed String read = "SELECT fld1, fld2, fld3, min(fld4)
> from table1 statment and it used to take ~ 3min. Even table2 individually
> has taken <15-20 mins (select * from table2). But reading table2 from
> table1
> have not worked ever till now.
> Besides that this problem it is working a little arbitrarily (same
> statment
> sometimes works fine and sometines give this heap error). Does creating
> more
> index be a reason that the select statement doesn't works?
>
>
>
> fredt wrote:
>>
>>>From the queries, it is not clear what causes the long query times. The
>> quries should take a long time, but not days.
>>
>> For testing, you can add some conditions to the queries to reduce the
>> expected size of the result and check the times.
>>
>> In general, you cannot have result sets that are larger than Java heap
>> space.
>>
>> Fred
>>
>>
>> ----- Original Message -----
>> From: "indianscorpio" <coo...@gm...>
>> To: <hsq...@li...>
>> Sent: 30 March 2009 10:32
>> Subject: [Hsqldb-user] Long Time taken to read data: is it normal?/ Error
>>
>>
>>
>> Time till now: more than 3 Days (still running)
>> Size of table1: 2355459 rows, 9 fields, all varchar
>> Size of table2: approx 5 time the table1 (Today I want even able to read
>> rhe
>> data from table2 - getting error OutOfMemory- Java Heap Space)
>> have done indexing and total db size after indexing is (.data file)
>> 3713544192 Bytes (~> 3 GB)
>> Tables created were cached tables
>> code:
>> String read = "SELECT fld1, fld2, fld3, min(fld4) from table1
>> group by fld1, fld2, fld3";
>> int count = 0;
>> statmnt.execute(read);
>> ResultSet rs = statmnt.getResultSet();
>> while (rs.next()) {
>> String f1= rs.getString(1);
>> String f2= rs.getString(2);
>> String f3= rs.getString(4);
>> String read_2 = "Select f11, f12, f13, f14, f15, f16 from table2 where
>> f11
>> =
>> f1 and f12=f2 and f13 <= f3
>> group by f11, f12, f13, f14, f15, f16";
>> count++;
>> if (count == 10000) {
>> DateFormat dateFormat = new
>> SimpleDateFormat("yyyy/MM/dd
>> HH:mm:ss");
>> java.util.Date date = new java.util.Date();
>> System.out.println(dateFormat.format(date));
>> count = 0;
>> }
>> statmnt.executeQuery(read_2);
>> }
>> statmnt.execute("SHUTDOWN");
>> --
>> View this message in context:
>> http://www.nabble.com/Long-Time-taken-to-read-data%3A-is-it-normal---Error-tp22779321p22779321.html
>> Sent from the HSQLDB - User mailing list archive at Nabble.com.
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Hsqldb-user mailing list
>> Hsq...@li...
>> https://lists.sourceforge.net/lists/listinfo/hsqldb-user
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Hsqldb-user mailing list
>> Hsq...@li...
>> https://lists.sourceforge.net/lists/listinfo/hsqldb-user
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Long-Time-taken-to-read-data%3A-is-it-normal---Error-tp22779321p22781688.html
> Sent from the HSQLDB - User mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Hsqldb-user mailing list
> Hsq...@li...
> https://lists.sourceforge.net/lists/listinfo/hsqldb-user
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Hsqldb-user mailing list
> Hsq...@li...
> https://lists.sourceforge.net/lists/listinfo/hsqldb-user
>
>
--
View this message in context: http://www.nabble.com/Long-Time-taken-to-read-data%3A-is-it-normal---Error-tp22779321p22782926.html
Sent from the HSQLDB - User mailing list archive at Nabble.com.
|