Menu
▾
▴
firebird-devel
|
From: Vlad K. <hv...@us...> - 2009-04-24 12:31:16
|
All,
Recall thread "Speed difference between 2.1.2 and 2.5.0 again" by Gabor Boros from April 8.
Gabor kindly sent to me script which reproduced problem. I can confirm : SS 2.1.2 executed
script by 25-27 sec while SS HEAD executed same script by 36-38 sec. The script contains
mostly INSERTS's and i found that most time engine spend in dsql\prepare.
Before v2.5 DSQL created own memory pool for every statement. When statement is released
its pool is destroyed too. Sometime at v2.5 stage Dmitry made DSQL pool allocated from common
database pool. It allows to somewhat reduce memory consumption. But this change make DSQL
pool to redirect most of all small requests to the parent (database) pool. This is tens or even
hundreds of small memory chunks. When DSQL pool is released it must deallocate all memory
chunks allocated from parent pool and pass them to the parent pool one-by-one. This is very slow
of course.
I disabled "parent_redirect" feature of memory pools and HEAD executed script by 26-28 sec
now. The question is - what should we do in HEAD with it ? I see two options :
a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU
b) rework "parent_redirect" to request medium sized memory chunks from parent pool (say 4-8KB)
and use it to allocate memory for immediate callers. When such pool is freed it (pool) have to
deallocate from parent pool much less number of memory blocks.
Opinions ?
Regards.
Vlad
|
|
From: Adriano d. S. F. <adr...@uo...> - 2009-04-24 12:59:16
|
Vlad Khorsun escreveu: > a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU > > b) rework "parent_redirect" to request medium sized memory chunks from parent pool (say 4-8KB) > and use it to allocate memory for immediate callers. When such pool is freed it (pool) have to > deallocate from parent pool much less number of memory blocks. > Not sure if this is possible, but c) Make a pool know about its child pools, and when a child pool is destroyed it just informs his parent, so that it could free at once or leave the memory for reuse? Adriano |
|
From: Alexander p. <pes...@ma...> - 2009-04-24 13:17:34
|
On Friday 24 April 2009 16:31:00 Vlad Khorsun wrote: > I see two options : > > a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU > > b) rework "parent_redirect" to request medium sized memory chunks from > parent pool (say 4-8KB) and use it to allocate memory for immediate > callers. When such pool is freed it (pool) have to deallocate from parent > pool much less number of memory blocks. > > Opinions ? In v.3.0 we plan to use vulcan's memory allocator. Looks like it can behave better, specially under MT load. Therefore I suggest to choose option (a) - sooner of all it's no use wasting time with MemoryPool optimization. What about memory - it's cheap :) |
|
From: Vlad K. <hv...@us...> - 2009-04-27 08:20:22
|
> Vlad Khorsun escreveu:
>> a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU
>>
>> b) rework "parent_redirect" to request medium sized memory chunks from parent pool (say 4-8KB)
>> and use it to allocate memory for immediate callers. When such pool is freed it (pool) have to
>> deallocate from parent pool much less number of memory blocks.
>>
> Not sure if this is possible, but c) Make a pool know about its child
> pools, and when a child pool is destroyed it just informs his parent, so
> that it could free at once or leave the memory for reuse?
It could be free at once only when parent pool is destroyed. And parent pool
is database pool which destroyed at process exit or with last disconnect.
Regards,
Vlad
|
|
From: Vlad K. <hv...@us...> - 2009-04-27 08:29:39
|
> On Friday 24 April 2009 16:31:00 Vlad Khorsun wrote:
>> I see two options :
>>
>> a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU
>>
>> b) rework "parent_redirect" to request medium sized memory chunks from
>> parent pool (say 4-8KB) and use it to allocate memory for immediate
>> callers. When such pool is freed it (pool) have to deallocate from parent
>> pool much less number of memory blocks.
>>
>> Opinions ?
>
> In v.3.0 we plan to use vulcan's memory allocator. Looks like it can behave
> better, specially under MT load. Therefore I suggest to choose option (a) -
> sooner of all it's no use wasting time with MemoryPool optimization. What
> about memory - it's cheap :)
I have concern with this simple solution. Before v2.5 we allocated some pools
from database pool - in DFW, DYN (i don't care about it much as it is short lived
temporary pools) and in CMP which is more questionable, i think.
Disable "parent_redirect" in CMP pool (this is request's pool) will make every
request allocate 64KB at least (despite of real needs). Is it what we want ?
Regards,
Vlad
|
|
From: Alexander p. <pes...@ma...> - 2009-04-27 09:30:26
|
On Monday 27 April 2009 12:29:16 Vlad Khorsun wrote: > > On Friday 24 April 2009 16:31:00 Vlad Khorsun wrote: > >> I see two options : > >> > >> a) disable "parent_redirect". It will make HEAD work almost as 2.1, > >> AFAIU > >> > >> b) rework "parent_redirect" to request medium sized memory chunks from > >> parent pool (say 4-8KB) and use it to allocate memory for immediate > >> callers. When such pool is freed it (pool) have to deallocate from > >> parent pool much less number of memory blocks. > >> > >> Opinions ? > > > > In v.3.0 we plan to use vulcan's memory allocator. Looks like it can > > behave better, specially under MT load. Therefore I suggest to choose > > option (a) - sooner of all it's no use wasting time with MemoryPool > > optimization. What about memory - it's cheap :) > > I have concern with this simple solution. Before v2.5 we allocated some > pools from database pool - in DFW, DYN (i don't care about it much as it is > short lived temporary pools) and in CMP which is more questionable, i > think. Did you take a look - how did it 1.5 where we had no "parent_redirect" ? > Disable "parent_redirect" in CMP pool (this is request's pool) will > make every request allocate 64KB at least (despite of real needs). Is it > what we want ? On the one hand, memory is cheap today. On the other - do we have some statistics about it? If we really use not more than Xkb (where X is much than 64) per typical request, may be request single medium-size memory chunk of Xkb from parent and later continue with normal operation? It will be something like inline storage in arrays. |
|
From: Vlad K. <hv...@us...> - 2009-04-30 07:00:54
|
> On Monday 27 April 2009 12:29:16 Vlad Khorsun wrote:
>> > On Friday 24 April 2009 16:31:00 Vlad Khorsun wrote:
>> >> I see two options :
>> >>
>> >> a) disable "parent_redirect". It will make HEAD work almost as 2.1,
>> >> AFAIU
>> >>
>> >> b) rework "parent_redirect" to request medium sized memory chunks from
>> >> parent pool (say 4-8KB) and use it to allocate memory for immediate
>> >> callers. When such pool is freed it (pool) have to deallocate from
>> >> parent pool much less number of memory blocks.
>> >>
>> >> Opinions ?
>> >
>> > In v.3.0 we plan to use vulcan's memory allocator. Looks like it can
>> > behave better, specially under MT load. Therefore I suggest to choose
>> > option (a) - sooner of all it's no use wasting time with MemoryPool
>> > optimization. What about memory - it's cheap :)
>>
>> I have concern with this simple solution. Before v2.5 we allocated some
>> pools from database pool - in DFW, DYN (i don't care about it much as it is
>> short lived temporary pools) and in CMP which is more questionable, i
>> think.
>
> Did you take a look - how did it 1.5 where we had no "parent_redirect" ?
As far as i can see JrdMemoryPool used in CMP_compile2 allocates at least
16KB with malloc at creation time. Current pools will allocate at least 64KB from
OS with VirtualAlloc\mmap.
>> Disable "parent_redirect" in CMP pool (this is request's pool) will
>> make every request allocate 64KB at least (despite of real needs). Is it
>> what we want ?
>
> On the one hand, memory is cheap today. On the other - do we have some
> statistics about it? If we really use not more than Xkb (where X is much than
> 64) per typical request, may be request single medium-size memory chunk of
> Xkb from parent and later continue with normal operation? It will be
> something like inline storage in arrays.
Is it different from my option (b) above ?
Another compromiss could be to reduce REDIRECT_THRESHOLD from
current 32KB to some less value.
Regards,
Vlad
|
|
From: Alexander p. <pes...@ma...> - 2009-04-30 07:24:04
|
On Thursday 30 April 2009 11:00:48 Vlad Khorsun wrote: > > On Monday 27 April 2009 12:29:16 Vlad Khorsun wrote: > >> > On Friday 24 April 2009 16:31:00 Vlad Khorsun wrote: > >> >> I see two options : > >> >> > >> >> a) disable "parent_redirect". It will make HEAD work almost as 2.1, > >> >> AFAIU > >> >> > >> >> b) rework "parent_redirect" to request medium sized memory chunks > >> >> from parent pool (say 4-8KB) and use it to allocate memory for > >> >> immediate callers. When such pool is freed it (pool) have to > >> >> deallocate from parent pool much less number of memory blocks. > >> >> > >> >> Opinions ? > >> > > >> > In v.3.0 we plan to use vulcan's memory allocator. Looks like it can > >> > behave better, specially under MT load. Therefore I suggest to choose > >> > option (a) - sooner of all it's no use wasting time with MemoryPool > >> > optimization. What about memory - it's cheap :) > >> > >> I have concern with this simple solution. Before v2.5 we allocated > >> some pools from database pool - in DFW, DYN (i don't care about it much > >> as it is short lived temporary pools) and in CMP which is more > >> questionable, i think. > > > > Did you take a look - how did it 1.5 where we had no "parent_redirect" ? > > As far as i can see JrdMemoryPool used in CMP_compile2 allocates at > least 16KB with malloc at creation time. Current pools will allocate at > least 64KB from OS with VirtualAlloc\mmap. Suppose that since that days average RAM size per box also became at least 4 time bigger :-) May be it's time to forget the problem? |
|
From: Paul R. <pr...@ib...> - 2009-04-30 07:51:07
|
On Thursday 30 April 2009, Alexander peshkoff wrote: > On Thursday 30 April 2009 11:00:48 Vlad Khorsun wrote: > > As far as i can see JrdMemoryPool used in CMP_compile2 allocates at > > least 16KB with malloc at creation time. Current pools will allocate at > > least 64KB from OS with VirtualAlloc\mmap. > > Suppose that since that days average RAM size per box also became at > least 4 time bigger :-) May be it's time to forget the problem? I'm not following this discussion too closely, but... How would this work with several hundred connections? Running Classic perhaps? Paul -- Paul Reeves http://www.ibphoenix.com Specialists in Firebird support |
|
From: Dmitry Y. <fir...@ya...> - 2009-04-30 08:34:59
|
Vlad Khorsun wrote: > > a) disable "parent_redirect". It will make HEAD work almost as 2.1, AFAIU > > b) rework "parent_redirect" to request medium sized memory chunks from parent pool (say 4-8KB) > and use it to allocate memory for immediate callers. When such pool is freed it (pool) have to > deallocate from parent pool much less number of memory blocks. I'd go for (b) for the time being. Dmitry |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X