Re: [Firebird-devel] RFC: Look-ahead disk space allocation and nouseof file system cache

[Vlad H. <hv...@us...>]

    Sean, thank you for comments

> > 1. Look-ahead disk space allocation
> > 
> >     To avoid problems with page buffers allocated in 
> > in-memory cache but not on disk i propose to grab enough disk 
> > space before any attempt to use pages not allocated on disk. 
> > 
> >     Only one thread (process) can extend file to avoid conflicts. 
> > 
> >     On Windows use SetFileValidData API call on those 
> > platforms where it is present (>= WinXP) to avoid very slow 
> > synchronous filling of a new place with zeros.
> > 
> >     To make process more efficient i offer to allocate space 
> > for several pages at a time. Low and high number of batch of 
> > allocated pages limited by hard-coded constants (i propose 
> > 128KB and 128MB but this is can be discussed of course).
> > Engine take 1/16 part of already allocated space and adjust 
> > this number against low and high limits above (1/16 also can 
> > be discussed).
> 
> Rather than this elaborate approach, could we not simply introduce a new
> setting "minimum DB growth" which would represent the number of pages
> which a database will grow when required?

    Yes we can introduce new setting of course. But my goal was to avoid
new setting if possible ;) Another opinions ?
 
> A value range between 1 (naturally) and 16384 would constrain the value,
> with 1 as the default.
> 
> There are some installs where file growth/size is critical, a low end
> value of 128KB would be problematic.

    We can reduce this limit. But empty database have initial size of 
620KB so i don't think minimum grows of 128K is a problem. 
 
> (Ideally, this growth setting should be defined/stored in the DB)

    If we'll introduce such settings - agree it must be per database

> > If look-ahead allocation failed engine divided 
> > requested number of pages by factor of 2 and repeat process 
> > until success or requested number more than zero. If request 
> > to allocate one page is failed too error is raised (also we 
> > can log it into firebird.log)
> 
> I don't know if this is really required, but it sure sounds nice.

    This is to avoid complains such as 

'I have 1MB free disk space but Firebird failed to allocate it' 

;)

> > 2. Allow engine to not use file system cache
> > 
> >     Currently engine works with database files by using file 
> > system cache (at least on Windows). This is good (and must) 
> > for classic server with its small page cache but can hurt 
> > performance of super server configured to use really large page cache.
> > If super server will not use file system cache this allow 
> > more efficient use of memory (if there are not enough memory 
> > to fit both kind of caches) and faster access to the disk.
> 
> I don't understand the case where this is required.
> 
> If there is not enough memory for the database cache, isn't it the DBA
> responsibility to reduce the database page cache settings?

    Imagine PC with, say 1GB RAM. Imagine also database with standard 
page size 4KB and page buffers of 128 * 1024 (our current maximum). It will
take 512MB for page buffers. If database is large enough, OS (at least windows)
will keep big part of it in file system cache. Why waste memory for double
caching ? We can better use it for sorting and metadata cache,  for second 
database or for something else

> Isn't the larger issue that since the SS cache is per connection, as the
> number of connection increases, that the allocation SS cache can
> outstrip the free physical RAM (as different from Virtual Memory) and
> therefore make the entire server grind (as the OS pages memory to the
> disk)?

    Can i return to you your phrase above :

"If there is not enough memory for the database cache, isn't it the DBA
responsibility to reduce the database page cache settings? "

:)

    This is another problem i want to discuss later 

Regards,
Vlad