Re: [Firebird-devel] RFC: Look-ahead disk space allocation and no use

[Vlad H. <hv...@us...>]

    Hi, all

Do you remember this RFC ? :)

>     Below i propose two change in a way engine worked with disk files
> 
> 1. Look-ahead disk space allocation
> 
>     To avoid problems with page buffers allocated in in-memory cache
> but not on disk i propose to grab enough disk space before any attempt 
> to use pages not allocated on disk. 
> 
>     Only one thread (process) can extend file to avoid conflicts. 
> 
>     On Windows use SetFileValidData API call on those platforms where it is 
> present (>= WinXP) to avoid very slow synchronous filling of a new place 
> with zeros.
> 
>     To make process more efficient i offer to allocate space for several pages at 
> a time. Low and high number of batch of allocated pages limited by hard-coded
> constants (i propose 128KB and 128MB but this is can be discussed of course).
> Engine take 1/16 part of already allocated space and adjust this number against
> low and high limits above (1/16 also can be discussed). If look-ahead allocation 
> failed engine divided requested number of pages by factor of 2 and repeat 
> process until success or requested number more than zero. If request to allocate 
> one page is failed too error is raised (also we can log it into firebird.log)
> 
>     Constants (128KB, 128MB and 1/16 part of already allocated space) are chosen 
> so to not grow small database too fast from the one side and to allocate enough 
> number of pages to not extend file too often and reduce disk fragmentation from 
> the other side.

    This part is reviewed by project leader and ready to be committed. After our first
conversation i made following changes :

a) Engine tracked really used pages (i.e. allocated by means of Firebird). To do it
    efficiently i use one reserved long on pointer page (this is also allow to not introduce
    new major ODS number). Nbackup also know about it

b) When allocating new page engine write it to disk immediately (if this page was never
    allocated before). This allows to detect out of disk space condition early and handle
    this error gracefully. Also this make file growth sequentially and improve performance
    on bulk inserts when database grow actively

c) Due to (b) i don't need SetFileValidData anymore

d) As was strongly proposed i introduce new configuration setting to control preallocation 
    of database files. This setting allow to set upper bound value for number of bytes
    preallocated and have default value of 128MB (was hardcoded in initial proposition).
    To disable disk space preallocation user can set it to zero

e) If "no reserve" flag is set in database header then no preallocation is performed

    Only thing i want to ask for is the name of new setting (as most hard part of whole work ;). 
I propose "MaxDatabaseFileGrowth"

    Opinions ?

Regards,
Vlad