Re: [Firebird-devel] FB 2.1 64bit and memory usage ??

[Fred P. <fps...@ya...>]

On 07/25/2007 06:30 AM, Vlad Horsun wrote:
>> My concern: the new sector standard may impact direct I/O
>> alignment constraints for I/O buffers, transfer sizes and file
>> offsets, depending on the implementation of direct I/O used.
> 
>     We not allow database page size less than 4KB since FB 2.1
> Old databases may still use small pages however

Hi Vlad,

Good to know. Currently, Linux 2.6's O_DIRECT requires 512
byte alignment. So, no problem there!

However, some O_DIRECT implementations (e.g. Linux 2.4) want
buffer alignment to be on a *filesystem* block size boundary
and, possibly, the read/write transfer size and file offset to
be a multiple of this value. So, an fs block size might be 4k,
8k, or more, depending on how the filesystem was built.  E.g.,
AIX typically wants 4k alignment, except for JFS+big_file
filesystems, which want 128k alignment.

Don't you just love non-standard feature implementations? :)

IMHO, rather than try to make FB support every OS/filesystem
combination in the first direct I/O implementation, it seems
reasonable to start with a popular subset and gain some
experience with this new tuning feature before expanding
support. Phased implementations can be a good thing.

That said, some questions remain: how smart should FB be about
detecting, reporting and/or recovering from a user's attempt
to use direct I/O when it isn't supported or encounters a
read/write error downstream? Should FB silently fallback to
non-direct I/O and retry the I/O op?  Or something else...?

It might be nice if FB could handle this is a consistent way
across all platforms.

> ...
>     posix_fadvise free from this drawback and not restrict us to use 
> aligned memory buffers, right ?

Correct. Also, posix_fadvise() might be useful for
tuning I/O elsewhere in FB (e.g., read-ahead?), but that's
another topic. :)

To be clear, posix_fadvise() is an "advisory" POSIX feature: a
system can choose not to implement it (e.g. Solaris).  But, I
suspect most POSIX vendors will support it in time.

>> I suspect that the FB's direct I/O option on linux is going to
>> need a lot of testing.  ;-)
> 
>     As it may be disabled (and will be disabledby default) i see no 
> problem with it.

Sounds good.

>     As for testing - are you taker ? :)

:) Maybe. That will depend on my schedule at the time. I may
be able to do some basic qualitative testing on Linux 2.6.

Fred P.