[Firebird-devel] VARCHAR data compression in Firebird 2.1.x

[Dunbar, N. <nor...@en...>]

Afternoon all,

I'm documenting Firebird Internals again. Page type 05 Data Pages. I've
inserted a few records into a VARCHAR(100) column, as follows:

INSERT INTO NORMAN VALUES ('666');
INSERT INTO NORMAN VALUES ('AaaaaBbbbbbbbbbCccccccccccccccDD');
INSERT INTO NORMAN VALUES (NULL);
...
COMMIT;

And I'm dumping the data page out to see how data are stored internally.
I see the following:

tux> ./fbdump ../blank.fdb -p 166

Page Buffer allocated. 4096 bytes at address 0x804b008
Page Offset = 679936l

DATABASE PAGE DETAILS
=====================
        Page Type:          5
        Sequence:           0
        Relation:           129
        Count:              8

...

        Data[0004].offset:  3908
        Data[0004].length:  24
        Data[0004].hex:     45 01 00 00 00 00 00 00 00 00 00 00 01 01 fe
fd
                            00 02 03 00 fd 36 9f 00
        Data[0004].ASCII:    E  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.
                             .  .  .  .  .  6  .  .

...

        Data[0006].offset:  3824
        Data[0006].length:  36
        Data[0006].hex:     45 01 00 00 00 00 00 00 00 00 00 00 01 01 fe
fd
                            00 03 20 00 41 fc 61 01 42 f7 62 01 43 f2 63
02
                            44 44 bc 00
        Data[0006].ASCII:    E  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.
                             .  .     .  A  .  a  .  B  .  b  .  C  .  c
.
                             D  D  .  .

        Data[0007].offset:  3800
        Data[0007].length:  22
        Data[0007].hex:     4b 01 00 00 00 00 00 00 00 00 00 00 01 01 ff
97
                            00 00 00 00 00 00
        Data[0007].ASCII:    K  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.
                             .  .  .  .  .  .


Record 4 is the '666' one, record 6 is the
'AaaaaBbbbbbbbbbCccccccccccccccDD' one and record 7 is the NULL one.

I can understand the compression used in the '666' example - a negative
byte is inserted holding the negative count of how many times the
following byte is to be repeated - so there is 0xfd followed by '6' for
"repeat '6' three times' - so far so good.

Record 6 is giving me a headache! the RLE byte and repeated character
are there but every 4 bytes I see another byte 0x01 or 0x02 inserted.
This reminds me of the older ODS version of Index compression but the
0x02 byte confuses me - I only have one column in the table!

Can someone please explain (or point me at the code!) the stored data
for the record 6 above please.

Is my assumption correct when a NULL is involved. The length of the data
is 0x0000?


A couple of further questions, again an explanation or a pointer into
the code will be a great help, what exactly is the format/use of the
first 0x12 bytes in a column's data storage? and also, the same question
for the last two bytes?

Two of my example records above have the first 0x12 bytes the same but
the NULL record differs. All three have different last two bytes.


Much obliged as usual.


Thanks.
Norman.


Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.   We have checked this email and its attachments for viruses. But you should still check any attachment before opening it. We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation.  Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.  If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506.  Find out more about the Environment Agency at www.environment-agency.gov.uk

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation.  Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506.  Find out more about the Environment Agency at www.environment-agency.gov.uk