Recent changes to FileFormat_Implemented

FileFormat_Implemented modified by silex6

silex6 — Sat, 21 Jun 2025 15:29:39 -0000

--- v13
+++ v14
@@ -38,3 +38,6 @@
 The backup process will sequentially load each page from the database file. If the loaded page is a data page or BLOB, then the backup process will copy the *content section* of the page to the backup file. Bitmaps and master B-Tree index pages will not be backed-up, as not needed - these structures will be rebuilt / updated during the restore operation. The incremental backup process consider the page serial number. No change of the serial number = no change on the page.

 The restore process will load data from the backup file and populate the file with the content. Full restore first create a blank database file. The loaded content is added to a change set. The indexes will be updated during commit.
+
+##See also
+[File format](File_format) specification

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:43:55 -0000

--- v12
+++ v13
@@ -35,5 +35,6 @@
 * Then the system lock all the pages in the scope, and actually apply the change. Nothing is changed before all the pages gain an exclusive lock

 ##Backup and restore
-The backup process will sequentially load each page from the database file. If the loaded page is a data page or BLOB, then the backup process will copy the *content section* of the page to the backup file. Bitmaps and master B-Tree index pages will not be backed-up, as not needed. The incremental backup process also consider the page serial number. No change of the serial number = no change on the page
+The backup process will sequentially load each page from the database file. If the loaded page is a data page or BLOB, then the backup process will copy the *content section* of the page to the backup file. Bitmaps and master B-Tree index pages will not be backed-up, as not needed - these structures will be rebuilt / updated during the restore operation. The incremental backup process consider the page serial number. No change of the serial number = no change on the page.
+
 The restore process will load data from the backup file and populate the file with the content. Full restore first create a blank database file. The loaded content is added to a change set. The indexes will be updated during commit.

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:40:14 -0000

--- v11
+++ v12
@@ -30,7 +30,7 @@
 The *content* section of each data or LOB page also start with a 4 bytes signature

 ##Banker's algorithm
-In order to avoid deadlocks and concurrent changes that led to broken database structure, the file layer implements a 2-phase updates - the banker's algorithm:
+In order to avoid deadlocks and conflicting changes that led to broken database structure, the file layer implements a 2-phase updates - the banker's algorithm:
 * during the first phase, the software simulate the data change, and record the scope - the set of pages that may need to be updated
 * Then the system lock all the pages in the scope, and actually apply the change. Nothing is changed before all the pages gain an exclusive lock

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:35:31 -0000

--- v10
+++ v11
@@ -10,7 +10,7 @@
 * a *LOB* page has similar structure as a data page, but is intended to be used for BLOB storage.
 * an *index* node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
     The index is used to map page virtual address (changeset+collection+page) to actual page location
-* a *bitmap* page contains a 35k bits map. Nth bit is '1' means the Nth page is in use. 
+* a *bitmap* page contains an array of 35k bits. Nth bit is '1' means the Nth page is in use. 
      A bitmap cover the usage of ~300 MB. The software will add a bitmap on the beginning of each 300 MB section.
 * an *empty* page. has a signature to indicate the page has been deleted and can be re-used
 * each *change set *has a header page with current status of the changeset

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:34:24 -0000

--- v9
+++ v10
@@ -5,17 +5,17 @@
 Each 8kb page starts with a 4 bytes signature used to identify / validate the type of the page.

 The **file pages** may be the following:
-* a data page. The header of each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page. 
+* a *data* page. The header of each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page. 
       This header part is followed by a *content* section that can be used to store data
-* a LOB page has similar structure as a data page, but is intended to be used for BLOB storage.
-* a bitmap page contains a 35k bits map. Nth bit is '1' means the Nth page is in use. 
+* a *LOB* page has similar structure as a data page, but is intended to be used for BLOB storage.
+* an *index* node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
+    The index is used to map page virtual address (changeset+collection+page) to actual page location
+* a *bitmap* page contains a 35k bits map. Nth bit is '1' means the Nth page is in use. 
      A bitmap cover the usage of ~300 MB. The software will add a bitmap on the beginning of each 300 MB section.
-* an index node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
-    The index is used to map page virtual address (changeset+collection+page) to actual page location
-* an empty page. has a signature to indicate the page has been deleted and can be re-used
-* each change set has a header page with current status of the changeset
-* each page has a serial number, used to track changes and implement incremental backup. 
-   A new serial number is generated each time a page has been updated.  
+* an *empty* page. has a signature to indicate the page has been deleted and can be re-used
+* each *change set *has a header page with current status of the changeset
+
+each page has a serial number, used to track changes and implement incremental backup.  A new serial number is generated each time a page has been updated.  

 The **virtual address** of a page consist of the following:
 * changeset number. Non-zero for shadow copies

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:32:56 -0000

--- v8
+++ v9
@@ -8,7 +8,7 @@
 * a data page. The header of each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page. 
       This header part is followed by a *content* section that can be used to store data
 * a LOB page has similar structure as a data page, but is intended to be used for BLOB storage.
-* a bitmap page contains a 35k bits map, where '1' means the corresponding page is in use. 
+* a bitmap page contains a 35k bits map. Nth bit is '1' means the Nth page is in use. 
      A bitmap cover the usage of ~300 MB. The software will add a bitmap on the beginning of each 300 MB section.
 * an index node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
     The index is used to map page virtual address (changeset+collection+page) to actual page location

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:31:59 -0000

--- v7
+++ v8
@@ -8,7 +8,7 @@
 * a data page. The header of each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page. 
       This header part is followed by a *content* section that can be used to store data
 * a LOB page has similar structure as a data page, but is intended to be used for BLOB storage.
-* a bitmap page contains a 8000 bits map, where '1' means the corresponding page is in use. 
+* a bitmap page contains a 35k bits map, where '1' means the corresponding page is in use. 
      A bitmap cover the usage of ~300 MB. The software will add a bitmap on the beginning of each 300 MB section.
 * an index node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
     The index is used to map page virtual address (changeset+collection+page) to actual page location

FileFormat_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:28:30 -0000

--- v6
+++ v7
@@ -1,3 +1,5 @@
+#File format
+
 In this database software stack, all data related to a schema is stored in a single file. The file is a set of 8Kb pages that may contains data.

 Each 8kb page starts with a 4 bytes signature used to identify / validate the type of the page.

FileFormat_Implemented modified by silex6

silex6 — Thu, 19 Jun 2025 20:54:08 -0000

--- v5
+++ v6
@@ -3,11 +3,12 @@
 Each 8kb page starts with a 4 bytes signature used to identify / validate the type of the page.

 The **file pages** may be the following:
-* a data page. Each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page
+* a data page. The header of each data page contains a changeset number + collection number + page number, that is the *virtual address* of the page. 
+      This header part is followed by a *content* section that can be used to store data
 * a LOB page has similar structure as a data page, but is intended to be used for BLOB storage.
 * a bitmap page contains a 8000 bits map, where '1' means the corresponding page is in use. 
      A bitmap cover the usage of ~300 MB. The software will add a bitmap on the beginning of each 300 MB section.
-* an index node. The master B-Tree index is used to retrieve data pages by their virtual address. 
+* an index node of the master B-Tree index. This index is used to retrieve data pages by their virtual address. 
     The index is used to map page virtual address (changeset+collection+page) to actual page location
 * an empty page. has a signature to indicate the page has been deleted and can be re-used
 * each change set has a header page with current status of the changeset
@@ -30,3 +31,7 @@
 In order to avoid deadlocks and concurrent changes that led to broken database structure, the file layer implements a 2-phase updates - the banker's algorithm:
 * during the first phase, the software simulate the data change, and record the scope - the set of pages that may need to be updated
 * Then the system lock all the pages in the scope, and actually apply the change. Nothing is changed before all the pages gain an exclusive lock
+
+##Backup and restore
+The backup process will sequentially load each page from the database file. If the loaded page is a data page or BLOB, then the backup process will copy the *content section* of the page to the backup file. Bitmaps and master B-Tree index pages will not be backed-up, as not needed. The incremental backup process also consider the page serial number. No change of the serial number = no change on the page
+The restore process will load data from the backup file and populate the file with the content. Full restore first create a blank database file. The loaded content is added to a change set. The indexes will be updated during commit.

FileFormat_Implemented modified by silex6

silex6 — Thu, 19 Jun 2025 20:40:58 -0000

--- v4
+++ v5
@@ -24,6 +24,8 @@
 * B-Tree indexes. stored in a virtual file as a set of nodes bound together in a tree structure. page number 0 contains the header and a link to the top of the tree. The links stored in a B-Tree are virtual addresses of either a B-Tree node, or a record
 * LOB data. a virtual file may contains a set of BLOBs (large binary data). Where each blob may exceed the 8kb size of the page. In this case the BLOB is stored as a set of chunks in different pages of a virtual file

+The *content* section of each data or LOB page also start with a 4 bytes signature
+
 ##Banker's algorithm
 In order to avoid deadlocks and concurrent changes that led to broken database structure, the file layer implements a 2-phase updates - the banker's algorithm:
 * during the first phase, the software simulate the data change, and record the scope - the set of pages that may need to be updated