Recent changes to FileLayer_Implemented

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 08:22:21 -0000

--- v22
+++ v23
@@ -11,7 +11,7 @@
 * Population - give access to a *population* of records or objects. Load and store the *tuples* that contains record / object data
 * Index - give access to B-Tree structures used for indexes. allows to read, search, traverse, and update the index data
 * Bookmark - represent a reference to a record in a population, and is stored in an index
-* Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer
+* Validator - the data consistency check API is consumed by the file layer, and implemented by the [catalog](Catalog_Implemented) layer

 #Container implementation
 The container layer implements virtual file manipulation. A virtual file is a set of virtual pages. Each file has a unique collection number. 
@@ -37,7 +37,7 @@
 The **index** API implements B-Tree structures manipulation. B-Trees are used for indexes. The API provide operations like search, add and remove of indexed keys.
 B-Tree is a sophisticated tree structure. A tree node will be split in two parts if there is too many keys inside. The algorithm will try to merge together tree nodes that have too few keys. This will keep the tree B = balanced, even if there are many duplicates

-The **LobSet** API give access to a collection of BLOBs. Each BLOB has a unique number. The API provide standard java.io.InputStream and java.io.Reader interfaces to read / write BLOBs. BLOBs typically exceeds the page size - 8kb - and will be stored in several pages. BLOB feature is used by the catalog layer to store views and stored procedures
+The **LobSet** API give access to a collection of BLOBs. Each BLOB has a unique number. The API provide standard java.io.InputStream and java.io.Reader interfaces to read / write BLOBs. BLOBs typically exceeds the page size - 8kb - and will be stored in several pages. BLOB feature is used by the [catalog](Catalog_Implemented) layer to store views and stored procedures

 When data is written to a population, index or BLOB, the file layer will create shadow copies and the change will be applied permanently to the database only after the transaction has been committed. The shadow copy feature also limit the number of locks on database pages

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 08:20:48 -0000

--- v21
+++ v22
@@ -42,3 +42,6 @@
 When data is written to a population, index or BLOB, the file layer will create shadow copies and the change will be applied permanently to the database only after the transaction has been committed. The shadow copy feature also limit the number of locks on database pages

 Access to the data use the [datatype framework](DataType_Implemented) layer, that provide a generic interfaces to load / store data regardless of the type of the data
+
+#See also
+[file layer](Physical_engine) specification

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 08:10:36 -0000

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 08:10:35 -0000

--- v19
+++ v20
@@ -14,15 +14,16 @@
 * Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer

 #Container implementation
-The container layer implements virtual file manipulation. Each file has a unique collection number. 
+The container layer implements virtual file manipulation. A virtual file is a set of virtual pages. Each file has a unique collection number. 
 It implements shadow copies and multi-version concurrency. When the content layer write data to a page, the container first duplicate the page, and then applies changes to the copy. There can be more than one copy of each page
 The changes will be applied to the origin page when the changeset is committed
 When a changeset is committed, the shadow copies it contains will immediately be visible to all threads. 

 The container layer do not keep the full history of changes in the database: FileSpaceRecyclerThread is a background thread that will merge together all committed changesets - it will visit changeset's shadow copies, apply the changes to the original page and delete the  shadow copies. 

-The container layer implements space allocation, and reuse of free space in the database file. It maintains bitmaps where '1' means the page is in-use. The bitmaps may be rebuilt if broken. The container layer also implements a primary B-Tree index of the pages. The B-Tree structure is maintained by the container layer. This critical database structure may be rebuilt if broken. This B-Tree structure is also used by the content layer to implements clustered storage
-The container implements the banker's algorithm in order to secure concurrent changes of the database file - avoid deadlocks and overwrite that may lead to a broken database
+The container layer implements a primary B-Tree index of the pages, in order to retrieve pages using their *virtual location*. The B-Tree structure is maintained by the container layer. This critical database structure may be rebuilt if broken. This B-Tree structure is also used by the content layer to implements clustered storage
+
+The container layer also implements space allocation, and reuse of free space in the database file. It maintains bitmaps where '1' means the page is in-use. The bitmaps may be rebuilt if broken. The container implements the banker's algorithm in order to secure concurrent changes of the database file - avoid deadlocks and overwrite that may lead to a broken database

 The container layer also provide file backup and restore API, as well as API to check the database structure, recover data and repair damages

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 08:01:24 -0000

--- v18
+++ v19
@@ -19,7 +19,7 @@
 The changes will be applied to the origin page when the changeset is committed
 When a changeset is committed, the shadow copies it contains will immediately be visible to all threads. 

-The container layer do not keep the full history of changes in the database: FileSpaceRecyclerThread is a background thread that will merge together all committed changesets - it will visit changeset's shadow copies, apply the changes to the original page and delete the  empty changesets. 
+The container layer do not keep the full history of changes in the database: FileSpaceRecyclerThread is a background thread that will merge together all committed changesets - it will visit changeset's shadow copies, apply the changes to the original page and delete the  shadow copies. 

 The container layer implements space allocation, and reuse of free space in the database file. It maintains bitmaps where '1' means the page is in-use. The bitmaps may be rebuilt if broken. The container layer also implements a primary B-Tree index of the pages. The B-Tree structure is maintained by the container layer. This critical database structure may be rebuilt if broken. This B-Tree structure is also used by the content layer to implements clustered storage
 The container implements the banker's algorithm in order to secure concurrent changes of the database file - avoid deadlocks and overwrite that may lead to a broken database

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:58:44 -0000

--- v17
+++ v18
@@ -7,7 +7,6 @@

 The file layer implementation is in org.whitebear.file. The file layer is stable since 2024
 #Main interfaces
-* Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer
 * Transaction - give access to indexes, records and BLOBs, all changes will be applied when the transaction is committed
 * Population - give access to a *population* of records or objects. Load and store the *tuples* that contains record / object data
 * Index - give access to B-Tree structures used for indexes. allows to read, search, traverse, and update the index data

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:58:09 -0000

--- v16
+++ v17
@@ -5,7 +5,7 @@
 * The **container** layer implements virtual files aka *collections*, shadow copies and change sets. A change set is a set of shadow copies of pages from one of more collections. Features of the container layer are mainly used by the content layer - records and indexes are stored in collections
 * The **content** layer implements reading and writing B-tree structures used for indexes, populations structures used for tables, and BLOB storage

-The file layer implementation is in org.whitebear.file
+The file layer implementation is in org.whitebear.file. The file layer is stable since 2024
 #Main interfaces
 * Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer
 * Transaction - give access to indexes, records and BLOBs, all changes will be applied when the transaction is committed

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:57:27 -0000

--- v15
+++ v16
@@ -5,6 +5,7 @@
 * The **container** layer implements virtual files aka *collections*, shadow copies and change sets. A change set is a set of shadow copies of pages from one of more collections. Features of the container layer are mainly used by the content layer - records and indexes are stored in collections
 * The **content** layer implements reading and writing B-tree structures used for indexes, populations structures used for tables, and BLOB storage

+The file layer implementation is in org.whitebear.file
 #Main interfaces
 * Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer
 * Transaction - give access to indexes, records and BLOBs, all changes will be applied when the transaction is committed

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:56:38 -0000

--- v14
+++ v15
@@ -11,6 +11,8 @@
 * Population - give access to a *population* of records or objects. Load and store the *tuples* that contains record / object data
 * Index - give access to B-Tree structures used for indexes. allows to read, search, traverse, and update the index data
 * Bookmark - represent a reference to a record in a population, and is stored in an index
+* Validator - the data consistency check API is consumed by the file layer, and implemented by the catalog layer
+
 #Container implementation
 The container layer implements virtual file manipulation. Each file has a unique collection number. 
 It implements shadow copies and multi-version concurrency. When the content layer write data to a page, the container first duplicate the page, and then applies changes to the copy. There can be more than one copy of each page

FileLayer_Implemented modified by silex6

silex6 — Fri, 20 Jun 2025 07:55:07 -0000

--- v13
+++ v14
@@ -1,5 +1,6 @@
 # File layer
 In this database software stack, the file layer implements storage structures manipulation. It implements a  *container* with *content*. 
+
 The layer is split in two different layers:
 * The **container** layer implements virtual files aka *collections*, shadow copies and change sets. A change set is a set of shadow copies of pages from one of more collections. Features of the container layer are mainly used by the content layer - records and indexes are stored in collections
 * The **content** layer implements reading and writing B-tree structures used for indexes, populations structures used for tables, and BLOB storage