Recent changes to DesignDecisions

DesignDecisions modified by Graham Shanks

Graham Shanks — Fri, 27 Jun 2014 21:49:50 -0000

--- v1
+++ v2
@@ -25,14 +25,14 @@

 There are a number of consequences of this: 

-  1. We need to store additional intermediate objects as member variables to ensure that we can release intermediate objects in the correct order 
-  2. Certain simple functions need to reorganised to ensure that objects are released. For instance the obvious implementation of the NumberOfRows methods is 
-        
+1. We need to store additional intermediate objects as member variables to ensure that we can release intermediate objects in the correct order 
+
+2. Certain simple functions need to reorganised to ensure that objects are released. For instance the obvious implementation of the NumberOfRows methods is 
+
         public int NumberOfRows { get { return myWorksheet.UsedRange.Rows.Count; } }
-        
-
-However this creates two temporary COM objects that cannot be released properly. The solution is to refactor this to replace the temporaries with assignments to actual variables so that we can release them properly: 
-        
+
+3. However this creates two temporary COM objects that cannot be released properly. The solution is to refactor this to replace the temporaries with assignments to actual variables so that we can release them properly: 
+
         public int NumberOfRows
         {
           get
@@ -47,10 +47,9 @@
             return result;
           }
         }
-        
-
-  3. Care needs to be taken to ensure that objects are released even in the presence of exceptions - in practise this means that methods that use interop functions need to include exception handlers to perform proper clean-up. In fact the actual implementation of NumberOfRows is: 
-        
+
+4. Care needs to be taken to ensure that objects are released even in the presence of exceptions - in practise this means that methods that use interop functions need to include exception handlers to perform proper clean-up. In fact the actual implementation of NumberOfRows is: 
+
         public int NumberOfRows
         {
           get
@@ -80,18 +79,19 @@
         }

-  4. Users really need to call the _Dispose_ method on the Excel wrappers (e.g. CRfile and CRstatus). In fact we implement the _IDisposable_ interface allowing us to use the _using_ construct, which leads to much nicer looking user code: 
-    
-    using (ChangeRequest.CRfile file = new ChangeRequest.CRfile("FooBar.xls"))
-    {
-      System.Console.Writeline("Change request version: {0}", file.version);
-      ...
-    }
+5. Users really need to call the _Dispose_ method on the Excel wrappers (e.g. CRfile and CRstatus). In fact we implement the _IDisposable_ interface allowing us to use the _using_ construct, which leads to much nicer looking user code: 
+
+
+        using (ChangeRequest.CRfile file = new ChangeRequest.CRfile("FooBar.xls"))
+        {
+          System.Console.Writeline("Change request version: {0}", file.version);
+          ...
+        }

 ## Accessing Excel Headers and Footers

-I found that the unit tests for manipulating Excel headers and footers was taking a long time (the test took about 5 minutes). I originally thought that the problem was creating and deleting the Page Setup object. However, as many first thoughts about performance are, this was wrong. The problem is that any access to element in the page setup references the default printer. In my case the defualt printer is a network printer, so if disconnected from the network (which is the case for most of my work on ESI tools) then the access to the printer is the reason for the slow performance (). The way round this is to set the printer to the Microsoft XPS Document Writer, which is guaranteed to be local. Unfortunately we need to also determine which port the printer is on. Since the port is not necessarily the same on all computers we need to loop through possible ports - most are of the form nexx (where xx is is a numeric value), or LPTx (where x is numeric). We loop through each of the ports trying to set the active printer to "Microsoft XPS Document Writer on XYZ" (where XYZ is the port) if we succeed then we continue - if we don't succeed then we revert to using the default printer and take the performance hit. On my machine using the Microsoft XPS Document Writer results in a 10x performance gain (~30s against ~300s for the header/footer test) which is definately worthwhile. 
+I found that the unit tests for manipulating Excel headers and footers was taking a long time (the test took about 5 minutes). I originally thought that the problem was creating and deleting the Page Setup object. However, as many first thoughts about performance are, this was wrong. The problem is that any access to element in the page setup references the default printer. In my case the default printer is a network printer, so if disconnected from the network (which is the case for most of my work on ESI tools) then the access to the printer is the reason for the slow performance (). The way round this is to set the printer to the Microsoft XPS Document Writer, which is guaranteed to be local. Unfortunately we need to also determine which port the printer is on. Since the port is not necessarily the same on all computers we need to loop through possible ports - most are of the form nexx (where xx is is a numeric value), or LPTx (where x is numeric). We loop through each of the ports trying to set the active printer to "Microsoft XPS Document Writer on XYZ" (where XYZ is the port) if we succeed then we continue - if we don't succeed then we revert to using the default printer and take the performance hit. On my machine using the Microsoft XPS Document Writer results in a 10x performance gain (~30s against ~300s for the header/footer test) which is definately worthwhile. 

 ## Out of Data Change Requests

@@ -105,25 +105,12 @@

 We should catch mismatches conflicts between the Change Request Number (CRN) contained in the file name (e.g., _CR166 - Add My Frigate.xls_) and the CRN contained inside the Change Request (CR) file. 

-**Name CRN**
-**Internal CRN**
-**Comments**
-
-0 
-0 
-This is valid and indicates that the CR is new 
-
-0 
-Non-Zero 
-This may indicate that the CR Submitter created a new CR by copying from a previous one. This is OK, but the Tool Superviser ought to check the CR referenced in the CR to check that this is not a mistake. The Registrar shall report an error. 
-
-Non-zero 
-0 
-For version 1 CRs this is expected and shall not be flagged as an error. For version 2 CRs this is probably a simple error by the CR Submitter (possibly it is meant to be an update, with the Submitter merely editing her original file rather than the official one posted back to the reflector. The Tool Supervisor ought to check. The Registrar shall report an error. 
-
-Non-zero 
-Non-zero 
-If the two CRNs are the same then this is OK, however if they are not the same then this is an error, which the Registrar shall report 
+**Name CRN** | **Internal CRN** | **Comments**
+-------------|------------------|--------------
+0            | 0                | This is valid and indicates that the CR is new 
+0            | Non-Zero         | This may indicate that the CR Submitter created a new CR by copying from a previous one. This is OK, but the Tool Superviser ought to check the CR referenced in the CR to check that this is not a mistake. The Registrar shall report an error. 
+Non-zero     | 0                | For version 1 CRs this is expected and shall not be flagged as an error. For version 2 CRs this is probably a simple error by the CR Submitter (possibly it is meant to be an update, with the Submitter merely editing her original file rather than the official one posted back to the reflector. The Tool Supervisor ought to check. The Registrar shall report an error. 
+Non-zero     | Non-zero         | If the two CRNs are the same then this is OK, however if they are not the same then this is an error, which the Registrar shall report 

 ## Verification Mode of Operation

@@ -180,177 +167,33 @@

 OK, this is one that I've put a lot of thought into and it should start from a fresh DB. The issue is tracking updated CRs. Consider the ASLAV CR posted by Bernard Leclerc in October last year (CR2308). His original CR was for: 

-Add 
-1 
-1 
-39 
-5 
-1 
-ASLAV 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-1 
-ASLAV-25 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-2 
-ASLAV-PC 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-3 
-ASLAV-C 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-4 
-ASLAV-S 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-5 
-ASLAV-A 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-6 
-ASLAV-F 
-
-Add 
-1 
-1 
-39 
-5 
-1 
-7 
-ASLAV-R 
+**Type**|**Kind**|**Domain**|**Country**|**Category**|**Subcategory**|**Specific**|**Name**
+--------|--------|----------|-----------|------------|---------------|------------|--------
+Add     | 1      | 1        | 39        | 5          | 1             |            | ASLAV 
+Add     | 1      | 1        | 39        | 5          | 1             | 1          | ASLAV-25 
+Add     | 1      | 1        | 39        | 5          | 1             | 2          | ASLAV-PC 
+Add     | 1      | 1        | 39        | 5          | 1             | 3          | ASLAV-C 
+Add     | 1      | 1        | 39        | 5          | 1             | 4          | ASLAV-S 
+Add     | 1      | 1        | 39        | 5          | 1             | 5          | ASLAV-A 
+Add     | 1      | 1        | 39        | 5          | 1             | 6          | ASLAV-F 
+Add     | 1      | 1        | 39        | 5          | 1             | 7          | ASLAV-R 

 After some discussions an update was posted (CR2308A) with the following: 

-Add 
-1 
-1 
-39 
-Country 39 = Canada 
-
-Add 
-1 
-1 
-39 
-2 
-Category 2 = Armored Fighting Vehicle 
-
-Add 
-1 
-1 
-39 
-2 
-1 
-Subcategory 1 = Light Armored Vehicle (LAV) 
-
-Add 
-1 
-1 
-39 
-2 
-1 
-1 
-ASLAV-25 Reconnaissance 
-
-Add 
-1 
-1 
-39 
-2 
-1 
-2 
-ASLAV-PC Personnel Carrier 
-
-Add 
-1 
-1 
-39 
-2 
-1 
-3 
-ASLAV-S Surveillance 
-
-Add 
-1 
-1 
-39 
-3 
-Category 3 = Armored Utility Vehicle 
-
-Add 
-1 
-1 
-39 
-3 
-1 
-Subcategory 1 = Light Armored Vehicle (LAV) 
-
-Add 
-1 
-1 
-39 
-3 
-1 
-1 
-ASLAV-C Command 
-
-Add 
-1 
-1 
-39 
-3 
-1 
-2 
-ASLAV-A Ambulance 
-
-Add 
-1 
-1 
-39 
-3 
-1 
-3 
-ASLAV-F Fitters 
-
-Add 
-1 
-1 
-39 
-3 
-1 
-4 
-ASLAV-R Recovery 
+**Type**|**Kind**|**Domain**|**Country**|**Category**|**Subcategory**|**Specific**|**Name**
+--------|--------|----------|-----------|------------|---------------|------------|--------
+Add     | 1      | 1        | 39        |   |   |   | Country 39 = Canada 
+Add     | 1      | 1        | 39        | 2 |   |   | Category 2 = Armored Fighting Vehicle 
+Add     | 1      | 1        | 39        | 2 | 1 |   | Subcategory 1 = Light Armored Vehicle (LAV) 
+Add     | 1      | 1        | 39        | 2 | 1 | 1 | ASLAV-25 Reconnaissance 
+Add     | 1      | 1        | 39        | 2 | 1 | 2 | ASLAV-PC Personnel Carrier 
+Add     | 1      | 1        | 39        | 2 | 1 | 3 | ASLAV-S Surveillance 
+Add     | 1      | 1        | 39        | 3 |   |   | Category 3 = Armored Utility Vehicle 
+Add     | 1      | 1        | 39        | 3 | 1 |   | Subcategory 1 = Light Armored Vehicle (LAV) 
+Add     | 1      | 1        | 39        | 3 | 1 | 1 | ASLAV-C Command 
+Add     | 1      | 1        | 39        | 3 | 1 | 2 | ASLAV-A Ambulance 
+Add     | 1      | 1        | 39        | 3 | 1 | 3 | ASLAV-F Fitters 
+Add     | 1      | 1        | 39        | 3 | 1 | 4 | ASLAV-R Recovery 

 If we don't start from a fresh DB then the Updater needs to undo CR2308. Since these are all additions then this is not too difficult, but it does mean that it needs to process the original CR file and undo the additions, then process the new file and add the new enumerations. If the CR updates the description for an enumeration then the problem is even worse since the old description is not in the database any more, nor is it contained in the CR. To undo a changed description then the Updater has to process all previous CRs all the way back to the last baseline (i.e., approved XML database) just in case one of them added or changed the description and, if none of the previous CRs affected the enumeration or its description, then extract the description from the baselined XML database. 

@@ -430,55 +273,30 @@

 The Op Manual (SISO-REF-010.1) show examples on how categories that are in the XML file, but not contained in the CR are automatically added. For example consider the following CR: 

-**Change Type**
-**Enumeration**
-**Name**
-
-Add 
-1.3.224.5.1 
-Type 77 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|---------
+Add             | 1.3.224.5.1     | Type 77 

 Currently the UK has no category 5 entities. However, category 5 for the surface domain is defined. Therefore the toolset will automatically add the "missing" category, so that it would be as if the submitter had asked for the following: 

-**Change Type**
-**Enumeration**
-**Name**
-
-Add 
-1.3.224.5 
-Destroyer (DD) 
-
-Add 
-1.3.224.5.1 
-Type 77 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.3.224.5       | Destroyer (DD) 
+Add             | 1.3.224.5.1     | Type 77 

 The civilian aircraft modifications extended this by specifying subcategories as well, so 

-**Change Type**
-**Enumeration**
-**Name**
-
-Add 
-1.2.225.85.11 
-Cessna 172 Skyhawk 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.2.225.85.11   | Cessna 172 Skyhawk 

 The toolset will automatically add the category and subcategory as follows: 

-**Change Type**
-**Enumeration**
-**Name**
-
-Add 
-1.2.225.85 
-Civilian Fixed Wing Aircraft, Large (up to 255,000 lbs / 115,666 kg) 
-
-Add 
-1.2.225.85.11 
-Single Piston Engine 
-
-Add 
-1.2.225.85.11.1 
-Cessna 172 Skyhawk 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.2.225.85      | Civilian Fixed Wing Aircraft, Large (up to 255,000 lbs / 115,666 kg) 
+Add             | 1.2.225.85.11   | Single Piston Engine 
+Add             | 1.2.225.85.11.1 | Cessna 172 Skyhawk 

 Early versions of the toolset hard wired the look up. Ideally we would like to use information in the XML database to drive this information 

@@ -501,9 +319,9 @@

 To show how the toolset fills in the missing elements take the following example of an old British glider: 

-Add 
-1.2.224.83.1.1 
-Vickers-Slingsby T-65 Vega 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.2.224.83.1.1  | Vickers-Slingsby T-65 Vega 

 Working through the flowchart we get the following: 

@@ -522,9 +340,9 @@

 Consider the slightly modified version of the above example: 

-Add 
-1.2.224.83.3.1 
-Vickers-Slingsby T-65 Vega 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.2.224.83.3.1  | Vickers-Slingsby T-65 Vega 

 Working through the flowchart we get the following: 

@@ -542,9 +360,9 @@

 To show what happens when the user tries to add something that is already specified in a category/subcategory table consider the following: 

-Add 
-1.2.224.83.1 
-Sail Plane 
+**Change Type** | **Enumeration** | **Name**
+----------------|-----------------|----------
+Add             | 1.2.224.83.1    | Sail Plane 

 Working through the flowchart we get the following: 

@@ -563,25 +381,12 @@

 Each entity type category and subcategory definition table has an associated _Applicability_ attribute which is a string with kind, domain and, if applicable, category. To make things more readable we use the letter _X_ in place of the country code. Ranges are allowed (since, for instance, some subcategory tables are applicable to multiple catagories). So for instance we have 

-**Applicability**
-**Table UID**
-**Description**
-
-1.2.X 
-10 
-Categories for air platforms 
-
-1.2.X.81 
-275 
-Subcategories for Civilian Ultralight Aircraft, Rigid Wing 
-
-1.2.X.83 
-276 
-Subcategories for Civilian Fixed Wing Aircraft, Glider 
-
-1.2.X.84-88 
-277 
-Subcategories for Civilian Fixed Wing Aircraft Light Sport; Small; Medium; Large; and Heavy 
+**Applicability** | **Table UID** | **Description**
+------------------|---------------|----------------
+1.2.X             | 10            | Categories for air platforms 
+1.2.X.81          | 275           | Subcategories for Civilian Ultralight Aircraft, Rigid Wing 
+1.2.X.83          | 276           | Subcategories for Civilian Fixed Wing Aircraft, Glider 
+1.2.X.84-88       | 277           | Subcategories for Civilian Fixed Wing Aircraft Light Sport; Small; Medium; Large; and Heavy 

 So a search for entity _1.2.225_ will return table 10, a search for entity _1.2.224.83_ (see example 1) will return table 276, a search for _1.2.222.85_ will return table 277 (85 is in the range 84 to 88), whereas both _1.2.72.82_ and _1.2.224.83.1_ will not find any table.

DesignDecisions modified by

Fri, 27 Jun 2014 15:06:34 -0000

Design Decisions

This section records some of the design decisions made during the development of the EBV tools. Note that these are decisions made about the design of the tools, and to a lesser extent the work flow, not decisions about the requirements, which are decided by the Enumerations Working Group (EWG) of the DIS Product Support Group.

Rationale for Info tool

Manual Change Requests (CRs) often add tables into the database, each table requires the allocation of a UID - which is one greater than the largest current UID. Since the Registrar already does this calculation, and it would be error prone to do this manually, the Info tool will calculate and display the current largest UID to the user

Ensuring proper clean-up of Excel interop objects

One of the major problems of using Excel via interop is ensuring that the interop objects are cleaned up properly. The issue, and its solution, is described by Microsoft in http://support.microsoft.com/kb/317109. A good discussion of the issue can be found at http://stackoverflow.com/questions/158706/how-to-properly-clean-up-excel-interop-objects-in-c. Note that so long as the com objects are released properly it appears not be be necessary (so far) to force the garbage collection to cycle.

The basic rule is to assign all COM objects to variable and to call ReleaseComObject on all variables in reverse order of their creation. Oh, and assign the variable to nul after the release. To prevent the creation of temporary objects we need to obey the two dots rule: never use two dots with COM objects. So instead of

Workbook myWorkbook = myApp.Workbooks.Open(...);

which creates a temporary Workbooks object which can never be released, we need to use

Workbooks myBooks = myApp.Workbooks;
Workbook myWorkbook = myBooks.Open(...);

Then myBooks and myWorkbook can be safely released.

There are a number of consequences of this:

We need to store additional intermediate objects as member variables to ensure that we can release intermediate objects in the correct order
Certain simple functions need to reorganised to ensure that objects are released. For instance the obvious implementation of the NumberOfRows methods is
```
public int NumberOfRows { get { return myWorksheet.UsedRange.Rows.Count; } }
```

However this creates two temporary COM objects that cannot be released properly. The solution is to refactor this to replace the temporaries with assignments to actual variables so that we can release them properly:

    public int NumberOfRows
    {
      get
      {
        Range range = myWorksheet.UsedRange;
        Range rows = range.Rows;
        int result = rows.Count;
        while (System.Runtime.InteropServices.Marshal.ReleaseComObject(rows) != 0) { }
        rows = null;
        while (System.Runtime.InteropServices.Marshal.ReleaseComObject(range) != 0) { }
        range = null;
        return result;
      }
    }

Care needs to be taken to ensure that objects are released even in the presence of exceptions - in practise this means that methods that use interop functions need to include exception handlers to perform proper clean-up. In fact the actual implementation of NumberOfRows is:

public int NumberOfRows
{
  get
  {
    Range range = null;
    Range rows = null;
    try
    {
      range = myWorksheet.UsedRange;
      rows = range.Rows;
      return rows.Count;
    }
    finally
    {
      if (rows != null)
      {
        while (System.Runtime.InteropServices.Marshal.ReleaseComObject(rows) != 0) { }
        rows = null;
      }
      if (range != null)
      {
        while (System.Runtime.InteropServices.Marshal.ReleaseComObject(range) != 0) { }
        range = null;
      }
    }
  }
}

Users really need to call the Dispose method on the Excel wrappers (e.g. CRfile and CRstatus). In fact we implement the IDisposable interface allowing us to use the using construct, which leads to much nicer looking user code:

using (ChangeRequest.CRfile file = new ChangeRequest.CRfile("FooBar.xls"))
{
System.Console.Writeline("Change request version: {0}", file.version);
...
}

Accessing Excel Headers and Footers

I found that the unit tests for manipulating Excel headers and footers was taking a long time (the test took about 5 minutes). I originally thought that the problem was creating and deleting the Page Setup object. However, as many first thoughts about performance are, this was wrong. The problem is that any access to element in the page setup references the default printer. In my case the defualt printer is a network printer, so if disconnected from the network (which is the case for most of my work on ESI tools) then the access to the printer is the reason for the slow performance (http://social.msdn.microsoft.com/Forums/en-US/vsto/thread/8b2a0988-7d09-4abf-a4d3-76e81f708d40/). The way round this is to set the printer to the Microsoft XPS Document Writer, which is guaranteed to be local. Unfortunately we need to also determine which port the printer is on. Since the port is not necessarily the same on all computers we need to loop through possible ports - most are of the form nexx (where xx is is a numeric value), or LPTx (where x is numeric). We loop through each of the ports trying to set the active printer to "Microsoft XPS Document Writer on XYZ" (where XYZ is the port) if we succeed then we continue - if we don't succeed then we revert to using the default printer and take the performance hit. On my machine using the Microsoft XPS Document Writer results in a 10x performance gain (~30s against ~300s for the header/footer test) which is definately worthwhile.

Out of Data Change Requests

Over time the templates used for Change Requests (CRs) have evolved (although it is expected that the rate of change of templates will be slower in the future, there may still be changes in the future). Although we don't want to update every CR to the very latest template since this causes far too much traffic on the reflector for very little benefit (this was done in the early days, but is seen to be no longer worth while), there are certain minimum versions that are required. Currently we need to have support for UUIDs in the CR. CRs without UUID support need upgrading and are termed Out Of Date. CRs that use the latest template have the Uses Latest Template property. Out Of Date CRs need to be updated (automatically) to the latest templates. CRs that are not Out Of Date but do not use the latest template are not automatically upgraded, however the user can use the Upgrader tool to force an upgrade to the latest tempate.

Split between User Interface and Business Logic

In order to maximise the potential reuse of the ESI tool code the user interface has been split from the main code (often called business logic by many commentators). The main code for each of the tools is contained in a separate assembly from the tool itself. With most of the tools being command line tools, this ends up with the tool executable just consisting of a call to the Execute method of the assembly, passing the command line parameters as parameters.

Change Request Number (CRN) Checking

We should catch mismatches conflicts between the Change Request Number (CRN) contained in the file name (e.g., CR166 - Add My Frigate.xls) and the CRN contained inside the Change Request (CR) file.

Name CRN
Internal CRN
Comments

0
0
This is valid and indicates that the CR is new

0
Non-Zero
This may indicate that the CR Submitter created a new CR by copying from a previous one. This is OK, but the Tool Superviser ought to check the CR referenced in the CR to check that this is not a mistake. The Registrar shall report an error.

Non-zero
0
For version 1 CRs this is expected and shall not be flagged as an error. For version 2 CRs this is probably a simple error by the CR Submitter (possibly it is meant to be an update, with the Submitter merely editing her original file rather than the official one posted back to the reflector. The Tool Supervisor ought to check. The Registrar shall report an error.

Non-zero
Non-zero
If the two CRNs are the same then this is OK, however if they are not the same then this is an error, which the Registrar shall report

Verification Mode of Operation

During the development of the Registrar application it became apparent that an extra mode of operation would be useful, namely the verify mode. This mode would be used by the tool maintainer to check that the CR Status file was a correct reflection of the submitted CRs.

It performs the following for each CR in a specified directory:

If the CR has not got a CRN in the file name then ignore it (it is a new CR and not processed yet)
If the CR is not in the CR Status file then report an error
Check that the CRN in the file name matches the CRN inside the file, report an error if it doesn't
Check that the CR's revision letter (from the file name) matches the revision letter in the CR Status file, report an error if it doesn't
Check that the title of the CR matches the title in the CR Status file, report an error if it doesn't
Check that the submitter of the CR matches the submitter in the CR Status file, if it doesn't and the submitter name in the CR Status file is empty then update if; if it doesn't match and the name in the CR Status file is not empty then report an error

For example:

registrar -v -f "C:\Path\To\Status\File\CR Status.xls"

If the status file is in the same directory as the change requests (and has the name "CR Status.xls" then this simplifies to:

registrar -v

Upgrade Mode of Operation

During development of the Change Request utilities it became apparent that handling different versions of the templates was becoming increasingly messy. It was decided that rather than support all previous versions of the templates the Change Request code should only handle the latest version. Rather than upgrade all previous versions by hand, we should get the tool set to do it. The obvious place to do this is within the Registrar (a separate tool was considered, but the sequence of events would then be:

Run the registrar tool on the CR file, which would throw an exception
Manually kick off the upgrade tool
Manually re-run the registrar tool on the CR file

This is more work for the tool custodian, therefore the sensible thing to do would be incorporate this functionality into the Registrar. As well as performing the upgrade automatically when registering the CR, a separate mode upgrade of operation is provided.

To use this mode of operation a source file and the template file must be provided as follows:

registrar -u -s Old-CR-File.xls -t C:\Path\To\Template\Directory\Blank-EntityType-CR.xls

The template file is defaulted, so the easiest thing is to copy the template file into the current directory. The command line then simplifies to the following:

registrar -u -s Old-CR-File.xls

The upgrader function runs before the registration function but after the check for matching CRNs (the upgrader function can thus work with the preconditions that the CRN within the file matches the CRN in the file name). If the CR file contains a CRN then the revision letter is upgraded (if there is no revision letter then the the CR becomes rev A)

Updater Design Decisions

Updater Starts from a Fresh Database

Lance Marrou wrote:

3. Does each time the Updater run, it starts from a "fresh" DB? I don't think so, though I don't think that would be a bad idea either. If it doesn't, then the updater does in fact need to process withdrawn elements otherwise said element changes (add OR delete) won't actually be removed.

OK, this is one that I've put a lot of thought into and it should start from a fresh DB. The issue is tracking updated CRs. Consider the ASLAV CR posted by Bernard Leclerc in October last year (CR2308). His original CR was for:

Add
1
1
39
5
1
ASLAV

Add
1
1
39
5
1
1
ASLAV-25

Add
1
1
39
5
1
2
ASLAV-PC

Add
1
1
39
5
1
3
ASLAV-C

Add
1
1
39
5
1
4
ASLAV-S

Add
1
1
39
5
1
5
ASLAV-A

Add
1
1
39
5
1
6
ASLAV-F

Add
1
1
39
5
1
7
ASLAV-R

After some discussions an update was posted (CR2308A) with the following:

Add
1
1
39
Country 39 = Canada

Add
1
1
39
2
Category 2 = Armored Fighting Vehicle

Add
1
1
39
2
1
Subcategory 1 = Light Armored Vehicle (LAV)

Add
1
1
39
2
1
1
ASLAV-25 Reconnaissance

Add
1
1
39
2
1
2
ASLAV-PC Personnel Carrier

Add
1
1
39
2
1
3
ASLAV-S Surveillance

Add
1
1
39
3
Category 3 = Armored Utility Vehicle

Add
1
1
39
3
1
Subcategory 1 = Light Armored Vehicle (LAV)

Add
1
1
39
3
1
1
ASLAV-C Command

Add
1
1
39
3
1
2
ASLAV-A Ambulance

Add
1
1
39
3
1
3
ASLAV-F Fitters

Add
1
1
39
3
1
4
ASLAV-R Recovery

If we don't start from a fresh DB then the Updater needs to undo CR2308. Since these are all additions then this is not too difficult, but it does mean that it needs to process the original CR file and undo the additions, then process the new file and add the new enumerations. If the CR updates the description for an enumeration then the problem is even worse since the old description is not in the database any more, nor is it contained in the CR. To undo a changed description then the Updater has to process all previous CRs all the way back to the last baseline (i.e., approved XML database) just in case one of them added or changed the description and, if none of the previous CRs affected the enumeration or its description, then extract the description from the baselined XML database.

Allocation of 'X' Values

Lance Marrou wrote:

The Registrar needs access to the DB in order to auto-generate the 'x' values and send them back out. [snip]

The Registrar does not access the DB nor should it. The Updater already needs to walk the entity type hierarchy and other parts of the XML structure making it the ideal place to auto-generate the 'x' values (it is a short step from checking that a suggested value is not already allocated to determining the next unused value).

Obviously if the CR contains any 'x' values then the updater needs to modify the CR. The question does the modified CR become an updated CR or just a new version of the CR? For instance, consider the following use case:

Somebody submits a CR, called "Add FooBar.xls", and wants the tool set to automatically allocate values for one or more enumerations (i.e., 'x' values)
The Registrar picks up the CR, determines that it is a new CR and allocates it a number, renaming the file to "CR2048 - Add FooBar.xls", say.
The Updater takes the new CR file, realises that it contains 'x'values to be added and parses the XML database to find out what value should be used for the enumerations, it then:
1. Updates the XML database with enumeration and its value
2. Updates the CR and saves it

The question is what should the file name in step 3b) be? The choices are:

Option A - make this a new revision, so the file name would become "CR2048A - Add FooBar.xls"

Option B - save it with the same file name, i.e. "CR2048 - Add FooBar.xls"

With Option B then only the modified CR file ("CR2048 - Add FooBar.xls") should be sent to the reflector.

With Option A, both the assigned CR file ("CR2048 - Add FooBar.xls") and the updated CR file ("CR2048A - Add FooBar.xls") should be sent to the reflector.

If Option A is chosen the the Updater would need to send the updated CR to the reflector - adding this would not be particularly difficult, so this is not an issue.

However if Option B is chosen then I think that neither the Registrar nor the Updater should send the CR file to the reflector - both tools should merely add the CR file to a queue which another tool (either the Status Publisher or a new Publisher) would read and send the CR file back to the reflector.

In fact, after a bit more thought I think that having a separate publisher is a good idea. This allows the Updater to operate in a mode that would be useful to people who want to add exercise specific enumerations for their own internal use - just create a set of CR forms containing the exercise specific enumerations, run the Updater on them and you have your exercise specific XML database.

Option B has been selected

Dependent Change Requests

Lance Marrou wrote:

I just had a though that maybe we should consider how to handle dependent change requests. These would be follow-up change requests that require approval of a previous change request. The current example that made me think of this is CR1924 and CR1925. 1925 contains Link 16 Terminals that use the new Radio Category requested in 1924. If 1924 were to be rejected or even updated, then 1925 would be severely affected (possibly being similarly rejected depending on the CR contents). This should be something we consider for the next quarter perhaps as I can see it would require changes to the templates and the Registrar and Updater (or maybe just Updater). For now, obviously the workaround is manual

I'm not sure we need to expend too much effort on this. I think it is unlikely that we would reject a CR without someone realising that the other CR is affected. However, let's consider what would happen if this did actually occur.

Perhaps the missing information from the tool chain description is that when adding an enumeration (in particular when adding an entity type or radio type) the Updater also validates if the "parent" enumeration already exists and reports an error if it doesn't exist.

Now, if the first CR were updated to change the category then when the Updater is run then the mismatch with the second CR would cause an error to be raised - the reflector would be notified and the second CR could then be modified accordingly

If we went all the way through to rejecting the first CR, then when the release candidate was created then the mismatch would be detected. The production of the release candidate would be halted and the reflector notified. We then have the opportunity to correct the problem (reconsider the rejection of the first CR, reject the second CR as well, put both on hold, etc.)

In both situations you postulate the problem would be detected and not affect the final approved product. I think it would be worth waiting to see if there is a real problem before going to a lot of effort to stop it.

Producing the List of Change Requests in the Revision History

Since the CR status file is not guaranteed to be ordered by change request number, creating a consistent set of change request ranges for the revision history is more difficult than it might first appear. The CR status file provides a method for iterating over all CRs that have been disposed by the EWG. To enable us to turn this into a list of change request ranges/individual change request IDs we need a class that:

the user can add change request IDs one at a time
sorts the change request IDs into increasing order
can 'output' the stored IDs as a set of individual IDs/Ranges

So for instance the following input set:

1722 1701 1698 1702 1708 1700 1699 1707 1695

is ordered as follows:

1695 1698 1699 1700 1701 1702 1707 1708 1722

and will produce the following output:

1695 1698-1702 1707-1708 1722

Output is performed by a visitor class, providing functions for processing a single change request ID, e.g. Process(int ID), and processing a change request ID range, e.g. Process(int value_min, int value_max)

Add Missing Categories, etc.

The Op Manual (SISO-REF-010.1) show examples on how categories that are in the XML file, but not contained in the CR are automatically added. For example consider the following CR:

Change Type
Enumeration
Name

Add
1.3.224.5.1
Type 77

Currently the UK has no category 5 entities. However, category 5 for the surface domain is defined. Therefore the toolset will automatically add the "missing" category, so that it would be as if the submitter had asked for the following:

Change Type
Enumeration
Name

Add
1.3.224.5
Destroyer (DD)

Add
1.3.224.5.1
Type 77

The civilian aircraft modifications extended this by specifying subcategories as well, so

Change Type
Enumeration
Name

Add
1.2.225.85.11
Cessna 172 Skyhawk

The toolset will automatically add the category and subcategory as follows:

Change Type
Enumeration
Name

Add
1.2.225.85
Civilian Fixed Wing Aircraft, Large (up to 255,000 lbs / 115,666 kg)

Add
1.2.225.85.11
Single Piston Engine

Add
1.2.225.85.11.1
Cessna 172 Skyhawk

Early versions of the toolset hard wired the look up. Ideally we would like to use information in the XML database to drive this information

When adding an entity type the toolset follows the logic shown in the following figure:

The toolset checks if the entity to be added exists. If it does then an error is generated and the process terminates. However, in this case the entity doesn't exists so we proceed to the second step
The toolset checks if the entity type to be added is predefined, i.e. it appears in one of the category (or subcategory) tables. The OpMan recommends that such entity types are not included in CRs, but it is not strictly speaking forbidden. The check has two parts:
1. We check to see if the parent entity type has an associated table. If it does not then the entity cannot be predefined and we proceed to step 4.
2. Next we find if the type is defined in its parents table. If it isn't then an error is generated (categories/subcategories must be added to the appropriate table before use) and the process terminates. Otherwise we proceed to step 3
If we get here then the entity type to be added is predefined, so we check that the description matches that already in the database. If it doesn't match then an error is generated (predefined descriptions cannot be changed when adding and entity) and the process terminates. If it does match, we generate a warning and continue to the next step.
The tool set checks if the parent exists. If it does then we add the entity to its parent and exits successfully. Otherwise we procedd to the next step
The parent does not exist so we check if the parent is predefined. Again the check has two parts:
1. We check if the parent's parent (ie. the new entity's grandparent) has an associated table. If it doesn't then the parent is not predefined and an error is generated (no parent exists) and the process terminates
2. Next we check that the parent is in the grandparent's table If it doesn't then an error is generated (categories/subcategories must be added to the appropriate table before use) and the process terminates.
Finally we add the missing ancestor(s), i.e. the parent and possibly grandparent to the database then add the new entity to the now existing parent and exit successfully

Fundamental to this process is how to determine if the entity type is predefined. The method is to associated database tables (as defined by their UID) with a set of entity types. The entity type is compared with this set and if it matches then the entity type is predefined. Early implementations hard wired this decision, but it is hoped that additional data can be added to the database to help this. For instance, the "Subcategories for Air Platform Category 83" table (UID 276) has a single element of "1.2.X.83"

Worked Example 1

To show how the toolset fills in the missing elements take the following example of an old British glider:

Add
1.2.224.83.1.1
Vickers-Slingsby T-65 Vega

Working through the flowchart we get the following:

Entity 1.2.224.83.1.1 is not in the database so we proceed to step 2
We check to see if the entity is predefined:
1. Entity 1.2.224.83.1 does not have an associated table so we move to step 4
2. This step is skipped
This step is skipped
The entity 1.2.224.83.1 does not exist
We check to see if the entity's parent is predefined:
1. The grandparent 1.2.224.83 does have an associated table (UID = 276) so we proceed to the next step
2. Subcategory 1 appears in table 276 so we proceed to the next step
We add both the grandparent (1.2.224.83 - Civilian Fixed Wing Aircraft, Glider) and parent (1.2.224.83.1 - Sail Plane) are added before finally adding the new entity itself

Worked Example 2

Consider the slightly modified version of the above example:

Add
1.2.224.83.3.1
Vickers-Slingsby T-65 Vega

Working through the flowchart we get the following:

Entity 1.2.224.83.3.1 is not in the database so we proceed to step 2
We check to see if the entity is predefined:
1. Entity 1.2.224.83.3 does not have an associated table so we move to step 4
2. This step is skipped
This step is skipped
The entity 1.2.224.83.3 does not exist
We check to see if the entity's parent is predefined:
1. The grandparent 1.2.224.83 does have an associated table (UID = 276) so we proceed to the next step
2. Subcategory 3 does not appears in table 276, and we cannot modify table 276 when adding an entity, so we generate an error and terminate

Worked Example 3

To show what happens when the user tries to add something that is already specified in a category/subcategory table consider the following:

Add
1.2.224.83.1
Sail Plane

Working through the flowchart we get the following:

Entity 1.2.224.83.1 is not in the database so we proceed to step 2
We check to see if the entity is predefined:
1. Entity 1.2.224.83 does have an associated table (UID = 276) so we continue
2. Subcategory 1 is in table 276 so we proceed to step 3
The description provided in the CR matches that in table 276 so we generate a warning and proceed to the next step
The entity 1.2.224.83 does not exist
We check to see if the entity's parent is predefined:
1. The grandparent 1.2.224 does have an associated table (UID = 10) so we proceed to the next step
2. Subcategory 83 appears in table 10 so we proceed to the next step
We add the parent (1.2.224.83 - Civilian Fixed Wing Aircraft, Glider) and then add the new entity itself

How Associated Tables are Found

Each entity type category and subcategory definition table has an associated Applicability attribute which is a string with kind, domain and, if applicable, category. To make things more readable we use the letter X in place of the country code. Ranges are allowed (since, for instance, some subcategory tables are applicable to multiple catagories). So for instance we have

Applicability
Table UID
Description

1.2.X
10
Categories for air platforms

1.2.X.81
275
Subcategories for Civilian Ultralight Aircraft, Rigid Wing

1.2.X.83
276
Subcategories for Civilian Fixed Wing Aircraft, Glider

1.2.X.84-88
277
Subcategories for Civilian Fixed Wing Aircraft Light Sport; Small; Medium; Large; and Heavy

So a search for entity 1.2.225 will return table 10, a search for entity 1.2.224.83 (see example 1) will return table 276, a search for 1.2.222.85 will return table 277 (85 is in the range 84 to 88), whereas both 1.2.72.82 and 1.2.224.83.1 will not find any table.

Track Changes

Speed is not the most important thing in the ESI tool set but there is no doubt that the major operations (updating the XML database for a release and generating the Word document) do take a long time. Therefore it is worth addressing speed problems in tasks that occur frequently in these operations.

One thing that I noticed when watching the Word Generator in action was that updates slowed down when track changes were on (most obvious when updating the emitter table). Looking at the code I noticed that it was turning track changes on, writing the text into the cell, and then turning track changes off again. My first attempt to speed things up (implemented in v20 draft 0) was based upon the observation that the emitter table contains 4 columns but generally only 2 of them are populated on each row. So I changed the emitter processor to only write non empty text in the cell. Net gain: 50% increase in speed for the emitter processing.

Job done? Well no. When v20 draft 0 was released the spacing on the emitter table was strange; the first few rows were double line height, before settling down to the desired single line height.

When a row is appended to a table each cell "inherits" the cell height from the previous row. When text is written to a cell the cell height is set to be the minimum of the text (so a single word would be single line height). If no text is written to the cell it retains inherited height. The height of the row is the maximum height of all cells in that row. In the emitter table the heading row is double line height so all the cells in the first row are all doubkle line height by default. Since the first row in the emitter table has blank NATO Reporting Name and Commercial Designation then the first row becomes double line height if we do not write the empty string into these cells. Single line height does not happen until the 23rd row when the Commercial Designation cell is written to for the first time.

Microsoft Word's Broken List Numbering

AutoWord.RangeEx range = new AutoWord.RangeEx(doc.Doc.Range(doc.Doc.Content.Start, doc.Doc.Content.End));
Microsoft.Office.Interop.Word.WdListType previousListType = Microsoft.Office.Interop.Word.WdListType.wdListNoNumbering;
var listTemplate = doc.Doc.Styles["Letter List"].ListTemplate;
foreach (Microsoft.Office.Interop.Word.Paragraph paragraph in range.TheRange.Paragraphs)
{
  Microsoft.Office.Interop.Word.WdListType listType = paragraph.Range.ListFormat.ListType;
  if (Microsoft.Office.Interop.Word.WdListType.wdListSimpleNumbering == listType)
  {
    bool needsUpdate = false;
    int originalListValue = paragraph.Range.ListFormat.ListValue;

    if ((previousListType != listType) && (1 != originalListValue))
    {
      needsUpdate = true;
      //var listTemplate = paragraph.Range.ListFormat.ListTemplate;
      //paragraph.Range.ListFormat.ApplyListTemplate(listTemplate, false);
    }

    Console.WriteLine(
      "{0} {1} -> {2} {3}: {4}",
      paragraph.Range.ListFormat.ListType.ToString(),
      originalListValue,
      paragraph.Range.ListFormat.ListValue,
      needsUpdate,
      paragraph.Range.Text);
  }
  previousListType = listType;
}