This section records some of the design decisions made during the development of the EBV tools. Note that these are decisions made about the design of the tools, and to a lesser extent the work flow, not decisions about the requirements, which are decided by the Enumerations Working Group (EWG) of the DIS Product Support Group.
Manual Change Requests (CRs) often add tables into the database, each table requires the allocation of a UID - which is one greater than the largest current UID. Since the Registrar already does this calculation, and it would be error prone to do this manually, the Info tool will calculate and display the current largest UID to the user
One of the major problems of using Excel via interop is ensuring that the interop objects are cleaned up properly. The issue, and its solution, is described by Microsoft in http://support.microsoft.com/kb/317109. A good discussion of the issue can be found at http://stackoverflow.com/questions/158706/how-to-properly-clean-up-excel-interop-objects-in-c. Note that so long as the com objects are released properly it appears not be be necessary (so far) to force the garbage collection to cycle.
The basic rule is to assign all COM objects to variable and to call ReleaseComObject on all variables in reverse order of their creation. Oh, and assign the variable to nul after the release. To prevent the creation of temporary objects we need to obey the two dots rule: never use two dots with COM objects. So instead of
Workbook myWorkbook = myApp.Workbooks.Open(...);
which creates a temporary Workbooks object which can never be released, we need to use
Workbooks myBooks = myApp.Workbooks;
Workbook myWorkbook = myBooks.Open(...);
Then myBooks and myWorkbook can be safely released.
There are a number of consequences of this:
We need to store additional intermediate objects as member variables to ensure that we can release intermediate objects in the correct order
Certain simple functions need to reorganised to ensure that objects are released. For instance the obvious implementation of the NumberOfRows methods is
public int NumberOfRows { get { return myWorksheet.UsedRange.Rows.Count; } }
However this creates two temporary COM objects that cannot be released properly. The solution is to refactor this to replace the temporaries with assignments to actual variables so that we can release them properly:
public int NumberOfRows
{
get
{
Range range = myWorksheet.UsedRange;
Range rows = range.Rows;
int result = rows.Count;
while (System.Runtime.InteropServices.Marshal.ReleaseComObject(rows) != 0) { }
rows = null;
while (System.Runtime.InteropServices.Marshal.ReleaseComObject(range) != 0) { }
range = null;
return result;
}
}
Care needs to be taken to ensure that objects are released even in the presence of exceptions - in practise this means that methods that use interop functions need to include exception handlers to perform proper clean-up. In fact the actual implementation of NumberOfRows is:
public int NumberOfRows
{
get
{
Range range = null;
Range rows = null;
try
{
range = myWorksheet.UsedRange;
rows = range.Rows;
return rows.Count;
}
finally
{
if (rows != null)
{
while (System.Runtime.InteropServices.Marshal.ReleaseComObject(rows) != 0) { }
rows = null;
}
if (range != null)
{
while (System.Runtime.InteropServices.Marshal.ReleaseComObject(range) != 0) { }
range = null;
}
}
}
}
Users really need to call the Dispose method on the Excel wrappers (e.g. CRfile and CRstatus). In fact we implement the IDisposable interface allowing us to use the using construct, which leads to much nicer looking user code:
using (ChangeRequest.CRfile file = new ChangeRequest.CRfile("FooBar.xls"))
{
System.Console.Writeline("Change request version: {0}", file.version);
...
}
I found that the unit tests for manipulating Excel headers and footers was taking a long time (the test took about 5 minutes). I originally thought that the problem was creating and deleting the Page Setup object. However, as many first thoughts about performance are, this was wrong. The problem is that any access to element in the page setup references the default printer. In my case the default printer is a network printer, so if disconnected from the network (which is the case for most of my work on ESI tools) then the access to the printer is the reason for the slow performance (http://social.msdn.microsoft.com/Forums/en-US/vsto/thread/8b2a0988-7d09-4abf-a4d3-76e81f708d40/). The way round this is to set the printer to the Microsoft XPS Document Writer, which is guaranteed to be local. Unfortunately we need to also determine which port the printer is on. Since the port is not necessarily the same on all computers we need to loop through possible ports - most are of the form nexx (where xx is is a numeric value), or LPTx (where x is numeric). We loop through each of the ports trying to set the active printer to "Microsoft XPS Document Writer on XYZ" (where XYZ is the port) if we succeed then we continue - if we don't succeed then we revert to using the default printer and take the performance hit. On my machine using the Microsoft XPS Document Writer results in a 10x performance gain (~30s against ~300s for the header/footer test) which is definately worthwhile.
Over time the templates used for Change Requests (CRs) have evolved (although it is expected that the rate of change of templates will be slower in the future, there may still be changes in the future). Although we don't want to update every CR to the very latest template since this causes far too much traffic on the reflector for very little benefit (this was done in the early days, but is seen to be no longer worth while), there are certain minimum versions that are required. Currently we need to have support for UUIDs in the CR. CRs without UUID support need upgrading and are termed Out Of Date. CRs that use the latest template have the Uses Latest Template property. Out Of Date CRs need to be updated (automatically) to the latest templates. CRs that are not Out Of Date but do not use the latest template are not automatically upgraded, however the user can use the Upgrader tool to force an upgrade to the latest tempate.
In order to maximise the potential reuse of the ESI tool code the user interface has been split from the main code (often called business logic by many commentators). The main code for each of the tools is contained in a separate assembly from the tool itself. With most of the tools being command line tools, this ends up with the tool executable just consisting of a call to the Execute method of the assembly, passing the command line parameters as parameters.
We should catch mismatches conflicts between the Change Request Number (CRN) contained in the file name (e.g., CR166 - Add My Frigate.xls) and the CRN contained inside the Change Request (CR) file.
| Name CRN | Internal CRN | Comments |
|---|---|---|
| 0 | 0 | This is valid and indicates that the CR is new |
| 0 | Non-Zero | This may indicate that the CR Submitter created a new CR by copying from a previous one. This is OK, but the Tool Superviser ought to check the CR referenced in the CR to check that this is not a mistake. The Registrar shall report an error. |
| Non-zero | 0 | For version 1 CRs this is expected and shall not be flagged as an error. For version 2 CRs this is probably a simple error by the CR Submitter (possibly it is meant to be an update, with the Submitter merely editing her original file rather than the official one posted back to the reflector. The Tool Supervisor ought to check. The Registrar shall report an error. |
| Non-zero | Non-zero | If the two CRNs are the same then this is OK, however if they are not the same then this is an error, which the Registrar shall report |
During the development of the Registrar application it became apparent that an extra mode of operation would be useful, namely the verify mode. This mode would be used by the tool maintainer to check that the CR Status file was a correct reflection of the submitted CRs.
It performs the following for each CR in a specified directory:
For example:
registrar -v -f "C:\Path\To\Status\File\CR Status.xls"
If the status file is in the same directory as the change requests (and has the name "CR Status.xls" then this simplifies to:
registrar -v
During development of the Change Request utilities it became apparent that handling different versions of the templates was becoming increasingly messy. It was decided that rather than support all previous versions of the templates the Change Request code should only handle the latest version. Rather than upgrade all previous versions by hand, we should get the tool set to do it. The obvious place to do this is within the Registrar (a separate tool was considered, but the sequence of events would then be:
This is more work for the tool custodian, therefore the sensible thing to do would be incorporate this functionality into the Registrar. As well as performing the upgrade automatically when registering the CR, a separate mode upgrade of operation is provided.
To use this mode of operation a source file and the template file must be provided as follows:
registrar -u -s Old-CR-File.xls -t C:\Path\To\Template\Directory\Blank-EntityType-CR.xls
The template file is defaulted, so the easiest thing is to copy the template file into the current directory. The command line then simplifies to the following:
registrar -u -s Old-CR-File.xls
The upgrader function runs before the registration function but after the check for matching CRNs (the upgrader function can thus work with the preconditions that the CRN within the file matches the CRN in the file name). If the CR file contains a CRN then the revision letter is upgraded (if there is no revision letter then the the CR becomes rev A)
Lance Marrou wrote:
3. Does each time the Updater run, it starts from a "fresh" DB? I don't think so, though I don't think that would be a bad idea either. If it doesn't, then the updater does in fact need to process withdrawn elements otherwise said element changes (add OR delete) won't actually be removed.
OK, this is one that I've put a lot of thought into and it should start from a fresh DB. The issue is tracking updated CRs. Consider the ASLAV CR posted by Bernard Leclerc in October last year (CR2308). His original CR was for:
| Type | Kind | Domain | Country | Category | Subcategory | Specific | Name |
|---|---|---|---|---|---|---|---|
| Add | 1 | 1 | 39 | 5 | 1 | ASLAV | |
| Add | 1 | 1 | 39 | 5 | 1 | 1 | ASLAV-25 |
| Add | 1 | 1 | 39 | 5 | 1 | 2 | ASLAV-PC |
| Add | 1 | 1 | 39 | 5 | 1 | 3 | ASLAV-C |
| Add | 1 | 1 | 39 | 5 | 1 | 4 | ASLAV-S |
| Add | 1 | 1 | 39 | 5 | 1 | 5 | ASLAV-A |
| Add | 1 | 1 | 39 | 5 | 1 | 6 | ASLAV-F |
| Add | 1 | 1 | 39 | 5 | 1 | 7 | ASLAV-R |
After some discussions an update was posted (CR2308A) with the following:
| Type | Kind | Domain | Country | Category | Subcategory | Specific | Name |
|---|---|---|---|---|---|---|---|
| Add | 1 | 1 | 39 | Country 39 = Canada | |||
| Add | 1 | 1 | 39 | 2 | Category 2 = Armored Fighting Vehicle | ||
| Add | 1 | 1 | 39 | 2 | 1 | Subcategory 1 = Light Armored Vehicle (LAV) | |
| Add | 1 | 1 | 39 | 2 | 1 | 1 | ASLAV-25 Reconnaissance |
| Add | 1 | 1 | 39 | 2 | 1 | 2 | ASLAV-PC Personnel Carrier |
| Add | 1 | 1 | 39 | 2 | 1 | 3 | ASLAV-S Surveillance |
| Add | 1 | 1 | 39 | 3 | Category 3 = Armored Utility Vehicle | ||
| Add | 1 | 1 | 39 | 3 | 1 | Subcategory 1 = Light Armored Vehicle (LAV) | |
| Add | 1 | 1 | 39 | 3 | 1 | 1 | ASLAV-C Command |
| Add | 1 | 1 | 39 | 3 | 1 | 2 | ASLAV-A Ambulance |
| Add | 1 | 1 | 39 | 3 | 1 | 3 | ASLAV-F Fitters |
| Add | 1 | 1 | 39 | 3 | 1 | 4 | ASLAV-R Recovery |
If we don't start from a fresh DB then the Updater needs to undo CR2308. Since these are all additions then this is not too difficult, but it does mean that it needs to process the original CR file and undo the additions, then process the new file and add the new enumerations. If the CR updates the description for an enumeration then the problem is even worse since the old description is not in the database any more, nor is it contained in the CR. To undo a changed description then the Updater has to process all previous CRs all the way back to the last baseline (i.e., approved XML database) just in case one of them added or changed the description and, if none of the previous CRs affected the enumeration or its description, then extract the description from the baselined XML database.
Lance Marrou wrote:
The Registrar needs access to the DB in order to auto-generate the 'x' values and send them back out. [snip]
The Registrar does not access the DB nor should it. The Updater already needs to walk the entity type hierarchy and other parts of the XML structure making it the ideal place to auto-generate the 'x' values (it is a short step from checking that a suggested value is not already allocated to determining the next unused value).
Obviously if the CR contains any 'x' values then the updater needs to modify the CR. The question does the modified CR become an updated CR or just a new version of the CR? For instance, consider the following use case:
The question is what should the file name in step 3b) be? The choices are:
Option A - make this a new revision, so the file name would become "CR2048A - Add FooBar.xls"
Option B - save it with the same file name, i.e. "CR2048 - Add FooBar.xls"
With Option B then only the modified CR file ("CR2048 - Add FooBar.xls") should be sent to the reflector.
With Option A, both the assigned CR file ("CR2048 - Add FooBar.xls") and the updated CR file ("CR2048A - Add FooBar.xls") should be sent to the reflector.
If Option A is chosen the the Updater would need to send the updated CR to the reflector - adding this would not be particularly difficult, so this is not an issue.
However if Option B is chosen then I think that neither the Registrar nor the Updater should send the CR file to the reflector - both tools should merely add the CR file to a queue which another tool (either the Status Publisher or a new Publisher) would read and send the CR file back to the reflector.
In fact, after a bit more thought I think that having a separate publisher is a good idea. This allows the Updater to operate in a mode that would be useful to people who want to add exercise specific enumerations for their own internal use - just create a set of CR forms containing the exercise specific enumerations, run the Updater on them and you have your exercise specific XML database.
Option B has been selected
Lance Marrou wrote:
I just had a though that maybe we should consider how to handle dependent change requests. These would be follow-up change requests that require approval of a previous change request. The current example that made me think of this is CR1924 and CR1925. 1925 contains Link 16 Terminals that use the new Radio Category requested in 1924. If 1924 were to be rejected or even updated, then 1925 would be severely affected (possibly being similarly rejected depending on the CR contents). This should be something we consider for the next quarter perhaps as I can see it would require changes to the templates and the Registrar and Updater (or maybe just Updater). For now, obviously the workaround is manual
I'm not sure we need to expend too much effort on this. I think it is unlikely that we would reject a CR without someone realising that the other CR is affected. However, let's consider what would happen if this did actually occur.
Perhaps the missing information from the tool chain description is that when adding an enumeration (in particular when adding an entity type or radio type) the Updater also validates if the "parent" enumeration already exists and reports an error if it doesn't exist.
Now, if the first CR were updated to change the category then when the Updater is run then the mismatch with the second CR would cause an error to be raised - the reflector would be notified and the second CR could then be modified accordingly
If we went all the way through to rejecting the first CR, then when the release candidate was created then the mismatch would be detected. The production of the release candidate would be halted and the reflector notified. We then have the opportunity to correct the problem (reconsider the rejection of the first CR, reject the second CR as well, put both on hold, etc.)
In both situations you postulate the problem would be detected and not affect the final approved product. I think it would be worth waiting to see if there is a real problem before going to a lot of effort to stop it.
Since the CR status file is not guaranteed to be ordered by change request number, creating a consistent set of change request ranges for the revision history is more difficult than it might first appear. The CR status file provides a method for iterating over all CRs that have been disposed by the EWG. To enable us to turn this into a list of change request ranges/individual change request IDs we need a class that:
So for instance the following input set:
1722 1701 1698 1702 1708 1700 1699 1707 1695
is ordered as follows:
1695 1698 1699 1700 1701 1702 1707 1708 1722
and will produce the following output:
1695 1698-1702 1707-1708 1722
Output is performed by a visitor class, providing functions for processing a single change request ID, e.g. Process(int ID), and processing a change request ID range, e.g. Process(int value_min, int value_max)
The Op Manual (SISO-REF-010.1) show examples on how categories that are in the XML file, but not contained in the CR are automatically added. For example consider the following CR:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.3.224.5.1 | Type 77 |
Currently the UK has no category 5 entities. However, category 5 for the surface domain is defined. Therefore the toolset will automatically add the "missing" category, so that it would be as if the submitter had asked for the following:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.3.224.5 | Destroyer (DD) |
| Add | 1.3.224.5.1 | Type 77 |
The civilian aircraft modifications extended this by specifying subcategories as well, so
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.2.225.85.11 | Cessna 172 Skyhawk |
The toolset will automatically add the category and subcategory as follows:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.2.225.85 | Civilian Fixed Wing Aircraft, Large (up to 255,000 lbs / 115,666 kg) |
| Add | 1.2.225.85.11 | Single Piston Engine |
| Add | 1.2.225.85.11.1 | Cessna 172 Skyhawk |
Early versions of the toolset hard wired the look up. Ideally we would like to use information in the XML database to drive this information
When adding an entity type the toolset follows the logic shown in the following figure:
Fundamental to this process is how to determine if the entity type is predefined. The method is to associated database tables (as defined by their UID) with a set of entity types. The entity type is compared with this set and if it matches then the entity type is predefined. Early implementations hard wired this decision, but it is hoped that additional data can be added to the database to help this. For instance, the "Subcategories for Air Platform Category 83" table (UID 276) has a single element of "1.2.X.83"
To show how the toolset fills in the missing elements take the following example of an old British glider:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.2.224.83.1.1 | Vickers-Slingsby T-65 Vega |
Working through the flowchart we get the following:
Consider the slightly modified version of the above example:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.2.224.83.3.1 | Vickers-Slingsby T-65 Vega |
Working through the flowchart we get the following:
To show what happens when the user tries to add something that is already specified in a category/subcategory table consider the following:
| Change Type | Enumeration | Name |
|---|---|---|
| Add | 1.2.224.83.1 | Sail Plane |
Working through the flowchart we get the following:
Each entity type category and subcategory definition table has an associated Applicability attribute which is a string with kind, domain and, if applicable, category. To make things more readable we use the letter X in place of the country code. Ranges are allowed (since, for instance, some subcategory tables are applicable to multiple catagories). So for instance we have
| Applicability | Table UID | Description |
|---|---|---|
| 1.2.X | 10 | Categories for air platforms |
| 1.2.X.81 | 275 | Subcategories for Civilian Ultralight Aircraft, Rigid Wing |
| 1.2.X.83 | 276 | Subcategories for Civilian Fixed Wing Aircraft, Glider |
| 1.2.X.84-88 | 277 | Subcategories for Civilian Fixed Wing Aircraft Light Sport; Small; Medium; Large; and Heavy |
So a search for entity 1.2.225 will return table 10, a search for entity 1.2.224.83 (see example 1) will return table 276, a search for 1.2.222.85 will return table 277 (85 is in the range 84 to 88), whereas both 1.2.72.82 and 1.2.224.83.1 will not find any table.
Speed is not the most important thing in the ESI tool set but there is no doubt that the major operations (updating the XML database for a release and generating the Word document) do take a long time. Therefore it is worth addressing speed problems in tasks that occur frequently in these operations.
One thing that I noticed when watching the Word Generator in action was that updates slowed down when track changes were on (most obvious when updating the emitter table). Looking at the code I noticed that it was turning track changes on, writing the text into the cell, and then turning track changes off again. My first attempt to speed things up (implemented in v20 draft 0) was based upon the observation that the emitter table contains 4 columns but generally only 2 of them are populated on each row. So I changed the emitter processor to only write non empty text in the cell. Net gain: 50% increase in speed for the emitter processing.
Job done? Well no. When v20 draft 0 was released the spacing on the emitter table was strange; the first few rows were double line height, before settling down to the desired single line height.
When a row is appended to a table each cell "inherits" the cell height from the previous row. When text is written to a cell the cell height is set to be the minimum of the text (so a single word would be single line height). If no text is written to the cell it retains inherited height. The height of the row is the maximum height of all cells in that row. In the emitter table the heading row is double line height so all the cells in the first row are all doubkle line height by default. Since the first row in the emitter table has blank NATO Reporting Name and Commercial Designation then the first row becomes double line height if we do not write the empty string into these cells. Single line height does not happen until the 23rd row when the Commercial Designation cell is written to for the first time.
AutoWord.RangeEx range = new AutoWord.RangeEx(doc.Doc.Range(doc.Doc.Content.Start, doc.Doc.Content.End));
Microsoft.Office.Interop.Word.WdListType previousListType = Microsoft.Office.Interop.Word.WdListType.wdListNoNumbering;
var listTemplate = doc.Doc.Styles["Letter List"].ListTemplate;
foreach (Microsoft.Office.Interop.Word.Paragraph paragraph in range.TheRange.Paragraphs)
{
Microsoft.Office.Interop.Word.WdListType listType = paragraph.Range.ListFormat.ListType;
if (Microsoft.Office.Interop.Word.WdListType.wdListSimpleNumbering == listType)
{
bool needsUpdate = false;
int originalListValue = paragraph.Range.ListFormat.ListValue;
if ((previousListType != listType) && (1 != originalListValue))
{
needsUpdate = true;
//var listTemplate = paragraph.Range.ListFormat.ListTemplate;
//paragraph.Range.ListFormat.ApplyListTemplate(listTemplate, false);
}
Console.WriteLine(
"{0} {1} -> {2} {3}: {4}",
paragraph.Range.ListFormat.ListType.ToString(),
originalListValue,
paragraph.Range.ListFormat.ListValue,
needsUpdate,
paragraph.Range.Text);
}
previousListType = listType;
}