Re: [Quickfix-developers] Custom Messages/Classes Tutorial

[Brian E. <azz...@ya...>]

Feel free. Make any changes you think need to be made - a lot of it was guesswork. You'll probably have to flesh out the Java/Ruby/Python stuff.

- Brian

----- Original Message ----
From: Oren Miller <or...@qu...>
To: Brian Erst <azz...@ya...>
Cc: qui...@li...
Sent: Tuesday, October 30, 2007 2:18:58 PM
Subject: Re: [Quickfix-developers] Custom Messages/Classes Tutorial


QuickFIX Documentation:
 http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

Fantastic work Brian.  I'll try to incorporate this into the standard documentation if you don't mind.

--oren

On Oct 30, 2007, at 2:04 PM, Brian Erst wrote:

QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html


 I decided to write a little tutorial on adding custom messages that act exactly like QuickFIX classes, as I haven't found anything in the mailing lists (and nothing in the documentation) that covers this subject. Oren, et al, feel free to critique, adapt and extend as needed.

Everything below assumes you are working on a Windows machine using VS2005 - most of the steps should be identical in other environments. Using these techniques will cause you to create a customized version of QuickFIX - you will need to repeat most of these steps whenever you wish to upgrade to a new version of QuickFIX.

Customizing QuickFIX with New Message Classes and Fields

Many counterparties have extended the FIX standard with custom messages and fields. If the only changes are a couple of custom fields, this tutorial is serious overkill; you are much better off simply defining the custom fields directly:
#include "quickfix/Field.h"

  namespace FIX
  {
    USER_DEFINE_STRING(MyStringField, 6123);
    USER_DEFINE_PRICE(MyPriceField, 8756);
  }Even if your counterparty has defined a new outbound message, you can simply create a Message object and set the fields manually.

If, however, your counterparty has created a whole host of new messages, inbound and outbound, you may want to build support for those messages directly into QuickFIX. In order to do this, you will need to create a custom build. This requires a couple of tools you may not have lying around, some comfort editing XML files and projects, a little understanding about the differences in how QuickFIX builds source code for different languages and a little fortitude.

New Tools

QuickFIX uses a combination of XML, XSL and Ruby scripts in order to generate the source code of messages and fields. In the Windows environment, it requires two tools that many developers will not have handy: msxsl and Ruby. Fortunately, both of these tools are easy to get and free for use.

msxsl is a Microsoft tool that takes XML, transforms it using rules defined in XSL style sheets and outputs new files. As of this writing, it can be found here: http://www.microsoft.com/downloads/details.aspx?FamilyId=2FB55371-C94E-4373-B0E9-DB4816552E41&displaylang=en. If that doesn't work, just use Google to look up "msxsl" - it's the first link returned. Place the msxsl.exe file somewhere on your PATH.

Ruby is a scripting language created by Yukihiro Matsumoto. It is similar to perl, python and other scripting languages. You can download the latest versions of Ruby from the following link: http://www.ruby-lang.org/en/downloads/  One caveat: the current version of the One-Click Installer for Ruby (1.8.6-25) has a bug in its modification of your PATH variable - it will wipe it out while adding the Ruby path. Save a copy of your PATH variable prior to install, and then restore it, making sure to keep/add the Ruby path.

Getting Prepared

Before building your custom version of QuickFIX, you should be sure to have the latest source code (currently 1.12.4). You will need the tools described above, as well as a copy of Visual Studio. Currently, VS 6.0, 2003 and 2005 are supported.

Get acquainted with the way the source code and projects are laid out. The subdirectories we're most interested in are .\quickfix (where the projects/solutions reside) and .\quickfix\spec (where the XML, XSL and code generation scripts reside).

Also, have a copy of your counterparty's documentation, and figure out which fields and/or messages you need to implement.

 Editing the XML/XSL
 
QuickFIX defines all of the FIX messages in terms of XML data dictionaries (FIX4n.XML, where 'n' is the minor version number) in the .\quickfix\spec directory. These data dictionaries are a two-fer: they not only control how QuickFIX validates messages at run-time, but they are also used to create the message classes, fields and field value constants. With a little care, you can add new messages and fields to the XML files and generate new classes using the provided generate script. If you have new fields, you may also need to edit one of the XSL files, but this is much simpler.

Two last things before we dive into editing the XML/XSL files. First, QuickFIX takes one shortcut when defining fields and message crackers - it assumes that FIX44.XML contains a superset of all messages and fields. This means that even if your counterparty uses FIX 4.2, you cannot just modify the FIX42.XML file. You must also create at least a "stub" copy of the messages and full copies of the fields in FIX44.XML. We'll see how in a minute.

Second, QuickFIX uses the "required" attribute of tags when generating constructors for message objects. If the required attribute is set to "Y", that field will become part of a constructor. Don't be tempted to set the required attribute to "Y" on the existing fields of an existing message. Even if your counterparty requires the RawData field on a Logon message, this is not the place to force that validation. If you do it here, much of the QuickFIX build will fail, as many of the acceptance tests will not be able to construct message objects that have suddenly gained a required parameter. Once you've built your custom version of QuickFIX, make a copy of your XML files to your deployment environment and modify those copies to make fields required at run-time. Perhaps a future version of QuickFIX could add another attribute that could distinguish between "required for constructors" and "required for run-time validation".

The various FIX4n.XML files are laid out in a standard way. The main node is <fix>, with the major and minor version numbers defined. The main subnodes are <header> (defining the header fields), <trailer> (defining the trailer fields), <messages> (a list message definitions supported by this version of FIX) and <fields> (a list of fields, and optional values, supported by this version of FIX). We're most interested in <messages> and <fields>, but as messages are simply groups of fields, let's look at fields first.

Adding New Fields

If your custom message(s) only use fields already defined in your version of FIX, you're in luck! You can skip this section. For the rest of us unfortunates, we will need to edit at least the XML and probably one XSL file (FieldNumbers.xsl).

If your counterparty has any sense at all (an iffy proposition considering that they created custom messages), they will confine their new fields to the proper User Defined Fields range of 5000-9999 defined in the FIX protocol. If not, and they've done something so stupid as to use tag numbers that exist in subsequent versions of the FIX specification - I'm sorry, but you've fallen out of the scope of this document. But you should be able to read on to get some idea as to what you might need to do.

For the rest of us, let's look at adding a new field. The ICE exchange has extended FIX 4.2 to add the field 9208 as CTICode, an integer field that can only contain the values 1-4, each of which has a specific meaning. To add this field to various messages, we need to define it within the <fields> node of FIX42.XML. This entails creating a new node <field> within the <fields> node. The best and easiest place to do this is at the end of the existing list. Just look for </fields> and move up a line.

We now need to create that new node. You will notice that the other fields have a very standardized format, with a couple of attributes and an optional list of values defined as subnodes. You can make a copy of an existing field that most closely matches your need and modify it, or just create a new one. It's pretty simple; here's the definition of CTICode:

    <field number="9208" name="CTICode" type="INT">
      <value enum="1" description="OWN_ACCOUNT"/>
      <value enum="2" description="HOUSE_ACCOUNT"/>
      <value enum="3" description="OTHER_BROKER_ACCOUNT"/>
      <value enum="4" description="CUSTOMER_ACCOUNT"/>
    </field>

We've just defined a new field. It has a number/value (9208), a name (CTICode) and a type (INT - integer). This field will now become a recognized, named field within QuickFIX (or, it will after we finish this process) and can be added to new and existing messages. It will also have a list of enumerated values, also named and usable throughout QuickFIX (in C++, the first value will become const int CTICode_OWN_ACCOUNT = 1 in Values.h).

Look at the existing fields to find examples of strings, currencies and amounts, as well as fields that do not have enumerated values (they're even simpler - just one line). You can also now modify existing messages to add this custom field - just be sure not to use the required="Y" attribute, as it will modify the constructors and cause the build to fail. You can set the required="Y" attribute in your deployment version of the XML if you need it at run-time.

But we're not done with this field. We have now extended the range of valid field numbers within the FIX 4.2 namespace - we've got to let the message generator know this. This is handled within the FieldNumbers.XSL stylesheet. Within that file, you will see some template text, one line of which is:

,FIX42_LastField=EncodedListStatusText,

You need to change this value to be the name of the field with the largest field number that you have added. In our case, the largest number is 9208, or CTICode, so we modify the line to be:

,FIX42_LastField=CTICode,

Save that change.

The final step in the process is that we have to add these fields to FIX44.XML. If you're lucky enough to have found a counterparty that actually uses FIX 4.4, you're done! If not, you have to add the same field definitions to the FIX44.XML file - the code generator scripts use this file only to generate fields and values. You should not, however, add the new fields to any of the messages that you modified in the other XML files unless you plan on using those fields with FIX 4.4.

Now, on to custom messages.

Adding New Messages
 
If you just read about adding fields, you'll find that adding messages is very similar. If you skipped that section, don't worry - it's pretty simple. 

Let's look at adding a new message. The ICE exchange has extended FIX 4.2 to add the message TraderLogout, defined as message type "CH". To add this message, we need to define any new fields it might use (as above), and then define it within the <messages> node of FIX42.XML. This entails creating a new node <message> within the <messages> node. The best and easiest place to do this is at the end of the existing list. Just look for </messages> and move up a line.
 
 We now need to create that new node. You will notice that the other messages have a very standardized format, with a couple of attributes and an list of fields defined as subnodes. You can make a copy of an existing message that most closely matches your need and modify it, or just create a new one. It's pretty simple; here's the definition of TraderLogout:
 
    <message name="TraderLogout" msgtype="CH" msgcat="app">
      <field name="ClientID" required="N"/>
      <field name="Username" required="Y"/>
    </message>    

 We've just defined a new message. It has a name (TraderLogout), a message type (tag 35 will be set to "CH") and a category (either "app" or "admin" - while it mostly just determines how the messages will be cracked, you will almost always be creating "app" messages). This message will now become a recognized, named class within QuickFIX (or, it will after we finish this process) with its own source code (in separate files for C++, Java and .Net, or as part of a larger file in Python and Ruby).
 
For our custom messages, feel free to use the required="Y" attribute - this is used by the code generator to add that field to the list of parameters on the class constructor. If the field is required, we might as well require it to construct the object.
 
 But we're not done with this message. The final step in the process is that we have to add these messages to FIX44.XML. If you're lucky enough to have found a counterparty that actually uses FIX 4.4, you're done! If not, you have to add a version of these messages to the FIX44.XML file - the code generator scripts use this file when defining the message types. Without modifying FIX44.XML, you will not be able to crack these new messages (and the build will fail). Unless you plan on using these messages with FIX 4.4, you do not actually have to fully define the message - there's no need to have fields in the message. In some cases, it is critical not to copy the whole message - any message that contains a field deprecated in FIX 4.4 (like ClientID) will not build properly if it exists in the 4.4 XML.
 
 Now that we've added our fields and messages, let's generate the source code.

 Generating the Source Code

This is the easiest part - assuming that everything was edited correctly and you have installed the msxsl and Ruby tools. From the command line, go to the .\quickfix\spec directory and type:

generate

This fires off a batch file that will generate the source code. Various nested batch files will be invoked, each of which use the msxsl and Ruby tools to generate code and, in the case of the Java build, modify the batch files themselves. 

The XML transforms via msxsl are done relatively quickly. These build the initial versions of the Field, FieldNumbers and Values files.

The Ruby scripts generate the bulk of the code (message and field classes) and take quite a while to complete. The scripts iterate through each of versions of FIX (4.1-4.4) and each of the languages (C++, Java, .Net, Python and Ruby). You can watch the code being updated by watching the .\quickfix\src\[language]\...\fix4n directories.

If things go badly, it's sometimes difficult to figure out why. The scripts don't do a lot of logging, so you may be in for some headscratching. Things to look out for are deprecated fields (a big problem - avoid this at all costs), fields from a subsequent version of FIX used in a prior version (example: a FIX 4.3 field in a 4.2 message - this can be handled by adding the field to the 4.2 XML) and improper formatting of the XML.

Good luck!

Building QuickFix (Visual Studio 2005)

I'd love to tell you that now that the code has been generated, that building it is a snap. It almost is! The problem is that in at least one case (.Net), the code generator has generated new source code files that have to be added to the build project. The C++ version primarily creates header files - as they are #included via other header files, they don't need to be added to a project in order for it to build, but you may want to for completeness sake.

In the case of .Net, you will need to add the generated .cs files to the project. This is pretty simple - in Visual Studio 2005, there is a quickfix_net_messages_vs8 project that contains all the version-specific code in subprojects (fix41-fix44). If you right-click one of those subprojects and "Add-->Existing Item...", you should easily be able to add the new files needed. The simplest way is to simply select all the .cs files in the appropriate subdirectory (e.g., fix42) and add them. Preexisting files will be ignored and the new files will be added.

You can now build the solution and, if everything went well, you will have a new set of .lib and .dll files that can be used as replacements for the standard QuickFIX libraries.

I have not used the Java, Ruby or Python versions, so building those are left as an exercise for the reader.

Using Your Custom Version of QuickFIX

Using the customized version of QuickFIX is exactly the same as the standard version - you just have to add .lib or .dll files to the right places and use the custom-generated header files. All other aspects of QuickFIX programming are the same.
 
I hope this tutorial has been helpful for anyone looking to use custom messages with QuickFIX. I wish I had had it when I started! Please let me know if I've missed anything or got some detail wrong - I'll need to know myself!

- Brian Erst
Thynk Software, Inc.



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers