quickfix-developers

[Quickfix-developers] [qfj] Publishing Messages to JMS

[Brad H. <Bra...@gb...>]

Hi,=20

I'm having a look at quickfixj (quickfixj-1.0.0-beta1-src.tar.gz) at the
moment with a view to making it exchange messages with my application
via JMS.  I'm instantiating quickfixj from a web app listener using
Spring (http://www.springframework.org).  After a little jiggling I've
been able connect to the order executor sample and have messages
published to JMS as object messages.  quickfixj is certainly easier to
use for this sort of thing than trying to make jni play nicely with the
container.

I came across a few problems along the way which I was able to find a
solution to:

The DataDictionary class isn't serializable, and messages keep a
reference to this class which causes a java.io.NotSerializableException:
quickfix.DataDictionary when I try to publish to JMS. =20

There's nothing stopping the DataDictionary and dependent classes from
being made Serializable, but it may not be desirable to be sending this
on every message.  I notice Message's clone method has been overridden
to not copy the data dictionary, so this can be used to workaround the
problem.  What impact will this have when I go to actually trying to use
the message on the other side of the JMS queue?  I haven't got to the
stage of trying messages with repeating groups yet.

Many of the Serializable classes are missing serialVersionUID fields.  I
think this is only a practical problem if you're using different
compiled classes on either side of the serialization (I was initially
publishing with eclipse compiled classes and receiving with the ant
built jar and ran into some serialVersionUID mismatch problems).

I wanted to be able to pick up the data dictionaries from my classpath.
DefaultSessionFactory tries to load using FileInputStream(path) with no
easy way to plug in something else.  As a temporary workaround I have
replaced this with a Spring resource loader:

     	ResourceLoader defaultResourceLoader =3D new
DefaultResourceLoader();
      dataDictionary =3D new
DataDictionary(defaultResourceLoader.getResource(path).getInputStream())
;

I can now specify the data dictionary location as classpath:FIX42.xml to
pick it up from the classpath, FIX42.xml to read from file, or even
http://www.quickfixengine.org/FIX42.xml if I want to eat up some
bandwidth :)  Obviously this approach isn't directly suitable for
quickfixj unless you want to add Spring as a dependency.  Perhaps a
pluggable DataDictionaryFactory would be helpful?

quickfix.SocketInitiator.onPoll() throws a ClassCastException if it gets
a Session object from the event queue - it needs the same instanceof
checks as onBlock().

quickfix.netty.AbstractSocketInitiator.quickfixSessions never gets
filled in as far as I can tell, so isLoggedOn() and getSessions() don't
work.

There were also some I didn't look into further:

- If I start an initiator and send a message before the logon process is
completed the message doesn't go anywhere.  If I explicitly wait until
the session is logged on then it works fine.
- If a session is disconnected it doesn't seem to logon again.  It says
'Initiating Logon' but nothing further happens.

Sorry if I seem to be pointing out a lot of negatives - I do realise
quickfixj is beta.  Keep up the good work!
Regards,
Brad.

RE: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Steve B. <st...@te...>]

> Brad Harvey wrote:
> The DataDictionary class isn't serializable, and messages keep a
> reference to this class which causes a java.io.NotSerializableException:
> quickfix.DataDictionary when I try to publish to JMS.
> 
> There's nothing stopping the DataDictionary and dependent classes from
> being made Serializable, but it may not be desirable to be sending this
> on every message.  I notice Message's clone method has been overridden
> to not copy the data dictionary, so this can be used to workaround the
> problem.  What impact will this have when I go to actually trying to use
> the message on the other side of the JMS queue?  I haven't got to the
> stage of trying messages with repeating groups yet.

Hi Brad,

This is a bug. We have unit tests for message serialization but the
data dictionary is always null so it doesn't cause a problem. That 
the problem when not doing test-first development. :-) It appears the
data dictionary might only be used for message parsing. If so, I'm
not sure if we need to keep a reference to it or not. I'll look into
some more after modifying the unit test.

Can you add a bug report for this?

http://www.quickfixengine.org/bugtracker/
 
> Many of the Serializable classes are missing serialVersionUID fields.  I
> think this is only a practical problem if you're using different
> compiled classes on either side of the serialization (I was initially
> publishing with eclipse compiled classes and receiving with the ant
> built jar and ran into some serialVersionUID mismatch problems).

I see the Field, FieldMap, and the FieldOrderComparator inner class
are missing serialVersionUIDs. I'll add those. Were there other classes
that were causing you problems?

> I wanted to be able to pick up the data dictionaries from my classpath.
> DefaultSessionFactory tries to load using FileInputStream(path) with no
> easy way to plug in something else.  As a temporary workaround I have
> replaced this with a Spring resource loader:
> 
>      	ResourceLoader defaultResourceLoader = new
> DefaultResourceLoader();
>       dataDictionary = new
> DataDictionary(defaultResourceLoader.getResource(path).getInputStream())
> ;

The constructor that takes a string for the filename will try to load
the data dictionary from either a URL or a file path. This was, more
or less, the intent of the JNI data dictionary API. The constructor
taking the input stream was intended to be used for the purpose you
describe above and for testing. At some point I'd like to add a data
dictionary locator class that encapsulates the mechanisms 
for finding data dictionaries.
 
> I can now specify the data dictionary location as classpath:FIX42.xml to
> pick it up from the classpath, FIX42.xml to read from file, or even
> http://www.quickfixengine.org/FIX42.xml if I want to eat up some
> bandwidth :)  Obviously this approach isn't directly suitable for
> quickfixj unless you want to add Spring as a dependency.  Perhaps a
> pluggable DataDictionaryFactory would be helpful?

The string based constructor should already be able to load data
dictionaries using HTTP. Opening a stream for a java.net.URL is
the first technique the data dictionary class attempts to use to
load it's data. The FileInputStream is the backup strategy. It's
a hassle with Java, but you could actually write a "classpath:" 
URL handler and use the existing URL-based constructor. The way
you are doing it probably easier.
 
> quickfix.SocketInitiator.onPoll() throws a ClassCastException if it gets
> a Session object from the event queue - it needs the same instanceof
> checks as onBlock().
> 
> quickfix.netty.AbstractSocketInitiator.quickfixSessions never gets
> filled in as far as I can tell, so isLoggedOn() and getSessions() don't
> work.

Can you also add bug reports for these issues?
 
> There were also some I didn't look into further:
> 
> - If I start an initiator and send a message before the logon process is
> completed the message doesn't go anywhere.  If I explicitly wait until
> the session is logged on then it works fine.

Is there a resend request after the logon? What is the expected behavior
of FIX when a message is sent before a session has been associated with
a connection? What does the C++ engine do in this case?

> - If a session is disconnected it doesn't seem to logon again.  It says
> 'Initiating Logon' but nothing further happens.

This has already been fixed in the beta branch of CVS.
 
> Sorry if I seem to be pointing out a lot of negatives - I do realise
> quickfixj is beta.  Keep up the good work!

It's no problem at all. I was hoping for more of this type of feedback,
so I appreciate it. We were planning to do another release with a few
previous bug fixes and an new SleepyCat JE message store, but 
I'd like to fix these problems first.

BTW, Barry Kaplan has been doing a lot of experimentation with Spring
and QuickFIX/J (but not JMS, so far). The code is not currently in CVS
but you may want to contact him about some collaboration or code sharing.
Eventually, I'd like to add the Spring-related code to CVS assuming there
is enough interest. Barry's email address is mailto://bkaplan@memelet.com/.

Regards,

Steve

Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Caleb E. <cal...@gm...>]

On 9/12/05, Steve Bate <st...@te...> wrote:
>=20
> > Brad Harvey wrote:
> > - If I start an initiator and send a message before the logon process i=
s
> > completed the message doesn't go anywhere. If I explicitly wait until
> > the session is logged on then it works fine.
>=20
> Is there a resend request after the logon? What is the expected behavior
> of FIX when a message is sent before a session has been associated with
> a connection? What does the C++ engine do in this case?


The C++ implementation always persists messages first, thereby assigning=20
them a sequence number, and then sends them (excepting of course resends).

So in the case that the conneciton is down when a message is sent by the=20
Application, it will be logged and resent as a PossDup once the connection=
=20
is re-established. The Application's first Logon message will have a higher=
=20
sequence number (e.g. 2) which will cause the counterparty to request a=20
resend, at which point the original message 1 will be resent.

--=20
Caleb Epstein
caleb dot epstein at gmail dot com

RE: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Steve B. <st...@te...>]

Hi Caleb,

If the connection is up, but there's no logged on session would
you expect the behavior to be the same? I was expecting a resend
as well, but I need to look into it further. We should have the
same behavior as the C++ implementation. Are there any acceptance
tests that test this behavior for the C++ engine?

Thanks,

Steve

________________________________________
From: Caleb Epstein [mailto:cal...@gm...] 
Sent: Monday, September 12, 2005 10:46 AM
To: Steve Bate
Cc: qui...@li...
Subject: Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

On 9/12/05, Steve Bate <st...@te...> wrote:
> Brad Harvey wrote:
> - If I start an initiator and send a message before the logon process is
> completed the message doesn't go anywhere.  If I explicitly wait until
> the session is logged on then it works fine. 

Is there a resend request after the logon? What is the expected behavior
of FIX when a message is sent before a session has been associated with
a connection? What does the C++ engine do in this case?

The C++ implementation always persists messages first, thereby assigning
them a sequence number, and then sends them (excepting of course resends).

So in the case that the conneciton is down when a message is sent by the
Application, it will be logged and resent as a PossDup once the connection
is re-established.  The Application's first Logon message will have a higher
sequence number (e.g. 2) which will cause the counterparty to request a
resend, at which point the original message 1 will be resent.

RE: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Steve B. <st...@te...>]

Hi Caleb,

I'm looking at the C++ Session code as I see the following
fragment for non-admin messages:

m_application.toApp( message, m_sessionID );
message.toString( messageString );
if ( isLoggedOn() )
    result =3D send( messageString );
}

It looks like the message is not passed to the send() method
if the session is not logged on. The send() method is where
the message is persisted. Am I interpreting this correctly?

Steve

________________________________________
From: Caleb Epstein [mailto:cal...@gm...]=20
Sent: Monday, September 12, 2005 10:46 AM
To: Steve Bate
Cc: qui...@li...
Subject: Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

On 9/12/05, Steve Bate <st...@te...> wrote:
> Brad Harvey wrote:
> - If I start an initiator and send a message before the logon process =
is
> completed the message doesn't go anywhere.=A0=A0If I explicitly wait =
until
> the session is logged on then it works fine.=20

Is there a resend request after the logon? What is the expected behavior
of FIX when a message is sent before a session has been associated with
a connection? What does the C++ engine do in this case?

The C++ implementation always persists messages first, thereby assigning
them a sequence number, and then sends them (excepting of course =
resends).

So in the case that the conneciton is down when a message is sent by the
Application, it will be logged and resent as a PossDup once the =
connection
is re-established.=A0 The Application's first Logon message will have a =
higher
sequence number (e.g. 2) which will cause the counterparty to request a
resend, at which point the original message 1 will be resent.

--=20
Caleb Epstein
caleb dot epstein at gmail dot com=20

Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Caleb E. <cal...@gm...>]

On 9/12/05, Steve Bate <st...@te...> wrote:
>=20
> I'm looking at the C++ Session code as I see the following
> fragment for non-admin messages:
>=20
> m_application.toApp( message, m_sessionID );
> message.toString( messageString );
> if ( isLoggedOn() )
> result =3D send( messageString );
> }
>=20
> It looks like the message is not passed to the send() method
> if the session is not logged on. The send() method is where
> the message is persisted. Am I interpreting this correctly?



No, the persistence is done in this same method, a little bit below what yo=
u=20
pasted. I was actually mistaken when I said it is done before the send (it=
=20
probably should be):

if ( !num ) <-- num is !0 if we are resending
{
MsgSeqNum msgSeqNum;
header.getField( msgSeqNum );
m_state.set( msgSeqNum, messageString ); <-- persistence happens here
m_state.incrNextSenderMsgSeqNum();
}


--=20
Caleb Epstein
caleb dot epstein at gmail dot com

RE: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Steve B. <st...@te...>]

Hi Caleb,

OK, I understand. I was looking at the message
logging in the send() method rather than the state's set()
method. The Java code is implemented the same way so it should
have the same behavior.

However, there's another potential difference.

Is the Session::send() operation in C++ never expected to throw
an exception, even if the connection is down? It appears that
it's designed to return a boolean based on a low-level system
call return value. If so, then we'll need to slightly modify
the Java code to catch low-level socket-related exceptions and=20
return a boolean result value instead of propagating an exception.

Steve

________________________________________
From: Caleb Epstein [mailto:cal...@gm...]=20
Sent: Monday, September 12, 2005 12:45 PM
To: Steve Bate
Cc: qui...@li...
Subject: Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

On 9/12/05, Steve Bate <st...@te...> wrote:
I'm looking at the C++ Session code as I see the following
fragment for non-admin messages:

m_application.toApp( message, m_sessionID );
message.toString( messageString );
if ( isLoggedOn() )
=A0=A0=A0=A0result =3D send( messageString );=20
}

It looks like the message is not passed to the send() method
if the session is not logged on. The send() method is where
the message is persisted. Am I interpreting this correctly?


No, the persistence is done in this same method, a little bit below what =
you
pasted.=A0 I was actually mistaken when I said it is done before the =
send (it
probably should be):

=A0=A0=A0 if ( !num =
)=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0 <--
num is !0 if we are resending
=A0=A0=A0 {
=A0=A0=A0=A0=A0 MsgSeqNum msgSeqNum;
=A0=A0=A0=A0=A0 header.getField( msgSeqNum );
=A0=A0=A0=A0=A0 m_state.set( msgSeqNum, messageString );=A0=A0 <-- =
persistence happens
here
=A0=A0=A0=A0=A0 m_state.incrNextSenderMsgSeqNum();
=A0=A0=A0 }

--=20
Caleb Epstein
caleb dot epstein at gmail dot com=20

Re: [Quickfix-developers] [qfj] Publishing Messages to JMS

[Caleb E. <cal...@gm...>]

On 9/12/05, Steve Bate <st...@te...> wrote:
>=20
> Is the Session::send() operation in C++ never expected to throw
> an exception, even if the connection is down? It appears that


I believe that is the case. Maybe Oren could comment.

it's designed to return a boolean based on a low-level system
> call return value. If so, then we'll need to slightly modify
> the Java code to catch low-level socket-related exceptions and
> return a boolean result value instead of propagating an exception.


Anything that would prevent the persistence from happening would be a Bad=
=20
Thing.

--=20
Caleb Epstein
caleb dot epstein at gmail dot com