quickfix-developers — QuickFIX developers forum

Re: [Quickfix-developers] Cannot reconnect after ungraceful disconnect

[Oren M. <or...@qu...>]

The thread should become unregistered in the destructor of the 
ThreadedSocketConnection class.

This was implemented mostly for the sake of the acceptance tests.  It takes 
some amount of time to close a connection and free up the session.  The 
acceptance test runner brings up connections for each test.  If the 
connection from the previous test doesn't close down fast enough, then the 
test will fail because it is unable to aquire the session.  This just allows 
the connector to retry registering the session for some amount of time 
before declaring it a failure.  Since this is a multi threaded 
implementation, we cannot rely on all the events to be processed serially 
like the single threaded implementation.

If you are not able to aquire the session during an abnormal disconnect, it 
would suggest that the old connector is not being deleted (and thus no 
destructor call and no unregistering the session ).  The deletion of the 
connection class occurs in ThreadedSocketAcceptor::socketThread.  If for 
some reason pConnection->read() is never returning false, then perhaps the 
connection is not getting deleted.

--oren

----- Original Message ----- 
From: "John Debay" <joh...@ch...>
To: <qui...@li...>
Cc: "Nicholas Murdock" <nic...@ch...>
Sent: Thursday, October 27, 2005 10:32 AM
Subject: [Quickfix-developers] Cannot reconnect after ungraceful disconnect


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi,
>
> We are having issues re-establishing QuickFIX sessions after our initiator
> disconnects ungracefully. We have traced through the code, and our problem
> seems to be this loop inside ThreadedSocketConnection::setSession():
>
> // see if the session frees up within 5 seconds
> for ( int i = 1; i <= 5; ++i )
> {
>    m_pSession = Session::registerSession( sessionID );
>    if ( m_pSession ) break;
>    process_sleep( 1 );
> }
>
> Question 1: What is this loop doing? Who is going to free the session 
> within
> 5 seconds?
>
> As I step through Session::resgisterSession(), the problem seems to be 
> here:
>
> if ( isSessionRegistered( sessionID ) ) return 0;
>
> isSessionRegistered() always returns true, so the session is never
> re-established.
>
> Can anyone help me understand what is going on here? When should
> isSessionRegistered() return false? After a session has been physically
> disconnected? At some other time?
>
> Thanks in advance for any help or advice.
>
> John
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.
> Get Certified Today * Register for a JBoss Training Course
> Free Certification Exam for All Training Attendees Through End of 2005
> Visit http://www.jboss.com/services/certification for more information
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 

RE: [Quickfix-developers] Cannot reconnect after ungraceful disco nnect

[John D. <joh...@ch...>]

Version 1.10.2.
--
John 

> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...] 
> Sent: Thursday, October 27, 2005 1:12 PM
> To: John Debay; qui...@li...
> Cc: Nicholas Murdock
> Subject: Re: [Quickfix-developers] Cannot reconnect after 
> ungraceful disconnect
> 
> What version?
> 
> --oren
> 
> ----- Original Message -----
> From: "John Debay" <joh...@ch...>
> To: <qui...@li...>
> Cc: "Nicholas Murdock" <nic...@ch...>
> Sent: Thursday, October 27, 2005 10:32 AM
> Subject: [Quickfix-developers] Cannot reconnect after 
> ungraceful disconnect
> 
> 
> > QuickFIX Documentation: 
> > http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi,
> >
> > We are having issues re-establishing QuickFIX sessions 
> after our initiator
> > disconnects ungracefully. We have traced through the code, 
> and our problem
> > seems to be this loop inside ThreadedSocketConnection::setSession():
> >
> > // see if the session frees up within 5 seconds
> > for ( int i = 1; i <= 5; ++i )
> > {
> >    m_pSession = Session::registerSession( sessionID );
> >    if ( m_pSession ) break;
> >    process_sleep( 1 );
> > }
> >
> > Question 1: What is this loop doing? Who is going to free 
> the session 
> > within
> > 5 seconds?
> >
> > As I step through Session::resgisterSession(), the problem 
> seems to be 
> > here:
> >
> > if ( isSessionRegistered( sessionID ) ) return 0;
> >
> > isSessionRegistered() always returns true, so the session is never
> > re-established.
> >
> > Can anyone help me understand what is going on here? When should
> > isSessionRegistered() return false? After a session has 
> been physically
> > disconnected? At some other time?
> >
> > Thanks in advance for any help or advice.
> >
> > John
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.
> > Get Certified Today * Register for a JBoss Training Course
> > Free Certification Exam for All Training Attendees Through 
> End of 2005
> > Visit http://www.jboss.com/services/certification for more 
> information
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> > 
> 

Re: [Quickfix-developers] Cannot reconnect after ungraceful disconnect

[Oren M. <or...@qu...>]

What version?

--oren

----- Original Message ----- 
From: "John Debay" <joh...@ch...>
To: <qui...@li...>
Cc: "Nicholas Murdock" <nic...@ch...>
Sent: Thursday, October 27, 2005 10:32 AM
Subject: [Quickfix-developers] Cannot reconnect after ungraceful disconnect


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi,
>
> We are having issues re-establishing QuickFIX sessions after our initiator
> disconnects ungracefully. We have traced through the code, and our problem
> seems to be this loop inside ThreadedSocketConnection::setSession():
>
> // see if the session frees up within 5 seconds
> for ( int i = 1; i <= 5; ++i )
> {
>    m_pSession = Session::registerSession( sessionID );
>    if ( m_pSession ) break;
>    process_sleep( 1 );
> }
>
> Question 1: What is this loop doing? Who is going to free the session 
> within
> 5 seconds?
>
> As I step through Session::resgisterSession(), the problem seems to be 
> here:
>
> if ( isSessionRegistered( sessionID ) ) return 0;
>
> isSessionRegistered() always returns true, so the session is never
> re-established.
>
> Can anyone help me understand what is going on here? When should
> isSessionRegistered() return false? After a session has been physically
> disconnected? At some other time?
>
> Thanks in advance for any help or advice.
>
> John
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.
> Get Certified Today * Register for a JBoss Training Course
> Free Certification Exam for All Training Attendees Through End of 2005
> Visit http://www.jboss.com/services/certification for more information
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 

[Quickfix-developers] Cannot reconnect after ungraceful disconnect

[John D. <joh...@ch...>]

Hi,

We are having issues re-establishing QuickFIX sessions after our initiator
disconnects ungracefully. We have traced through the code, and our problem
seems to be this loop inside ThreadedSocketConnection::setSession():

// see if the session frees up within 5 seconds
for ( int i = 1; i <= 5; ++i )
{
    m_pSession = Session::registerSession( sessionID );
    if ( m_pSession ) break;
    process_sleep( 1 );
}

Question 1: What is this loop doing? Who is going to free the session within
5 seconds?

As I step through Session::resgisterSession(), the problem seems to be here:

if ( isSessionRegistered( sessionID ) ) return 0;

isSessionRegistered() always returns true, so the session is never
re-established.

Can anyone help me understand what is going on here? When should
isSessionRegistered() return false? After a session has been physically
disconnected? At some other time?

Thanks in advance for any help or advice.

John

Re: [Quickfix-developers] username / password in logon msg?

[Scott H. <sco...@fo...>]

Oh, I had put them into the .cfg file such as in the snippet below:

[SESSION]
TargetCompID=...
Username=the_username
Password=the_password
...

As of 1.9.4, quickfix would let you put anything you want into that 
SessionSettings; you'll have to check the code to make sure that's still 
the case.


On Thu, 27 Oct 2005, Alvin Wang wrote:

> Hi Scott,
>
> Just wonder how you put username/password into sessionId before the toAdmin callback?
> Thanks
> Alvin
>
> String username = sessionSettings.getString(sessionId, "Username");
>
>
>
>
>
> Scott Harrington <sco...@fo...>
> Sent by: qui...@li...
> 10/04/2005 06:08 PM
>
>
>        To:     qui...@li...
>        cc:
>        bcc:
>        Subject:        Re: [Quickfix-developers] username / password in logon msg?
>
>
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Alvin:
>
> You can stuff the special fields into the outgoing Logon message by
> implementing the toAdmin callback.  In Java I do something like this for
> counterparties that use this type of authentication:
>
>     if (message instanceof quickfix.fix43.Logon) {
>       quickfix.fix43.Logon logon = (quickfix.fix43.Logon) message;
>       try {
>         String username = sessionSettings.getString(sessionId, "Username");
>         logon.set(new quickfix.field.Username(username));
>         String password = sessionSettings.getString(sessionId,
> "Password");
>         logon.set(new quickfix.field.Password(password));
>       }
>       catch (Exception xx) {
>         throw new RuntimeException("Username and Password must be
> specified in QuickFIX configuration");
>       }
>     }
>
>
> On Tue, 4 Oct 2005, Alvin Wang wrote:
>
>> Hi,
>>
>> How to include username (553) and password (554) in Logon message? Is
>> there a way to do it in the configuration? That should be most intuitive
>>
>> Thanks
>> Alvin

Re: [Quickfix-developers] username / password in logon msg?

[Alvin W. <AW...@FF...>]

Hi Scott,

Just wonder how you put username/password into sessionId before the toAdmin callback?
Thanks
Alvin

String username = sessionSettings.getString(sessionId, "Username");





Scott Harrington <sco...@fo...>
Sent by: qui...@li...
10/04/2005 06:08 PM

 
        To:     qui...@li...
        cc: 
        bcc: 
        Subject:        Re: [Quickfix-developers] username / password in logon msg?


QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

Alvin:

You can stuff the special fields into the outgoing Logon message by 
implementing the toAdmin callback.  In Java I do something like this for 
counterparties that use this type of authentication:

     if (message instanceof quickfix.fix43.Logon) {
       quickfix.fix43.Logon logon = (quickfix.fix43.Logon) message;
       try {
         String username = sessionSettings.getString(sessionId, "Username");
         logon.set(new quickfix.field.Username(username));
         String password = sessionSettings.getString(sessionId, 
"Password");
         logon.set(new quickfix.field.Password(password));
       }
       catch (Exception xx) {
         throw new RuntimeException("Username and Password must be 
specified in QuickFIX configuration");
       }
     }


On Tue, 4 Oct 2005, Alvin Wang wrote:

> Hi,
>
> How to include username (553) and password (554) in Logon message? Is
> there a way to do it in the configuration? That should be most intuitive
>
> Thanks
> Alvin


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers



                                                                                                                                                                                                                                                                                                    **********************************************************************
This e-mail message is intended solely for the use of the addressee.
The message may contain information that is privileged and confidential.  
Disclosure to anyone other than the intended recipient is
prohibited. If you are not the intended recipient, please do not
disseminate, distribute or copy this communication, by e-mail or
otherwise. Instead, please notify us immediately by return e-mail
(including the original message with your reply) and then delete
and discard all copies of the message. We have taken precautions to
minimize the risk of transmitting software viruses but nevertheless
advise you to carry out your own virus checks on any attachment to
this message.  We accept no liability for any loss or damage caused
by software viruses.
**********************************************************************

[Quickfix-developers] How to implement the QF/J failover sulotion.

[Lin L. <le...@gm...>]

Hi,
  I try to deploy qf/j as fix engine in my project.I need double=20
servers redundantly for failover.I have seen the topic about load
balance and qf not support now.
  The two qf/j instances share the same store data(same jdbc data
source).The client connected to one of them throgh a TCP
switch,e.g.F5.And another instance is in idle state.The fix messages
and SeqNum could have the right value when the client connection
reconnect switching to the another qf/j instance.This is the process
of qf/j failover in my project plan.
  The qf/j default JdbcStore/MysqlStore cache the session data in
memory but not direct read from db.So the qf/j instances initialize
the state and cache the state data from db after start up.The SeqNum
of qf/j instance  increase in service,but the another one cache the
old SeqNum in memory because not in service.
  So,I implement my JdbcSybaseStore to remove the cache feature.And
then the qf/j failover start to work OK.Of cause,I lost little
perfomance in read/write db.
  I want to deploy this qf/j failover solution in my real project.Is
it OK?Can someone have a better solution for failover?

--
Lin Lejiang



quote:
Session management in load balanced situation?
http://sourceforge.net/mailarchive/message.php?msg_id=3D11104323
 On Tue, 8 Mar 2005 12:57:38 -0600, David Kuik <david.kuik@pr...> wrote:

 > I"m wondering how the engine manages sessions and message sequencing in =
a
 > load balanced environment.  Does each server in the farm to manage a
 > separate session or do they somehow share a session?
 >
 > I"m trying to understand how this will work in a fault-tolerant 7x24
 > environment that supports 1000"s of users.  Can someone offer some comme=
nts?

 You could load balance sessions across multiple machines using a
 database-backed message store (e.g. MysqlStoreFactory).  All of the
 session state is persisted to the database.  You could run a number of
 Acceptors in parallel on different machines, each configured to accept
 all of your MySQL-backed sessions (e.g. same config file).

 However I don"t think there is currently a facility in QF for
 "locking" a session across multiple engines.  In other words, if
 client A connects to server 1, that session should be "locked" in some
 way so any attempt to use it on servers 2 through n would result in an
 error in the same way that connecting to that session a second time on
 server 1 would.

 A similar approach would be to use the FileStoreFactory and share all
 the session state on an NFS mounted filesystem.

[Quickfix-developers] Any monitor/administrator utility ie "telnet" the fix application and send cmds

[George G. <gge...@ho...>]

Are there any utilities that attatch to an app like a telnet to monitor and 
administrate the quickfix application?
Some examples will be to reset the incoming or outgoing seqnums, stop or 
start a session
display the FixVersion, Shutdown the Fix Engine and exit, Replay/Resend 
session start-end seqnum subid.


Thanks

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

RE: [Quickfix-developers] Help on QUICKFIX/J

[Shepheard, T. (London) <Tob...@ml...>]

If you're using Quickfix/J as the subject line suggests, then it is a
pure java implementation and you don't need any non-java libraries.
Unless you need your own build for some reason, then you can just
download the .bin version and use the provided quickfixj.jar file along
with the other libraries in the lib folder. Make sure you include these
in your classpath when building your own application.
=20
If you need to build it yourself, then once you've extracted the
Quickfix/J code from the src zip you can find instructions under
quickfixj\doc\usermanual\installation.html#build. It assumes you have
Java 1.4.x or higher installed, and you'll also want ant
(http://ant.apache.org) for running the build.xml script.
=20
If you're using quickfix 1.10.2 then the java version is only a wrapper
around the base implementation - you need to build the java and build
the base too. I'm not familiar with it but there are some build.sh that
look like they'll do the job. As with QuickFIX/J, you'll want ant
installed I think. I would guess that libquickfix_jni.so is the output
of one of the earlier build scripts and then used by later scripts to do
the jni bindings.
=20
If you have further problems then posting what version you're using,
your environment and a list of what you've done and what errors you're
getting would really help.
=20
=20
=20
-----Original Message-----
From: qui...@li...
[mailto:qui...@li...] On Behalf Of
amit sharma
Sent: 26 October 2005 06:37
To: qui...@li...
Subject: [Quickfix-developers] Help on QUICKFIX/J



	Hello everyone,
	                    If anyone can please tell me that if
'libquickfix_jni.so' is necessary for quickfix-java implementation or
there is some work around.
	                    Please let me know as I m in urgent need.
	Thanks in advance
	Amit
--------------------------------------------------------

If you are not an intended recipient of this e-mail, please notify the =
sender, delete it and do not read, act upon, print, disclose, copy, =
retain or redistribute it. Click here for important additional terms =
relating to this e-mail.     http://www.ml.com/email_terms/
--------------------------------------------------------

[Quickfix-developers] (no subject)

[amit s. <aam...@gm...>]

Hello everyone,
If anyone can please tell me that if 'libquickfix_jni.so' is necessary for
quickfix-java implementation or there is some work around.
Please let me know as I m in urgent need.
Thanks in advance
Amit <qui...@li...>

[Quickfix-developers] Help on QUICKFIX/J

[amit s. <aam...@gm...>]

Hello everyone,
 If anyone can please tell me that if 'libquickfix_jni.so' is necessary for
quickfix-java implementation or there is some work around.
 Please let me know as I m in urgent need.
Thanks in advance
Amit

[Quickfix-developers] HELP needed on quickfix/j

[amit s. <aam...@gm...>]

Hello everyone,
 If anyone can help me with building and installing quickfix/j.
Thanks in advance
Amit

RE: [Quickfix-developers] Test Request Message

[Shankar K. <skr...@jw...>]

 

Apologies, will not do that again.

 

Thanks, understood that the FIX engine will do that when needed, but I have
been asked by the third party to ensure

That we are in fact handling the test request and the resent request
messages correctly, hence I would like to simulate

The scenario.

 

I think I can do this by tweaking the heartbeat interval.

 

Thanks for your time.

 

-

S

  _____  

From: Caleb Epstein [mailto:cal...@gm...] 
Sent: Thursday, October 20, 2005 3:11 PM
To: Shankar Krishnan
Cc: qui...@li...
Subject: Re: [Quickfix-developers] Test Request Message

 

On 10/20/05, Shankar Krishnan <skr...@jw... <mailto:skr...@jw...>
> wrote:

I would like to send a test request message, I have a handle to the
application class, how should I send a test request message,
I read the fix doc, apparently I need to set tag 112, how do I construct the

 

First off, please don't use "Reply" if you're starting a new thread.  And
when you do reply, please don't quote unnecessarily.

Why do you want to send a Test request?  QuickFIX will do this for you
if/when it is necessary?

If you do insist on sending one, create an e.g. FIX42::TestRequest object
(include the appropriate <quickfix/fix42/TestRequest.h> header file) and use
Session::send or Session::sendToTarget to send it.


-- 
Caleb Epstein
caleb dot epstein at gmail dot com 

Re: [Quickfix-developers] Test Request Message

[Caleb E. <cal...@gm...>]

On 10/20/05, Shankar Krishnan <skr...@jw...> wrote:
>
> I would like to send a test request message, I have a handle to the
> application class, how should I send a test request message,
> I read the fix doc, apparently I need to set tag 112, how do I construct
> the


First off, please don't use "Reply" if you're starting a new thread. And
when you do reply, please don't quote unnecessarily.

Why do you want to send a Test request? QuickFIX will do this for you
if/when it is necessary?

If you do insist on sending one, create an e.g. FIX42::TestRequest object
(include the appropriate <quickfix/fix42/TestRequest.h> header file) and us=
e
Session::send or Session::sendToTarget to send it.

--
Caleb Epstein
caleb dot epstein at gmail dot com

[Quickfix-developers] Test Request Message

[Shankar K. <skr...@jw...>]

Hi,

I would like to send a test request message, I have a handle to the
application class, how should I send a test request message,
I read the fix doc, apparently I need to set tag 112, how do I construct the
message.

Thanks

-----Original Message-----
From: Shankar Krishnan 
Sent: Thursday, October 20, 2005 11:52 AM
To: Shankar Krishnan
Cc: qui...@li...
Subject: RE: [Quickfix-developers] Where are the sequence numbers stored ? I
would like to reset the m


Thanks,

I added entries to the config file to reset sequence#,

Now I have realized you never really should be playing with seq num resets
though :-)


-----Original Message-----
From: Brian Erst [mailto:azz...@ya...] 
Sent: Thursday, October 20, 2005 11:15 AM
To: Shankar Krishnan
Cc: qui...@li...
Subject: Re: [Quickfix-developers] Where are the sequence numbers stored ? I
would like to reset the m

They are stored in the .seqnums file generated for each session.

FIXx.y-[SenderCompID]-[TargetCompID].seqnums

The file contains a single line:

0000000016 : 0000000022

The first number is the outbound sequence number, the second is the
inbound sequence number.

You can also modify a session's sequence numbers programmatically via
the setNext[Sender/Target]MsgSeqNum method on the Session object.
Obviously, if you are logged in, changing the sequence numbers can have
bad effects.

- Brian Erst
Thynk Software, Inc.

--- Shankar Krishnan <skr...@jw...> wrote:

>  
> 
> Is it picking up the sequence numbers from a file ?
> 
>  
> 
> Thanks
> 
>  
> 
>  
> 
>   _____  
> 
> From: Caleb Epstein [mailto:cal...@gm...] 
> Sent: Wednesday, October 19, 2005 3:13 PM
> To: Oren Miller
> Cc: qui...@li...
> Subject: Re: [Quickfix-developers] ThreadedSocketConnection: fix for
> memory
> growth under heavy load
> 
>  
> 
> On 10/19/05, Caleb Epstein <cal...@gm...
> <mailto:cal...@gm...> > wrote:
> 
> Attached is a complete patch which removes the extra thread from
> ThreadedSocketConnection as well as the functions and members which
> are no
> longer needed once its gone.
> 
> 
> Here's a slightly revised patch (one line change).  I needed to move
> the
> call to processStream before the recv call otherwise Initiators would
> not
> logon correctly.  I've now tested this code with both Initiators and
> Acceptors.
> 
> 
> 
> -- 
> Caleb Epstein
> caleb dot epstein at gmail dot com 
> 
> 

[Quickfix-developers] ThreadedSocketInitiator session time problem

[H. S. <st...@un...>]

hi oren, guys ...

i still have some issues with ThreadedSocketInitiator trying to connect
counterparts out of session times. i use current cvs version and oren
worked on that issue and patched ThreadedSocketInitiator.cpp (1.20).

still for me it seems that this patch only helps if the threads are
spawned out of session times. as soon as doConnect() is called and a
socketThread is spawned, it keeps running until the initiator is being
stopped. that keeps the thread trying to connect even if it is out of
session times for that session.
i am not sure if the right way would be to leave the thread or to send
it to sleep as i am not that much into the qf code to see where thread
cleanup and stuff is done. either way i guess that
ThreadedSocketInitiator::socketThread should be left when out of session
time to prevent if from keep it connecting.

cheers,
heri






-- 
This e-mail was scanned with a private,
non-commercial version of AntiVir MailGate.
See http://www.antivir.de for details.

RE: [Quickfix-developers] Where are the sequence numbers stored ? I would like to reset the m

[Shankar K. <skr...@jw...>]

Thanks,

I added entries to the config file to reset sequence#,

Now I have realized you never really should be playing with seq num resets
though :-)


-----Original Message-----
From: Brian Erst [mailto:azz...@ya...] 
Sent: Thursday, October 20, 2005 11:15 AM
To: Shankar Krishnan
Cc: qui...@li...
Subject: Re: [Quickfix-developers] Where are the sequence numbers stored ? I
would like to reset the m

They are stored in the .seqnums file generated for each session.

FIXx.y-[SenderCompID]-[TargetCompID].seqnums

The file contains a single line:

0000000016 : 0000000022

The first number is the outbound sequence number, the second is the
inbound sequence number.

You can also modify a session's sequence numbers programmatically via
the setNext[Sender/Target]MsgSeqNum method on the Session object.
Obviously, if you are logged in, changing the sequence numbers can have
bad effects.

- Brian Erst
Thynk Software, Inc.

--- Shankar Krishnan <skr...@jw...> wrote:

>  
> 
> Is it picking up the sequence numbers from a file ?
> 
>  
> 
> Thanks
> 
>  
> 
>  
> 
>   _____  
> 
> From: Caleb Epstein [mailto:cal...@gm...] 
> Sent: Wednesday, October 19, 2005 3:13 PM
> To: Oren Miller
> Cc: qui...@li...
> Subject: Re: [Quickfix-developers] ThreadedSocketConnection: fix for
> memory
> growth under heavy load
> 
>  
> 
> On 10/19/05, Caleb Epstein <cal...@gm...
> <mailto:cal...@gm...> > wrote:
> 
> Attached is a complete patch which removes the extra thread from
> ThreadedSocketConnection as well as the functions and members which
> are no
> longer needed once its gone.
> 
> 
> Here's a slightly revised patch (one line change).  I needed to move
> the
> call to processStream before the recv call otherwise Initiators would
> not
> logon correctly.  I've now tested this code with both Initiators and
> Acceptors.
> 
> 
> 
> -- 
> Caleb Epstein
> caleb dot epstein at gmail dot com 
> 
> 

Re: [Quickfix-developers] Where are the sequence numbers stored ? I would like to reset the m

[Brian E. <azz...@ya...>]

They are stored in the .seqnums file generated for each session.

FIXx.y-[SenderCompID]-[TargetCompID].seqnums

The file contains a single line:

0000000016 : 0000000022

The first number is the outbound sequence number, the second is the
inbound sequence number.

You can also modify a session's sequence numbers programmatically via
the setNext[Sender/Target]MsgSeqNum method on the Session object.
Obviously, if you are logged in, changing the sequence numbers can have
bad effects.

- Brian Erst
Thynk Software, Inc.

--- Shankar Krishnan <skr...@jw...> wrote:

>  
> 
> Is it picking up the sequence numbers from a file ?
> 
>  
> 
> Thanks
> 
>  
> 
>  
> 
>   _____  
> 
> From: Caleb Epstein [mailto:cal...@gm...] 
> Sent: Wednesday, October 19, 2005 3:13 PM
> To: Oren Miller
> Cc: qui...@li...
> Subject: Re: [Quickfix-developers] ThreadedSocketConnection: fix for
> memory
> growth under heavy load
> 
>  
> 
> On 10/19/05, Caleb Epstein <cal...@gm...
> <mailto:cal...@gm...> > wrote:
> 
> Attached is a complete patch which removes the extra thread from
> ThreadedSocketConnection as well as the functions and members which
> are no
> longer needed once its gone.
> 
> 
> Here's a slightly revised patch (one line change).  I needed to move
> the
> call to processStream before the recv call otherwise Initiators would
> not
> logon correctly.  I've now tested this code with both Initiators and
> Acceptors.
> 
> 
> 
> -- 
> Caleb Epstein
> caleb dot epstein at gmail dot com 
> 
> 

[Quickfix-developers] Where are the sequence numbers stored ? I would like to reset the m

[Shankar K. <skr...@jw...>]

 

Is it picking up the sequence numbers from a file ?

 

Thanks

 

 

  _____  

From: Caleb Epstein [mailto:cal...@gm...] 
Sent: Wednesday, October 19, 2005 3:13 PM
To: Oren Miller
Cc: qui...@li...
Subject: Re: [Quickfix-developers] ThreadedSocketConnection: fix for memory
growth under heavy load

 

On 10/19/05, Caleb Epstein <cal...@gm...
<mailto:cal...@gm...> > wrote:

Attached is a complete patch which removes the extra thread from
ThreadedSocketConnection as well as the functions and members which are no
longer needed once its gone.


Here's a slightly revised patch (one line change).  I needed to move the
call to processStream before the recv call otherwise Initiators would not
logon correctly.  I've now tested this code with both Initiators and
Acceptors.



-- 
Caleb Epstein
caleb dot epstein at gmail dot com 

Re: [Quickfix-developers] Defaults session settings ?

[Oren M. <or...@qu...>]

We are planning on adding support for default field values.  Currently you 
need to do this yourself, but hopefully by the next release it will be 
included as a feature.

--oren

----- Original Message ----- 
From: "Dale Wilson" <wil...@oc...>
To: "Martin Tanguay" <mta...@ho...>
Cc: <qui...@li...>
Sent: Wednesday, October 19, 2005 10:32 AM
Subject: Re: [Quickfix-developers] Defaults session settings ?


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Martin,
>
> Martin Tanguay wrote:
>
>> In my initiator file I have this session detailed with all its needed 
>> tag.
>> [SESSION]
>> HeartBtInt=30
>> BeginString=FIX.4.2
>> SenderSubID=abcd
>> SenderCompID=efgh
>> TargetCompID=ijkl
>> RawData=abcd
>> SocketConnectHost=x.x.x.x
>> SocketConnectPort=1000
>>
>> When my application try to connect to the host, the TAGs SenderSubID and 
>> RawData are not in the header of the logon message, but my host require 
>> them to allow the connection.
>>
>> How to make them part of the logon message?
>>
> RawData and SenderSubID are not standard QuickFIX settings entries.  See 
> http://www.quickfixengine.org/quickfix/doc/html/configuration.html
>
> To add these fields to a Logon message you must override toAdmin in your 
> FIX::Application class, test for Login message there, and add the fields 
> if necessary.
>
> It *is* possible to ask for the settings associated with the session and 
> to pull RawData and SenderSubID from these settings so you can configure 
> the values in the settings file if you want to, but you'll have to provide 
> the code to honor these values.
>
> Dale
>
> -- 
> -----------------------------------------------------
>  Dale Wilson, Senior Software Engineer      Object Computing, Inc. (OCI)
>  http://www.ociweb.com/  http://www.theaceorb.com/
> ----------------------------------------------------
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 

Re: [Quickfix-developers] Defaults session settings ?

[Dale W. <wil...@oc...>]

Hi Martin,

Martin Tanguay wrote:

> In my initiator file I have this session detailed with all its needed 
> tag.
> [SESSION]
> HeartBtInt=30
> BeginString=FIX.4.2
> SenderSubID=abcd
> SenderCompID=efgh
> TargetCompID=ijkl
> RawData=abcd
> SocketConnectHost=x.x.x.x
> SocketConnectPort=1000
>
> When my application try to connect to the host, the TAGs SenderSubID 
> and RawData are not in the header of the logon message, but my host 
> require them to allow the connection.
>
> How to make them part of the logon message?
>
RawData and SenderSubID are not standard QuickFIX settings entries.  See 
http://www.quickfixengine.org/quickfix/doc/html/configuration.html

To add these fields to a Logon message you must override toAdmin in your 
FIX::Application class, test for Login message there, and add the fields 
if necessary.

It *is* possible to ask for the settings associated with the session and 
to pull RawData and SenderSubID from these settings so you can configure 
the values in the settings file if you want to, but you'll have to 
provide the code to honor these values.

Dale

-- 
-----------------------------------------------------
  Dale Wilson, Senior Software Engineer      
  Object Computing, Inc. (OCI)
  http://www.ociweb.com/  http://www.theaceorb.com/
----------------------------------------------------

Re: [Quickfix-developers] ThreadedSocketConnection: fix for memory growth under heavy load

[Oren M. <or...@qu...>]

Well, the reason for the second thread is because during testing the =
socket would appear to get overloaded and the operating system would =
break the connection.  This would happen during any extended period =
where the traffic from the sender exceeded the receiver.  The whole =
thing would then pretty much collapse because resend requests would =
ensure equilibrium was never reached.

I don't know why the sender wasn't being blocked.  It seemed to me that =
data not picked up by recv was being placed in a buffer, which would at =
some point got overloaded and cause the socket to break.  Can you verify =
that during extended periods of load that this implementation remains =
stable and will simply throttle the sender?  We would need to verify =
this on both unix and windows I think.

--oren
  ----- Original Message -----=20
  From: Caleb Epstein=20
  To: qui...@li...=20
  Sent: Tuesday, October 18, 2005 5:23 PM
  Subject: [Quickfix-developers] ThreadedSocketConnection: fix for =
memory growth under heavy load


  I have spent the past couple of days torture-testing a QuickFIX C++ =
application with some simple test harnesses that bombard it with =
messages.  In the processes of this testing, I noticed that my =
application was using a ridiculous amount of memory, many times larger =
than would be expected by the size of the test data.  So I ran =
everything through valgrind (not much help - no leaks detected), and =
eventually came upon Google's excellent "tcmalloc" library (see =
http://goog-perftools.sourceforge.net/), which can be used to dump =
statistics about heap usage.  It turned out that 95% of the memory was =
taken up by the memory allocations done by =
ThreadedSocketConnection::read!

  The overall design of the ThreadedSocketConnection class is relatively =
simple, and on paper looks like it should work correctly.  There are two =
threads for each Session: one which reads from a socket into a new'd =
char buffer and then pushes the buffer + size onto a queue; the other =
thread pops elements off of this queue, adds this data to the Parser's =
buffer, calls the Parser class to process its buffer, and finally =
delete[]'s the buffer it first got from the queue.  The code appears to =
be correct, and doesn't leak memory in the purest sense, but the =
real-world behavior when incoming traffic is high is not good.

  When a counterparty sends data faster than your application can =
process it, the "read" thread in the ThreadedSocketConnection will do a =
fine job keeping up with the socket I/O, but the queue in between these =
two threads will end up growing very large, containing all of the data =
that has been read from the socket but which has yet to be processed by =
the Parser and Session.  In a perfect world, this memory usage would be =
identical to the size of the messages that have been recv'd, but memory =
allocators are inefficient and what ends up happening is that the =
application ends up using perhaps 10-20x more memory than this!  My =
application, which should have used perhaps 125 MB of memory (most of it =
due to a mmap'ed transaction log file) had ballooned up to 775 MB!  As a =
comparison, the FIX incoming log file is only 38 MB in size.

  Thankfully, there is a simple fix to this problem, and it actually =
simplifies the code and eliminates one thread per session.  Here's my =
version of ThreadedSocketConnection::read:

  ---8<---
  bool ThreadedSocketConnection::read()
  { QF_STACK_PUSH(ThreadedSocketConnection::read)

    int bytes =3D 0;
    char buffer[BUFSIZ];
    try
    {
      if ( !socket_isValid( m_socket ) )
        return false;

      socket_fionread( m_socket, bytes );
      if (!bytes)
         bytes =3D sizeof (buffer);
      int result =3D recv( m_socket, buffer, bytes, 0 );
      if ( result <=3D 0 ) { throw SocketRecvFailed(); }
      m_parser.addToStream (buffer, result);
      processStream ();
      return true;
    }
    catch ( SocketRecvFailed& e )
    {
      if (m_pSession)
        m_pSession->getLog()->onEvent( e.what() );
      return false;
    }

    QF_STACK_POP
  }
  ---8<---

  Compared to the current QF version, this function:

    a.. Uses a single fixed buffer for all reads, keeping memory usage =
more or less constant regardless of incoming data rates

    b.. Doesn't spawn an extra thread for parsing/application callbacks=20
    c.. Causes the counterparty to block when the QuickFIX application =
can't keep up with incoming data=20
  I think all of these are benefits, though some might argue the last =
point is not and that the current read-as-fast-as-possible behavior is =
desireable.  Perhaps this could be made configurable on a per-Session =
basis if folks want the old behavior, but I personally much prefer this =
approach where the sender is blocked if we can't handle their flow fast =
enough.

  Thoughts?  Opinions?

  --=20
  Caleb Epstein
  caleb dot epstein at gmail dot com 

Re: [Quickfix-developers] Re: WeeklySession patch, getWeekDay() patch

[Oren M. <or...@qu...>]

Caleb,

I'd be happy to make you a commiter.  Just send me your account id for =
your sourceforge account.

--oren
  ----- Original Message -----=20
  From: Caleb Epstein=20
  To: or...@qu... ; =
qui...@li...=20
  Sent: Wednesday, October 19, 2005 8:03 AM
  Subject: [Quickfix-developers] Re: WeeklySession patch, getWeekDay() =
patch


  On 10/3/05, Caleb Epstein <cal...@gm...> wrote:

    The problem with my original implementation is that I cribbed the =
formula from http://scienceworld.wolfram.com/astronomy/Weekday.html =
without proper testing.  The definition of "mod" in this context  =
doesn't match C/C++ operator%.  A mod B in terms of that equation should =
always have the same sign as B (B =3D=3D 7, so the result must be =
positive), so we need to compensate when the calculation goes negative.  =
Attached is a patch that fixes this error.


  Oren, this fix is still not in CVS.  Can you apply the patch to =
FieldTypes.h please.  I don't have write access to CVS - maybe I could =
be made a committer?

  Index: FieldTypes.h
  =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
  RCS file: /cvsroot/quickfix/quickfix/src/C++/FieldTypes.h,v
  retrieving revision 1.21
  diff -u -b -r1.21 FieldTypes.h
  --- FieldTypes.h        17 Jul 2005 19:27:06 -0000      1.21
  +++ FieldTypes.h        19 Oct 2005 13:00:57 -0000
  @@ -161,9 +161,10 @@
       int m =3D M >=3D 3 ? M - 2 : M + 10;
       int Yprime =3D M >=3D 3 ? Y : Y - 1;
       int y =3D Yprime % 100;
  -    int c =3D int (Yprime / 100);
  -    return 1 + (D + int (2.6 * m - 0.2) + y + int (y / 4) + int (c / =
4) -
  +    int c =3D Yprime / 100;
  +    int wd =3D (D + int (2.6 * m - 0.2) + y + int (y / 4) + int (c / =
4) -
                  (2 * c)) % 7;
  +    return 1 + (wd < 0 ? 7 + wd : wd);
     }
  =20
     /// Convert the DateTime to a time_t.  Note that this operation


  --=20
  Caleb Epstein
  caleb dot epstein at gmail dot com 

[Quickfix-developers] Re: ThreadedSocketConnection: fix for memory growth under heavy load

[Alexey Z. <ale...@gm...>]

Caleb,

I noted that QF is hungry for memory too but I was sure it is because of 
wide std::string usage.
Creating an array of FIX::Message from a big FIX log file is a problem.
May be it's only about a VC6 STL implementation? Does anyone use 
different STL ports?

Regarding you fix I believe we need a more complicate solution like 
using of a pool of preallocated buffers.
Overall it works fine and I think it's necessary to help with memory 
management only.

Regards,
		Alexey Zubko



Alexey Zubko wrote:

>
> Message: 2
> DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
>         s=beta; d=gmail.com <http://gmail.com>;
>         h=received:message-id:date:from:to:subject:mime-version:content-type; 
>
>         b=PAjl8BgCQLA5rvkERX2WdcCPeXsQuwlbHfINvY9kG6zwSM7LhkWtuXbTzOHbto3Pifq61LnKBNhg84aMaPQHBbw5IWM9u4PD60KjVToUflT1nrL/sZIOEW2ADk5VzWd8t+1ATjoD3di1qL8/W5/BacQ2fMmZdyZUNkNM4Rvavkg=
> Message-ID: < 
> 989...@ma... 
> <mailto:989...@ma...>>
> Date: Tue, 18 Oct 2005 18:23:39 -0400
> From: Caleb Epstein <cal...@gm... 
> <mailto:cal...@gm...>>
> To:
>         " qui...@li... 
> <mailto:qui...@li...>" 
> <qui...@li... 
> <mailto:qui...@li...>>
> MIME-Version: 1.0
> Content-Type: multipart/alternative;
>         boundary="----=_Part_2780_11602300.1129674219086"
> Subject: [Quickfix-developers] ThreadedSocketConnection: fix for 
> memory growth under heavy load
> Sender: qui...@li... 
> <mailto:qui...@li...>
> Precedence: bulk
> List-Unsubscribe: < 
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers>,
>         <mailto:qui...@li... 
> <mailto:qui...@li...>?subject=unsubscribe> 
>
> List-Id: <quickfix-developers.lists.sourceforge.net 
> <http://quickfix-developers.lists.sourceforge.net>>
> List-Post: <mailto:qui...@li... 
> <mailto:qui...@li...>>
> List-Help: <mailto:qui...@li... 
> <mailto:qui...@li...>?subject=help>
> List-Subscribe: < 
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers>,
>         <mailto:qui...@li... 
> <mailto:qui...@li...>?subject=subscribe> 
>
> List-Archive: 
> <http://sourceforge.net/mailarchive/forum.php?forum=quickfix-developers>
>
> ------=_Part_2780_11602300.1129674219086
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: inline
>
> I have spent the past couple of days torture-testing a QuickFIX C++
> application with some simple test harnesses that bombard it with 
> messages.
> In the processes of this testing, I noticed that my application was 
> using a
> ridiculous amount of memory, many times larger than would be expected 
> by th=
> e
> size of the test data. So I ran everything through valgrind (not much 
> help =
> -
> no leaks detected), and eventually came upon Google's excellent "tcmalloc"
> library (see http://goog-perftools.sourceforge.net/), which can be 
> used to
> dump statistics about heap usage. It turned out that 95% of the memory was
> taken up by the memory allocations done by ThreadedSocketConnection::read!
>
> The overall design of the ThreadedSocketConnection class is relatively
> simple, and on paper looks like it should work correctly. There are two
> threads for each Session: one which reads from a socket into a new'd char
> buffer and then pushes the buffer + size onto a queue; the other 
> thread pop=
> s
> elements off of this queue, adds this data to the Parser's buffer, 
> calls th=
> e
> Parser class to process its buffer, and finally delete[]'s the buffer it
> first got from the queue. The code appears to be correct, and doesn't 
> leak
> memory in the purest sense, but the real-world behavior when incoming
> traffic is high is not good.
>
> When a counterparty sends data faster than your application can 
> process it,
> the "read" thread in the ThreadedSocketConnection will do a fine job 
> keepin=
> g
> up with the socket I/O, but the queue in between these two threads 
> will end
> up growing very large, containing all of the data that has been read from
> the socket but which has yet to be processed by the Parser and 
> Session. In =
> a
> perfect world, this memory usage would be identical to the size of the
> messages that have been recv'd, but memory allocators are inefficient and
> what ends up happening is that the application ends up using perhaps 
> 10-20x
> more memory than this! My application, which should have used perhaps 
> 125 M=
> B
> of memory (most of it due to a mmap'ed transaction log file) had ballooned
> up to 775 MB! As a comparison, the FIX incoming log file is only 38 MB in
> size.
>
> Thankfully, there is a simple fix to this problem, and it actually
> simplifies the code and eliminates one thread per session. Here's my 
> versio=
> n
> of ThreadedSocketConnection::read:
>
> ---8<---
> bool ThreadedSocketConnection::read()
> { QF_STACK_PUSH(ThreadedSocketConnection::read)
>
> int bytes =3D 0;
> char buffer[BUFSIZ];
> try
> {
> if ( !socket_isValid( m_socket ) )
> return false;
>
> socket_fionread( m_socket, bytes );
> if (!bytes)
> bytes =3D sizeof (buffer);
> int result =3D recv( m_socket, buffer, bytes, 0 );
> if ( result <=3D 0 ) { throw SocketRecvFailed(); }
> m_parser.addToStream (buffer, result);
> processStream ();
> return true;
> }
> catch ( SocketRecvFailed& e )
> {
> if (m_pSession)
> m_pSession->getLog()->onEvent( e.what() );
> return false;
> }
>
> QF_STACK_POP
> }
> ---8<---
>
> Compared to the current QF version, this function:
>
>    - Uses a single fixed buffer for all reads, keeping memory usage more
>    or less constant regardless of incoming data rates
>     - Doesn't spawn an extra thread for parsing/application callbacks
>    - Causes the counterparty to block when the QuickFIX application can't
>    keep up with incoming data
>
> I think all of these are benefits, though some might argue the last 
> point i=
> s
> not and that the current read-as-fast-as-possible behavior is desireable.
> Perhaps this could be made configurable on a per-Session basis if 
> folks wan=
> t
> the old behavior, but I personally much prefer this approach where the
> sender is blocked if we can't handle their flow fast enough.
>
> Thoughts? Opinions?
>
> --
> Caleb Epstein
> caleb dot epstein at gmail dot com
>

[Quickfix-developers] Re: WeeklySession patch, getWeekDay() patch

[Caleb E. <cal...@gm...>]

On 10/3/05, Caleb Epstein <cal...@gm...> wrote:

> The problem with my original implementation is that I cribbed the formula
> from http://scienceworld.wolfram.com/astronomy/Weekday.html without prope=
r
> testing. The definition of "mod" in this context doesn't match C/C++
> operator%. A mod B in terms of that equation should always have the same
> sign as B (B =3D=3D 7, so the result must be positive), so we need to com=
pensate
> when the calculation goes negative. Attached is a patch that fixes this
> error.
>

Oren, this fix is still not in CVS. Can you apply the patch to
FieldTypes.hplease. I don't have write access to CVS - maybe I could
be made a
committer?

Index: FieldTypes.h
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/quickfix/quickfix/src/C++/FieldTypes.h,v
retrieving revision 1.21
diff -u -b -r1.21 FieldTypes.h
--- FieldTypes.h 17 Jul 2005 19:27:06 -0000 1.21
+++ FieldTypes.h 19 Oct 2005 13:00:57 -0000
@@ -161,9 +161,10 @@
int m =3D M >=3D 3 ? M - 2 : M + 10;
int Yprime =3D M >=3D 3 ? Y : Y - 1;
int y =3D Yprime % 100;
- int c =3D int (Yprime / 100);
- return 1 + (D + int (2.6 * m - 0.2) + y + int (y / 4) + int (c / 4) -
+ int c =3D Yprime / 100;
+ int wd =3D (D + int (2.6 * m - 0.2) + y + int (y / 4) + int (c / 4) -
(2 * c)) % 7;
+ return 1 + (wd < 0 ? 7 + wd : wd);
}

/// Convert the DateTime to a time_t. Note that this operation


--
Caleb Epstein
caleb dot epstein at gmail dot com