Thread: [Quickfix-developers] messages_log table entries
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-developers
|
From: Sean K. <sea...@pi...> - 2006-02-01 14:02:35
|
Hi Oren, I'm testing with 1.11.0 using the PostgreSQL store/log functionality and = had a question. I think it's great that the incoming/outgoing logs have = been condensed into the single messages_log table, but I was surprised = that the sendercompid / targetcompid column values don't change to = indicate the direction of the message. I'm guessing these values are = just set based on the sessionID for the configured session. Without = this functionality the text field has to be grepped through to determine = the direction of the message. Was this by design? I also found a slight bug in the sql scripts packaged to create the db = tables. The messages_log_table.sql file still creates the incoming_log = table, rather than the messages_log table. Changing the name corrected = the issue, but without the change, the Log doesn't write any of the = messages. Just a head's up. Regards, Sean Kirkpatrick |
|
From: Oren M. <or...@qu...> - 2006-02-01 15:39:33
|
Thanks Sean, I checked in a fix for the bug. We can't really alter the compid's as it would interfere with the counterparties log. Imagine you have an acceptor running on one machine, and an initiator on another which are connected. If they are sharing the same database, they will sort of walk all over each other and then things will really get confusing. So the sessionid fields need to remain constant so they can act as a key. One option could be to add a direction column which indicates if a message is incoming or outgoing. --oren Sean Kirkpatrick wrote: > Hi Oren, > > I'm testing with 1.11.0 using the PostgreSQL store/log functionality > and had a question. I think it's great that the incoming/outgoing > logs have been condensed into the single messages_log table, but I was > surprised that the sendercompid / targetcompid column values don't > change to indicate the direction of the message. I'm guessing these > values are just set based on the sessionID for the configured > session. Without this functionality the text field has to be grepped > through to determine the direction of the message. Was this by design? > > I also found a slight bug in the sql scripts packaged to create the db > tables. The messages_log_table.sql file still creates the > incoming_log table, rather than the messages_log table. Changing the > name corrected the issue, but without the change, the Log doesn't > write any of the messages. Just a head's up. > > Regards, > > Sean Kirkpatrick > > > |
|
From: Brian E. <azz...@ya...> - 2006-02-01 15:55:34
|
Oren - Is that actually a realistic scenario? I'm struggling to come up with a situation where someone would have a FIX acceptor and a FIX initiator talking to each other and sharing the same database log file outside of a badly constructed test environment. Isn't the log file supposed to show what actually was received, as opposed to some normalized view of the message data? Swapping the compIDs changes the message - if I want to reconstruct the message from the log table, I now have to have program logic that can figure out which direction the message was going and denormalize the data. I can't say that I like that design choice. Of course, I vastly prefered having separate log files. I wouldn't have minded having the option to have a single log (especially if it was *in addition to* the separate logs), but I now have to modify my workflow (and some scripts) to deal with one log. It's kind of a drag. - Brian Erst Thynk Software, Inc. ----- Original Message ---- From: Oren Miller <or...@qu...> To: Sean Kirkpatrick <sea...@pi...> Cc: qui...@li... Sent: Wednesday, February 01, 2006 9:32:34 AM Subject: Re: [Quickfix-developers] messages_log table entries QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX Support: http://www.quickfixengine.org/services.html Thanks Sean, I checked in a fix for the bug. We can't really alter the compid's as it would interfere with the counterparties log. Imagine you have an acceptor running on one machine, and an initiator on another which are connected. If they are sharing the same database, they will sort of walk all over each other and then things will really get confusing. So the sessionid fields need to remain constant so they can act as a key. One option could be to add a direction column which indicates if a message is incoming or outgoing. --oren Sean Kirkpatrick wrote: > Hi Oren, > > I'm testing with 1.11.0 using the PostgreSQL store/log functionality > and had a question. I think it's great that the incoming/outgoing > logs have been condensed into the single messages_log table, but I was > surprised that the sendercompid / targetcompid column values don't > change to indicate the direction of the message. I'm guessing these > values are just set based on the sessionID for the configured > session. Without this functionality the text field has to be grepped > through to determine the direction of the message. Was this by design? > > I also found a slight bug in the sql scripts packaged to create the db > tables. The messages_log_table.sql file still creates the > incoming_log table, rather than the messages_log table. Changing the > name corrected the issue, but without the change, the Log doesn't > write any of the messages. Just a head's up. > > Regards, > > Sean Kirkpatrick > > > ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ Quickfix-developers mailing list Qui...@li... https://lists.sourceforge.net/lists/listinfo/quickfix-developers |
|
From: Oren M. <or...@qu...> - 2006-02-01 16:49:20
|
Well, the message itself has the CompIDs in the correct place. We are *not* changing the contents of the message at all. The message column always reflects exactly what was sent or received. We are just talking about the database columns which identify the session. This concept has not changed from the previous release. The incoming and outgoing tables would contain the same values in these columns. The only thing that has happened is the tables have been consolidated to retain the sequence of events. All the data within the columns remains exactly the same. I don't think it is a contrived scenario to have the counterparties share the same database. Many people use FIX for interprocess communication within their internal systems. Those systems often share the same database. This is a setup I've seen several times, so I do believe its support is necessary. The only problem with the separate log files was again, losing the sequence of events. It actually made it very difficult to debug scenarios, especially when logs were posted to the mailing list and extra explanation had to be provided to indicate what happened in what order (if it could be determined at all). If we do go about supporting one or two files, I would definately make one file the default for this reason. It also seems to me to be the overall preference of the community, but I'd like to hear additional comments. Unfortunately in the log file it is rather difficult to add metadata, as then the file is no longer pure fix which could give it problems with some utilities. This precludes the file from having any sort of incoming/outgoing type metadata. --oren Brian Erst wrote: >Oren - > >Is that actually a realistic scenario? I'm struggling to come up with a situation where someone would have a FIX acceptor and a FIX initiator talking to each other and sharing the same database log file outside of a badly constructed test environment. > >Isn't the log file supposed to show what actually was received, as opposed to some normalized view of the message data? Swapping the compIDs changes the message - if I want to reconstruct the message from the log table, I now have to have program logic that can figure out which direction the message was going and denormalize the data. I can't say that I like that design choice. > >Of course, I vastly prefered having separate log files. I wouldn't have minded having the option to have a single log (especially if it was *in addition to* the separate logs), but I now have to modify my workflow (and some scripts) to deal with one log. It's kind of a drag. > >- Brian Erst >Thynk Software, Inc. > >----- Original Message ---- >From: Oren Miller <or...@qu...> >To: Sean Kirkpatrick <sea...@pi...> >Cc: qui...@li... >Sent: Wednesday, February 01, 2006 9:32:34 AM >Subject: Re: [Quickfix-developers] messages_log table entries > >QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html >QuickFIX Support: http://www.quickfixengine.org/services.html > >Thanks Sean, I checked in a fix for the bug. > >We can't really alter the compid's as it would interfere with the >counterparties log. Imagine you have an acceptor running on one >machine, and an initiator on another which are connected. If they are >sharing the same database, they will sort of walk all over each other >and then things will really get confusing. So the sessionid fields need >to remain constant so they can act as a key. > >One option could be to add a direction column which indicates if a >message is incoming or outgoing. > >--oren > >Sean Kirkpatrick wrote: > > > >>Hi Oren, >> >>I'm testing with 1.11.0 using the PostgreSQL store/log functionality >>and had a question. I think it's great that the incoming/outgoing >>logs have been condensed into the single messages_log table, but I was >>surprised that the sendercompid / targetcompid column values don't >>change to indicate the direction of the message. I'm guessing these >>values are just set based on the sessionID for the configured >>session. Without this functionality the text field has to be grepped >>through to determine the direction of the message. Was this by design? >> >>I also found a slight bug in the sql scripts packaged to create the db >>tables. The messages_log_table.sql file still creates the >>incoming_log table, rather than the messages_log table. Changing the >>name corrected the issue, but without the change, the Log doesn't >>write any of the messages. Just a head's up. >> >>Regards, >> >>Sean Kirkpatrick >> >> >> >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 >_______________________________________________ >Quickfix-developers mailing list >Qui...@li... >https://lists.sourceforge.net/lists/listinfo/quickfix-developers > > > > > > > |
|
From: Brian E. <azz...@ya...> - 2006-02-01 16:59:16
|
Oren - I'm sorry that I confused the message data from the meta-data in the tables - it seemed unclear in the original posting. I agree with the others that there needs to be some easy way to extract sent from received messages - a direction flag sounds like it would work. Initiators and acceptors sharing the same database table still seems like a recipe for disaster to me, but as you've seen it in the real world, supporting it is a must. I like having (the option of) separate incoming/outgoing log files as it makes it easy for me to concentrate on the counterparty's message flow - in my circumstance, we have a stable acceptor that doesn't change but a whole bunch of new counterparty initiators that usually have problems. I like being able to quickly select their problematic messages and send them an explanatory email with the attached messages. Also, as field ordering isn't particularly important in FIX, figuring out which messages are inbound and which are outbound can be a pain - the senderCompIDs are not going to conveniently line up. I've also written a few scripts that use the two files that will need to change. It's not a huge problem, and I can see how the new layout would be better for lots of people, but people build systems that expect certain behavior and when that behavior changes in a non-backward compatible way, it's disruptive. Compared to the work involved in creating QF, these are quibbles. Thanks for the good work on QF. - Brian ----- Original Message ---- From: Oren Miller <or...@qu...> To: Brian Erst <azz...@ya...> Cc: Sean Kirkpatrick <sea...@pi...>; qui...@li... Sent: Wednesday, February 01, 2006 10:34:06 AM Subject: Re: [Quickfix-developers] messages_log table entries Well, the message itself has the CompIDs in the correct place. We are *not* changing the contents of the message at all. The message column always reflects exactly what was sent or received. We are just talking about the database columns which identify the session. This concept has not changed from the previous release. The incoming and outgoing tables would contain the same values in these columns. The only thing that has happened is the tables have been consolidated to retain the sequence of events. All the data within the columns remains exactly the same. I don't think it is a contrived scenario to have the counterparties share the same database. Many people use FIX for interprocess communication within their internal systems. Those systems often share the same database. This is a setup I've seen several times, so I do believe its support is necessary. The only problem with the separate log files was again, losing the sequence of events. It actually made it very difficult to debug scenarios, especially when logs were posted to the mailing list and extra explanation had to be provided to indicate what happened in what order (if it could be determined at all). If we do go about supporting one or two files, I would definately make one file the default for this reason. It also seems to me to be the overall preference of the community, but I'd like to hear additional comments. Unfortunately in the log file it is rather difficult to add metadata, as then the file is no longer pure fix which could give it problems with some utilities. This precludes the file from having any sort of incoming/outgoing type metadata. --oren Brian Erst wrote: >Oren - > >Is that actually a realistic scenario? I'm struggling to come up with a situation where someone would have a FIX acceptor and a FIX initiator talking to each other and sharing the same database log file outside of a badly constructed test environment. > >Isn't the log file supposed to show what actually was received, as opposed to some normalized view of the message data? Swapping the compIDs changes the message - if I want to reconstruct the message from the log table, I now have to have program logic that can figure out which direction the message was going and denormalize the data. I can't say that I like that design choice. > >Of course, I vastly prefered having separate log files. I wouldn't have minded having the option to have a single log (especially if it was *in addition to* the separate logs), but I now have to modify my workflow (and some scripts) to deal with one log. It's kind of a drag. > >- Brian Erst >Thynk Software, Inc. > >----- Original Message ---- >From: Oren Miller <or...@qu...> >To: Sean Kirkpatrick <sea...@pi...> >Cc: qui...@li... >Sent: Wednesday, February 01, 2006 9:32:34 AM >Subject: Re: [Quickfix-developers] messages_log table entries > >QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html >QuickFIX Support: http://www.quickfixengine.org/services.html > >Thanks Sean, I checked in a fix for the bug. > >We can't really alter the compid's as it would interfere with the >counterparties log. Imagine you have an acceptor running on one >machine, and an initiator on another which are connected. If they are >sharing the same database, they will sort of walk all over each other >and then things will really get confusing. So the sessionid fields need >to remain constant so they can act as a key. > >One option could be to add a direction column which indicates if a >message is incoming or outgoing. > >--oren > >Sean Kirkpatrick wrote: > > > >>Hi Oren, >> >>I'm testing with 1.11.0 using the PostgreSQL store/log functionality >>and had a question. I think it's great that the incoming/outgoing >>logs have been condensed into the single messages_log table, but I was >>surprised that the sendercompid / targetcompid column values don't >>change to indicate the direction of the message. I'm guessing these >>values are just set based on the sessionID for the configured >>session. Without this functionality the text field has to be grepped >>through to determine the direction of the message. Was this by design? >> >>I also found a slight bug in the sql scripts packaged to create the db >>tables. The messages_log_table.sql file still creates the >>incoming_log table, rather than the messages_log table. Changing the >>name corrected the issue, but without the change, the Log doesn't >>write any of the messages. Just a head's up. >> >>Regards, >> >>Sean Kirkpatrick >> >> >> >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 >_______________________________________________ >Quickfix-developers mailing list >Qui...@li... >https://lists.sourceforge.net/lists/listinfo/quickfix-developers > > > > > > > |
|
From: Caleb E. <cal...@gm...> - 2006-02-01 17:03:28
|
On 2/1/06, Brian Erst <azz...@ya...> wrote: > > and I can see how the new layout would be better for lots of people, but > people build systems that expect certain behavior and when that behavior > changes in a non-backward compatible way, it's disruptive. Perhaps the 1-or-2 file behavior could be configurable through yet another Session setting? -- Caleb Epstein caleb dot epstein at gmail dot com |
|
From: Brian E. <azz...@ya...> - 2006-02-01 17:07:28
|
That would be a great idea. Again, I think having a 1-file message log is a great thing to offer - I'd just like to have the option of retaining the existing behavior. In fact, I'd love to be able to do both (3 log files - incoming, outgoing and interleaved). The more the merrier! >grin< - Brian ----- Original Message ---- From: Caleb Epstein <cal...@gm...> To: Brian Erst <azz...@ya...> Cc: Oren Miller <or...@qu...>; qui...@li... Sent: Wednesday, February 01, 2006 11:03:12 AM Subject: Re: [Quickfix-developers] messages_log table entries On 2/1/06, Brian Erst <azz...@ya...> wrote: and I can see how the new layout would be better for lots of people, but people build systems that expect certain behavior and when that behavior changes in a non-backward compatible way, it's disruptive. Perhaps the 1-or-2 file behavior could be configurable through yet another Session setting? -- Caleb Epstein caleb dot epstein at gmail dot com |
|
From: Oren M. <or...@qu...> - 2006-02-01 19:08:47
|
Ok, I'll look into something along these lines. --oren Brian Erst wrote: > That would be a great idea. Again, I think having a 1-file message log > is a great thing to offer - I'd just like to have the option of > retaining the existing behavior. In fact, I'd love to be able to do > both (3 log files - incoming, outgoing and interleaved). The more the > merrier! >grin< > > - Brian > > ----- Original Message ---- > From: Caleb Epstein <cal...@gm...> > To: Brian Erst <azz...@ya...> > Cc: Oren Miller <or...@qu...>; > qui...@li... > Sent: Wednesday, February 01, 2006 11:03:12 AM > Subject: Re: [Quickfix-developers] messages_log table entries > > On 2/1/06, *Brian Erst* <azz...@ya... > <mailto:azz...@ya...>> wrote: > > and I can see how the new layout would be better for lots of > people, but people build systems that expect certain behavior and > when that behavior changes in a non-backward compatible way, it's > disruptive. > > > Perhaps the 1-or-2 file behavior could be configurable through yet > another Session setting? > > -- > Caleb Epstein > caleb dot epstein at gmail dot com |
|
From: Caleb E. <cal...@gm...> - 2006-02-01 16:18:03
|
On 2/1/06, Oren Miller <or...@qu...> wrote:
> One option could be to add a direction column which indicates if a
> message is incoming or outgoing.
What about normalizing the data (or is that de-normalizing?) a bit more and
putting a unique ID on the sessions table and using that as a foreign key
for the event_log, message_log and messages tables. Then the *compid
columns could be per-message, if they are even needed. The single byte
direction column might be more sensible and save bit of space.
Something like (syntax may not be 100%):
CREATE TABLE sessions (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
beginstring CHAR(8) NOT NULL,
sendercompid VARCHAR(64) NOT NULL,
targetcompid VARCHAR(64) NOT NULL,
session_qualifier VARCHAR(64) NOT NULL,
creation_time DATETIME NOT NULL,
incoming_seqnum INT NOT NULL,
outgoing_seqnum INT NOT NULL,
PRIMARY KEY (id)
UNIQUE KEY session_id (beginstring, sendercompid, targetcompid,
session_qualifier)
)
CREATE TABLE message_log (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
session_id INT UNSIGNED NOT NULL,
direction ENUM ('in', 'out') NOT NULL,
text BLOB NOT NULL,
PRIMARY KEY (id),
FOREIGN KEY (session_id) REFERENCES sessions (id)
)
CREATE TABLE messages (
session_id INT UNSIGNED NOT NULL,
msgseqnum INT NOT NULL,
message BLOB NOT NULL,
UNIQUE KEY (session_id, msgseqnum),
FOREIGN KEY (session_id) REFERENCES sessions (id)
)
FYI, I just noticed this code in MySQLLog.cpp:
void MySQLLog::clear()
{ QF_STACK_PUSH(MySQLLog::clear)
MySQLQuery incoming( "DELETE FROM incoming_log" );
MySQLQuery outgoing( "DELETE FROM outgoing_log" );
MySQLQuery event( "DELETE FROM event_log" );
m_pConnection->execute( incoming );
m_pConnection->execute( outgoing );
m_pConnection->execute( event );
QF_STACK_POP
}
Shouldn't this query be qualifying the delete to match only messages
corresponding to its session? II don't think Log::clear ever gets called
though. It seems to be called from SessionState::clear, but I don't see
anything calling that method.
--
Caleb Epstein
caleb dot epstein at gmail dot com
|
