quickfixj-users — General mailing list for QuickFIX/J topics.

[Quickfixj-users] The order of the tags in the message I sent is mixed

[Pavel T. <pav...@wh...>]

 Hi,

I construct a message in the following way:


quickfix.fix44.MarketDataRequest message = new quickfix.fix44.
MarketDataRequest();
char zero = '0';
char one = '1';
message.setField(new quickfix.field.MDReqID("EURUSD")); // 262
message.setField(new quickfix.field.SubscriptionRequestType(one)); // 263
message.setField(new quickfix.field.MarketDepth(1)); // 264
message.setField(new quickfix.field.MDUpdateType(0)); // 265

message.setField(new quickfix.field.NoMDEntryTypes(2)); // 267
message.setField(new quickfix.field.MDEntryType(zero)); // 269
message.setField(new quickfix.field.MDEntryType(one)); // 269

message.setField(new quickfix.field.NoRelatedSym(1)); // 146
message.setField(new quickfix.field.SecurityID("4001")); // 48
message.setField(new quickfix.field.SecurityIDSource("8")); // 22
Session.sendToTarget(message, senderCompID, marketTargetCompID);

I also load a dictionary in which tags 146, 48 and 22 are after tag 269.
But the message sent to the trader looks like this:

8=FIX.4.4^A9=121^A35=V^A34=2^A49=USERNAME^A52=20190821-11:11:42.045^A56=TRADER^A22=8^A48=4001^A146=1^A262=EURUSD^A263=1^A264=1^A265=0^A267=2^A269=1^A10=191^A

The confusing thing is that tags 146, 48 and 22 are mixed and moved before
tag 262 even though the dictionary doesn't allow that.

I have two suggestion why this may happen:

   - Quickfix/J mixes the tags because of an issue or because I miss some
   configuration.
   - The dictionary provided by the trader is not loaded and I use some
   defaultdictionary which brings me to this issue.

I don't know if that would help but here is the cfg file:

[DEFAULT]
ConnectionType=initiator
BeginString=FIX.4.4

StartTime=08:30:00
EndTime=21:30:00
ReconnectInterval=5
HeartBtInt=5
LogonTimeout=60

FileStorePath=target/data/sessions/
FileLogPath=target/data/logs/

SenderCompID=USERNAME
UseDataDictionary=Y
SocketUseSSL=Y


[SESSION]
TargetCompID=TRADER
SocketConnectHost=URL_OF_THE_TRADER
SocketConnectPort=443
DataDictionary=/var/quickfixj/automated/DICTIONARY.xml


Regards,
Pavel

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Abhishek S. <atm...@gm...>]

Thanks for input, will try this and lets see what happens.

Regards,
Abhishek

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Robert N. <rob...@gm...>]

So it appears you cannot add them to a database fast enough? If that’s the case then either improve that or queue them on some queue technology providing you are able to process other messages without having read all of these first.

Any heap dump should tell you what you’re retaining and why you might be retaining too much.

> On Aug 19, 2019, at 8:07 AM, Colin DuPlantis <co...@ma...> wrote:
> 
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
> 
> 
> As a good test, if I were you guys, I would disable all your code in fromApp, ie, don't do anything with the messages you receive from ICE, and see if that helps your memory consumption.
> 
> What that will tell you is, if the memory consumption issue improves, the problem is in your own code, not QFJ, and the problem is likely somewhere where you're storing these strings in a Map or List, etc. If the memory consumption problem does not improve, then you might be right that there's something to look at in QFJ and it would be wise to use a newer version as the next test, as Chris suggests.
> 
> - Colin
> 
> -- 
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884
> https://www.marketcetera.com
> 
> 
> 
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Colin D. <co...@ma...>]

As a good test, if I were you guys, I would disable all your code in 
fromApp, ie, don't do anything with the messages you receive from ICE, 
and see if that helps your memory consumption.

What that will tell you is, if the memory consumption issue improves, 
the problem is in your own code, not QFJ, and the problem is likely 
somewhere where you're storing these strings in a Map or List, etc. If 
the memory consumption problem does not improve, then you might be right 
that there's something to look at in QFJ and it would be wise to use a 
newer version as the next test, as Chris suggests.

- Colin

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884
https://www.marketcetera.com

Re: [Quickfixj-users] Support for embedded FIX message in XML

[Philip W. <ph...@wh...>]

I think you want QuickFIXN’s mailing list. That’s not Java code.

http://quickfixn.org/help

Best,

Philip Whitehouse

> On 19 Aug 2019, at 12:16, sundar <sun...@cu...> wrote:
> 
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
> 
> 
> Hi All I am getting the below issue while processing the embedded fix message . QuickFix.MessageParseError: Could not parse message: Error at position (509) while parsing msg ( 8=FIX.4.2
9=501
35=n
34=5
369=3
52=20190819-07:55:16.780
43=Y
49=DUMMY
50=G
56=DUMMY
57=DUMMY
 122=20190819-07:51:43.657
143=IN
212=370
213=8=FIX.4.2
9=334
35=8
34=841
369=841
52=20190819-07:51:43.863
 49=DUMMY
50=G
56=8H1000N
57=DUMMY
143=US,IL
1=ACP
6=0
11=DUMMY
14=0
17=99530:13006
20=0
37=9951353869
38=78
 39=0
40=2
41=0
44=5866
48=997870
54=1
55=P$
59=0
60=20190819-07:51:43.854
107=0WSV9
150=0
151=78
167=FUT
432=20190819
1028=Y
 5979=1566201103854801450
9717=DUMMY
10=096

10=034
) System.FormatException: Input string was not in a correct format. at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info) at System.Convert.ToInt32(String value) at QuickFix.Message.ExtractField(String msgstr, Int32& pos, DataDictionary sessionDD, DataDictionary appDD) --- End of inner exception stack trace --- at QuickFix.Message.ExtractField(String msgstr, Int32& pos, DataDictionary sessionDD, DataDictionary appDD) at QuickFix.Message.FromString(String msgstr, Boolean validate, DataDictionary sessionDD, DataDictionary appDD, IMessageFactory msgFactory, Boolean ignoreBody) at QuickFix.Message.FromString(String msgstr, Boolean validate, DataDictionary sessionDD, DataDictionary appDD, IMessageFactory msgFactory) at QuickFix.MessageBuilder.Build() at QuickFix.Session.Next(MessageBuilder msgBuilder) at QuickFix.Session.NextMessage(String msgStr)    at QuickFix.Session.Next(String msgStr) at QuickFix.SocketInitiatorThread.ProcessStream() at QuickFix.SocketInitiatorThread.Read() Thanks in Advance for your help Thanks Sundar 
> Sent from the QuickFIX/J mailing list archive at Nabble.com.
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Re: [Quickfixj-users] Support for embedded FIX message in XML

[sundar <sun...@cu...>]

Hi AllI am getting the below issue while processing the embedded fix message
.  QuickFix.MessageParseError: Could not parse message: Error at position
(509) while parsing msg
(8=FIX.4.2
9=501
35=n
34=5
369=3
52=20190819-07:55:16.780
43=Y
49=DUMMY
50=G
56=DUMMY
57=DUMMY
122=20190819-07:51:43.657
143=IN
212=370
213=*8=FIX.4.2
9=334
35=8
34=841
369=841
52=20190819-07:51:43.863
49=DUMMY
50=G
56=8H1000N
57=DUMMY
143=US,IL
1=ACP
6=0
11=DUMMY
14=0
17=99530:13006
20=0
37=9951353869
38=78
39=0
40=2
41=0
44=5866
48=997870
54=1
55=P$
59=0
60=20190819-07:51:43.854
107=0WSV9
150=0
151=78
167=FUT
432=20190819
1028=Y
5979=1566201103854801450
9717=DUMMY
10=096
*
10=034
)System.FormatException:
Input string was not in a correct format.   at
System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer&
number, NumberFormatInfo info, Boolean parseDecimal)   at
System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo
info)   at System.Convert.ToInt32(String value)   at
QuickFix.Message.ExtractField(String msgstr, Int32& pos, DataDictionary
sessionDD, DataDictionary appDD)   --- End of inner exception stack trace
---   at QuickFix.Message.ExtractField(String msgstr, Int32& pos,
DataDictionary sessionDD, DataDictionary appDD)   at
QuickFix.Message.FromString(String msgstr, Boolean validate, DataDictionary
sessionDD, DataDictionary appDD, IMessageFactory msgFactory, Boolean
ignoreBody)   at QuickFix.Message.FromString(String msgstr, Boolean
validate, DataDictionary sessionDD, DataDictionary appDD, IMessageFactory
msgFactory)   at QuickFix.MessageBuilder.Build()   at
QuickFix.Session.Next(MessageBuilder msgBuilder)   at
QuickFix.Session.NextMessage(String msgStr)   at
QuickFix.Session.Next(String msgStr)   at
QuickFix.SocketInitiatorThread.ProcessStream()   at
QuickFix.SocketInitiatorThread.Read()Thanks in Advance for your
helpThanksSundar



--
Sent from: http://quickfix-j.364392.n2.nabble.com/

Re: [Quickfixj-users] Support for embedded FIX message in XML

[sundar <sun...@cu...>]

Hi AllI am getting the below issue while processing the embedded fix message
.  QuickFix.MessageParseError: Could not parse message: Error at position
(509) while parsing msg
(8=FIX.4.2
9=501
35=n
34=5
369=3
52=20190819-07:55:16.780
43=Y
49=DUMMY
50=G
56=DUMMY
57=DUMMY
122=20190819-07:51:43.657
143=IN
212=370
213=*8=FIX.4.2
9=334
35=8
34=841
369=841
52=20190819-07:51:43.863
49=DUMMY
50=G
56=8H1000N
57=DUMMY
143=US,IL
1=ACP
6=0
11=DUMMY
14=0
17=99530:13006
20=0
37=9951353869
38=78
39=0
40=2
41=0
44=5866
48=997870
54=1
55=P$
59=0
60=20190819-07:51:43.854
107=0WSV9
150=0
151=78
167=FUT
432=20190819
1028=Y
5979=1566201103854801450
9717=DUMMY
10=096
*
10=034
)System.FormatException:
Input string was not in a correct format.   at
System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer&
number, NumberFormatInfo info, Boolean parseDecimal)   at
System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo
info)   at System.Convert.ToInt32(String value)   at
QuickFix.Message.ExtractField(String msgstr, Int32& pos, DataDictionary
sessionDD, DataDictionary appDD)   --- End of inner exception stack trace
---   at QuickFix.Message.ExtractField(String msgstr, Int32& pos,
DataDictionary sessionDD, DataDictionary appDD)   at
QuickFix.Message.FromString(String msgstr, Boolean validate, DataDictionary
sessionDD, DataDictionary appDD, IMessageFactory msgFactory, Boolean
ignoreBody)   at QuickFix.Message.FromString(String msgstr, Boolean
validate, DataDictionary sessionDD, DataDictionary appDD, IMessageFactory
msgFactory)   at QuickFix.MessageBuilder.Build()   at
QuickFix.Session.Next(MessageBuilder msgBuilder)   at
QuickFix.Session.NextMessage(String msgStr)   at
QuickFix.Session.Next(String msgStr)   at
QuickFix.SocketInitiatorThread.ProcessStream()   at
QuickFix.SocketInitiatorThread.Read()Thanks in Advance for your
helpThanksSundar



--
Sent from: http://quickfix-j.364392.n2.nabble.com/

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Christoph J. <chr...@ma...>]

I am referring to the QuickFIX/J version. Just noticed that the version you are using is rather old.

In the fromApp() callback QFJ will notify you of new application messages, i.e. your security 
definitions.
https://github.com/quickfix-j/quickfixj#creating-a-quickfixj-application

I just wanted to take a look on how you implemented that method. I cannot imagine that QFJ itself 
holds onto the message strings unless of course you are getting messages which are hundred megabytes 
in size that do not fit into the heap.

Chris.


On 19.08.19 05:29, Abhishek Singh wrote:
> Yes, I am receiving messages.
> Which version you are referring to? Is something fixed in newer version?
>
> Didnot get your last que about fromApp callback ?
>
> On Sun, 18 Aug 2019 at 11:45 PM, Christoph John <chr...@ma... 
> <mailto:chr...@ma...>> wrote:
>
>     That case is about sending messages, i.e. a slow consumer.
>     I thought you are receiving the messages? However, could you try with a newer QFJ version?
>     What does your fromApp callback look like?
>
>     Cheers
>     Chris
>
>     Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh <atm...@gm...
>     <mailto:atm...@gm...>>:
>
>         Hi Chris,
>
>         my case looks similar to this -
>         http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894
>         any thought ?
>
>         On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...
>         <mailto:atm...@gm...>> wrote:
>
>             We are saving security definitions in our DB table.
>
>             It looks like QFJ keeps whole string in memory, whenever we try to fetch definitions
>             for multiple markets ( e. g. - Oil, Power, Financial Power etc.) simultaneously, heap
>             dump shows a lots of memory is used by QFJ while processing incoming definitions in
>             string format.
>
>             I am using QFJ 44 (version
>

>             .4).
>
>             On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...
>             <mailto:chr...@ma...>> wrote:
>
>                 I do not know how multithreading will improve your memory allocation?!
>
>                 What are you doing with the security definitions? I assume you store them in the
>                 same application that also uses QuickFIX/J? QFJ itself will not keep incoming
>                 messages in memory by itself.
>
>                 Cheers,
>                 Chris.
>
>                 Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...
>                 <mailto:atm...@gm...>>:
>
>                     QuickFIX/J Documentation:http://www.quickfixj.org/documentation/
>                     QuickFIX/J Support:http://www.quickfixj.org/support/
>
>
>                     We have implemented QuickFIX/J for Trade capture and fetching ICE security
>                     definitions.
>
>                     We have allocated 2 GB memory to our ICE Adapter which is responsible for
>                     connecting with ICE, flowing deals captured on ICE and fetching securities
>                     from ICE for various markets.
>
>                     When we try to fetch ICE Security Definitions for Financial Power market,
>                     ICE Adapter is going Out of Memory though we allocated 2 GB.
>                     When we increased memory allocation to 3 GB then it worked fine without
>                     going OOM.
>
>                     What we observe is, Quickfix engine is taking too much server memory while
>                     processing incoming definitions from ICE. If less number of definitions are
>                     coming then there is no issue.
>
>                     Do any one had similar issue in past ? If yes then how you fixed this. Is
>                     there any option in FIX protocol to make it multi threaded or something like
>                     that.
>
>                     Any information will be helpful, waiting for valuable feedback.
>
>                     Thanks,
>                     Abhishek Singh
>
>
>
>                     --
>                     Sent from:http://quickfix-j.364392.n2.nabble.com/
>                     ----------------------------------------------------------------------------------------------------
>                     Quickfixj-users mailing list
>                     Qui...@li...  <mailto:Qui...@li...>
>                     https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Abhishek S. <atm...@gm...>]

Yes, I am receiving messages.
Which version you are referring to? Is something fixed in newer version?

Didnot get your last que about fromApp callback ?

On Sun, 18 Aug 2019 at 11:45 PM, Christoph John <chr...@ma...>
wrote:

> That case is about sending messages, i.e. a slow consumer.
> I thought you are receiving the messages? However, could you try with a
> newer QFJ version?
> What does your fromApp callback look like?
>
> Cheers
> Chris
>
> Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh <
> atm...@gm...>:
>>
>> Hi Chris,
>>
>> my case looks similar to this -
>> http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894
>>
>> any thought ?
>>
>> On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...>
>> wrote:
>>
>>> We are saving security definitions in our DB table.
>>>
>>> It looks like QFJ keeps whole string in memory, whenever we try to fetch
>>> definitions for multiple markets ( e. g. - Oil, Power, Financial Power
>>> etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while
>>> processing incoming definitions in string format.
>>>
>>> I am using QFJ 44 (version 1.6.4).
>>>
>>> On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...>
>>> wrote:
>>>
>>>> I do not know how multithreading will improve your memory allocation?!
>>>>
>>>> What are you doing with the security definitions? I assume you store
>>>> them in the same application that also uses QuickFIX/J? QFJ itself will not
>>>> keep incoming messages in memory by itself.
>>>>
>>>> Cheers,
>>>> Chris.
>>>>
>>>> Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>:
>>>>>
>>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>>
>>>>>
>>>>> We have implemented QuickFIX/J for Trade capture and fetching ICE security
>>>>> definitions.
>>>>>
>>>>> We have allocated 2 GB memory to our ICE Adapter which is responsible for
>>>>> connecting with ICE, flowing deals captured on ICE and fetching securities
>>>>> from ICE for various markets.
>>>>>
>>>>> When we try to fetch ICE Security Definitions for Financial Power market,
>>>>> ICE Adapter is going Out of Memory though we allocated 2 GB.
>>>>> When we increased memory allocation to 3 GB then it worked fine without
>>>>> going OOM.
>>>>>
>>>>> What we observe is, Quickfix engine is taking too much server memory while
>>>>> processing incoming definitions from ICE. If less number of definitions are
>>>>> coming then there is no issue.
>>>>>
>>>>> Do any one had similar issue in past ? If yes then how you fixed this. Is
>>>>> there any option in FIX protocol to make it multi threaded or something like
>>>>> that.
>>>>>
>>>>> Any information will be helpful, waiting for valuable feedback.
>>>>>
>>>>> Thanks,
>>>>> Abhishek Singh
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://quickfix-j.364392.n2.nabble.com/
>>>>> ------------------------------
>>>>> Quickfixj-users mailing list
>>>>> Qui...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>>>>
>>>>>

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Christoph J. <chr...@ma...>]

That case is about sending messages, i.e. a slow consumer. 
I thought you are receiving the messages? However, could you try with a newer QFJ version? 
What does your fromApp callback look like? 

Cheers 
Chris 

Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh <atm...@gm...>:
>Hi Chris,
>
>my case looks similar to this -
>http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894
>
>any thought ?
>
>On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...>
>wrote:
>
>> We are saving security definitions in our DB table.
>>
>> It looks like QFJ keeps whole string in memory, whenever we try to
>fetch
>> definitions for multiple markets ( e. g. - Oil, Power, Financial
>Power
>> etc.) simultaneously, heap dump shows a lots of memory is used by QFJ
>while
>> processing incoming definitions in string format.
>>
>> I am using QFJ 44 (version 1.6.4).
>>
>> On Sun, Aug 18, 2019 at 6:56 PM Christoph John
><chr...@ma...>
>> wrote:
>>
>>> I do not know how multithreading will improve your memory
>allocation?!
>>>
>>> What are you doing with the security definitions? I assume you store
>them
>>> in the same application that also uses QuickFIX/J? QFJ itself will
>not keep
>>> incoming messages in memory by itself.
>>>
>>> Cheers,
>>> Chris.
>>>
>>> Am 18. August 2019 13:23:30 MESZ schrieb imabhi
><atm...@gm...>:
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> We have implemented QuickFIX/J for Trade capture and fetching ICE
>security
>>>> definitions.
>>>>
>>>> We have allocated 2 GB memory to our ICE Adapter which is
>responsible for
>>>> connecting with ICE, flowing deals captured on ICE and fetching
>securities
>>>> from ICE for various markets.
>>>>
>>>> When we try to fetch ICE Security Definitions for Financial Power
>market,
>>>> ICE Adapter is going Out of Memory though we allocated 2 GB.
>>>> When we increased memory allocation to 3 GB then it worked fine
>without
>>>> going OOM.
>>>>
>>>> What we observe is, Quickfix engine is taking too much server
>memory while
>>>> processing incoming definitions from ICE. If less number of
>definitions are
>>>> coming then there is no issue.
>>>>
>>>> Do any one had similar issue in past ? If yes then how you fixed
>this. Is
>>>> there any option in FIX protocol to make it multi threaded or
>something like
>>>> that.
>>>>
>>>> Any information will be helpful, waiting for valuable feedback.
>>>>
>>>> Thanks,
>>>> Abhishek Singh
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://quickfix-j.364392.n2.nabble.com/
>>>> ------------------------------
>>>> Quickfixj-users mailing list
>>>> Qui...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>>>
>>>>

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Abhishek S. <atm...@gm...>]

Hi Chris,

my case looks similar to this -
http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894

any thought ?

On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...> wrote:

> We are saving security definitions in our DB table.
>
> It looks like QFJ keeps whole string in memory, whenever we try to fetch
> definitions for multiple markets ( e. g. - Oil, Power, Financial Power
> etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while
> processing incoming definitions in string format.
>
> I am using QFJ 44 (version 1.6.4).
>
> On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...>
> wrote:
>
>> I do not know how multithreading will improve your memory allocation?!
>>
>> What are you doing with the security definitions? I assume you store them
>> in the same application that also uses QuickFIX/J? QFJ itself will not keep
>> incoming messages in memory by itself.
>>
>> Cheers,
>> Chris.
>>
>> Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>:
>>>
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> We have implemented QuickFIX/J for Trade capture and fetching ICE security
>>> definitions.
>>>
>>> We have allocated 2 GB memory to our ICE Adapter which is responsible for
>>> connecting with ICE, flowing deals captured on ICE and fetching securities
>>> from ICE for various markets.
>>>
>>> When we try to fetch ICE Security Definitions for Financial Power market,
>>> ICE Adapter is going Out of Memory though we allocated 2 GB.
>>> When we increased memory allocation to 3 GB then it worked fine without
>>> going OOM.
>>>
>>> What we observe is, Quickfix engine is taking too much server memory while
>>> processing incoming definitions from ICE. If less number of definitions are
>>> coming then there is no issue.
>>>
>>> Do any one had similar issue in past ? If yes then how you fixed this. Is
>>> there any option in FIX protocol to make it multi threaded or something like
>>> that.
>>>
>>> Any information will be helpful, waiting for valuable feedback.
>>>
>>> Thanks,
>>> Abhishek Singh
>>>
>>>
>>>
>>> --
>>> Sent from: http://quickfix-j.364392.n2.nabble.com/
>>> ------------------------------
>>> Quickfixj-users mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>>
>>>

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Abhishek S. <atm...@gm...>]

We are saving security definitions in our DB table.

It looks like QFJ keeps whole string in memory, whenever we try to fetch
definitions for multiple markets ( e. g. - Oil, Power, Financial Power
etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while
processing incoming definitions in string format.

I am using QFJ 44 (version 1.6.4).

On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...>
wrote:

> I do not know how multithreading will improve your memory allocation?!
>
> What are you doing with the security definitions? I assume you store them
> in the same application that also uses QuickFIX/J? QFJ itself will not keep
> incoming messages in memory by itself.
>
> Cheers,
> Chris.
>
> Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>:
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> We have implemented QuickFIX/J for Trade capture and fetching ICE security
>> definitions.
>>
>> We have allocated 2 GB memory to our ICE Adapter which is responsible for
>> connecting with ICE, flowing deals captured on ICE and fetching securities
>> from ICE for various markets.
>>
>> When we try to fetch ICE Security Definitions for Financial Power market,
>> ICE Adapter is going Out of Memory though we allocated 2 GB.
>> When we increased memory allocation to 3 GB then it worked fine without
>> going OOM.
>>
>> What we observe is, Quickfix engine is taking too much server memory while
>> processing incoming definitions from ICE. If less number of definitions are
>> coming then there is no issue.
>>
>> Do any one had similar issue in past ? If yes then how you fixed this. Is
>> there any option in FIX protocol to make it multi threaded or something like
>> that.
>>
>> Any information will be helpful, waiting for valuable feedback.
>>
>> Thanks,
>> Abhishek Singh
>>
>>
>>
>> --
>> Sent from: http://quickfix-j.364392.n2.nabble.com/
>> ------------------------------
>> Quickfixj-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>
>>

Re: [Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[Christoph J. <chr...@ma...>]

I do not know how multithreading will improve your memory allocation?!

What are you doing with the security definitions?  I assume you store them in the same application that also uses QuickFIX/J? QFJ itself will not keep incoming messages in memory by itself. 

Cheers,
Chris. 

Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>:
>QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>We have implemented QuickFIX/J for Trade capture and fetching ICE
>security
>definitions.
>
>We have allocated 2 GB memory to our ICE Adapter which is responsible
>for
>connecting with ICE, flowing deals captured on ICE and fetching
>securities
>from ICE for various markets.
>
>When we try to fetch ICE Security Definitions for Financial Power
>market,
>ICE Adapter is going Out of Memory though we allocated 2 GB. 
>When we increased memory allocation to 3 GB then it worked fine without
>going OOM.
>
>What we observe is, Quickfix engine is taking too much server memory
>while
>processing incoming definitions from ICE. If less number of definitions
>are
>coming then there is no issue.
>
>Do any one had similar issue in past ? If yes then how you fixed this.
>Is
>there any option in FIX protocol to make it multi threaded or something
>like
>that.
>
>Any information will be helpful, waiting for valuable feedback.
>
>Thanks,
>Abhishek Singh
>
>
>
>--
>Sent from: http://quickfix-j.364392.n2.nabble.com/
>
>
>_______________________________________________
>Quickfixj-users mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[Quickfixj-users] Server going Out of Memory when fetching ICE Security Definitions for Financial Power

[imabhi <atm...@gm...>]

We have implemented QuickFIX/J for Trade capture and fetching ICE security
definitions.

We have allocated 2 GB memory to our ICE Adapter which is responsible for
connecting with ICE, flowing deals captured on ICE and fetching securities
from ICE for various markets.

When we try to fetch ICE Security Definitions for Financial Power market,
ICE Adapter is going Out of Memory though we allocated 2 GB. 
When we increased memory allocation to 3 GB then it worked fine without
going OOM.

What we observe is, Quickfix engine is taking too much server memory while
processing incoming definitions from ICE. If less number of definitions are
coming then there is no issue.

Do any one had similar issue in past ? If yes then how you fixed this. Is
there any option in FIX protocol to make it multi threaded or something like
that.

Any information will be helpful, waiting for valuable feedback.

Thanks,
Abhishek Singh



--
Sent from: http://quickfix-j.364392.n2.nabble.com/

Re: [Quickfixj-users] Support for embedded FIX message in XML

[Christoph J. <chr...@ma...>]

Do you have an example? 
Which issues do you have?


Am 18. August 2019 10:05:11 MESZ schrieb sundar <sun...@cu...>:
>QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>Hi All
>
>   I am getting issues to parse the FIX message with <RTRF></RTRF> Tag
>embedded in the FIX message string . Can anyone assist 
>
>Thanks in Advance for your help
>
>Thanks
>Sundar
>
>
>
>--
>Sent from: http://quickfix-j.364392.n2.nabble.com/
>
>
>_______________________________________________
>Quickfixj-users mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Re: [Quickfixj-users] Support for embedded FIX message in XML

[sundar <sun...@cu...>]

Hi All

   I am getting issues to parse the FIX message with <RTRF></RTRF> Tag
embedded in the FIX message string . Can anyone assist 

Thanks in Advance for your help

Thanks
Sundar



--
Sent from: http://quickfix-j.364392.n2.nabble.com/

Re: [Quickfixj-users] QuickFIX/J 2.1.1 - SLF4J Log Categories appear incorrect

[philip <ph...@wh...>]

I'm saying per your second paragraph - i.e. reverting both commits and 
going back to the per-message append.

I'll push a PR for that momentarily.

It's fine - it took me a few trials to really grok the situation too :)

Philip

On 2019-08-15 13:17, Christoph John wrote:
> Hi,
> 
> do you mean the "correction" in
> https://github.com/quickfix-j/quickfixj/pull/232 has to be reverted?
> 
> To be honest, until Tommy's mail from a few hours ago I did not
> understand the original problem.
> But when I am not mistaken then we simply need to remove the logPrefix
> from the Logger again and add the prefix it to every message that is
> logged. Just as it was before (QFJ 1.6.3). Of course you do not gain
> any performance benefit from that but given that it breaks
> functionality it is pointless.
> 
> Sorry for misunderstanding the original problem.
> 
> Cheers,
> Chris.
> 
> On 15.08.19 14:08, philip wrote:
> 
>> Hi all,
>> 
>> I had an experiment with this as it's blocking us rolling out the
>> other stuff since this commit.
>> 
>> Just to expand a bit:
>> 
>> Previously you'd have a logging setup that looks like:
>> 
>> log4j.logger.quickfixj.event=INFO
>> log4j.appender.stdout.layout.ConversionPattern=%c{1} %p
>> %d{yyyyMMdd-HH:mm:ss.SSS} - %m%m
>> 
>> And with the default configuration you'd get
>> 
>> event INFO 20190102-03:04:05.678 - 8=FIX.4.4....
>> 
>> With the original suffix this gives you nothing because the
>> fully-qualified logger name is:
>> 
>> "quickfixj.eventFIX.4.4:SENDER->TARGET: "
>> 
>> With it as a prefix it gives you nothing because you have to declare
>> something like this for every session.
>> 
>> log4j.logger.FIX.4.4:SENDER->TARGET.quickfixj.event=INFO
>> 
>> because the fully-qualified logger name is:
>> 
>> "FIX.4.4:SENDER->TARGET.quickfixj.event" and so the heirarchical
>> structure fails.
>> 
>> I made a change locally to make the fully-qualified logger name:
>> 
>> "quickfixj.event.FIX.4.4:SENDER->TARGET: "
>> 
>> But even this isn't usable as it gives:
>> 
>> 4:SENDER->TARGET INFO 20190102-03:04:05.678 - 8=FIX.4.4....
>> 
>> I think we have to revert this change completely and rethink it.
>> Maybe we have to extend PatternLayout to provide the session ID as a
>> parameter instead.
>> 
>> Best regards,
>> 
>> On 2019-08-14 19:17, th...@ft... wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>> 
>> Hi Chris,
>> 
>> Apologies for the delay in response.  I just just saw where a
>> resolution for this issue was entered in JIRA for version 2.2.0.  I
>> do
>> not believe this will solve the problem.  The issue is with the
>> SLF4J
>> logger categories.
>> 
>> It appears the original intent of the code change was to eliminate
>> the
>> reallocation of the log message string for each log message
>> (presumably for performance consideration).  The problem is the
>> logger
>> “category” (a.k.a. “name") is also affected by the way it was
>> implemented.
>> 
>> For example, when setting the following configuration...
>> 
>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>> 
>> One would expect “event” log messages to be sent to Log4J with
>> category ’US.THEM.event’ when properties 'SenderCompID=US' and
>> 'TargetCompID=THEM’. However, what occurs is the log category
>> becomes 'US.THEM.eventFIX.4.4:US->THEM: ‘.  The effect of this is
>> to
>> invalidate any Log4J configuration that is applicable to the
>> expected
>> QF/J configured SLF4J log categories.
>> 
>> The category should not be changed if the apparent intent is to just
>> 
>> prepend the SessionID in the actual log messages (which is the way
>> it
>> worked previously in QF/J 1.6.3 and before).
>> 
>> All the current “fix” for QF/J 2.2.0 will do is make the
>> category
>> 'FIX.4.4:US->THEM: US.THEM.event’.  This is still incorrect as the
>> 
>> category does not match what is configured by
>> ‘SLF4JLogEventCategory’.
>> 
>> A quick review of the way Log4J categories work may be referenced in
>> 
>> the “Architecture - Logger Hierarchy” section of the Log4J
>> manual
>> at https://logging.apache.org/log4j/2.x/manual/architecture.html. It
>> 
>> is basically a namespace.
>> 
>> Regards - Tommy
>> 
>> On Jul 11, 2019, at 3:24 AM, Christoph John
>> <chr...@ma...> wrote:
>> 
>> Hi Tommy,
>> 
>> sorry for the late reply.
>> Just to understand: you were not specifying the setting
>> SLF4JLogPrependSessionID=Y before? I mean you shouldn't because its
>> default seems to be Y.
>> And when you now put SLF4JLogPrependSessionID=N then the behaviour
>> is as before, i.e. with QFJ 1.6.3?
>> 
>> Did the log filenames change or only the content of the files, i.e.
>> the logger name?
>> 
>> Cheers,
>> Chris.
>> 
>> On 26/04/2019 01:11, th...@ft... wrote:
>> 
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>> 
>> Has anybody with authority had a chance to look at this?  The
>> change introduced has the presumed unintended consequences of
>> breaking things for some of us using Log4J.
>> 
>> Regards - Tommy
>> 
>> On Apr 10, 2019, at 10:26 PM, th...@ft... wrote:
>> 
>> I recently downloaded QF/J 2.1.1 and am upgrading from QF/J 1.6.3.
>> Most of the work was converting my application code to utilize the
>> ‘java.time’ package.  I have everything running and going thru
>> some tests.  In the process, I noticed log messages in files they
>> were not in before.  We use Log4J and configure the following QF/J
>> properties…
>> 
>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>> 
>> 
> SLF4JLogIncomingMessageCategory=${senderCompID}.${targetCompID}.msg.incoming
>> 
>> 
>> 
> SLF4JLogOutgoingMessageCategory=${senderCompID}.${targetCompID}.msg.outgoing
>> 
>> 
>> ..and the recently discovered (not in the user manual)…
>> 
>> 
> SLF4JLogErrorEventCategory=${senderCompID}.${targetCompID}.errorEvent
>> 
>> 
>> I have spent hours reviewing and comparing QF/J source code trying
>> to determine what changed.
>> 
>> It appears there is a configuration setting
>> 'SLF4JLogPrependSessionID’ described as "Controls whether session
>> ID is prepended to log message.”
>> 
>> The behavior of this setting has been completely changed by the
>> following commit of source file ‘SLF4JLog.java’...
>> 
>> 
> https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263
>> 
>> 
>> It appears the intent of this change did not create the desired
>> effect as my log “categories" now all have the SessionID appended.
>> 
>> (e.g. 'SENDER.TARGET.eventFIX.4.4:SENDER->TARGET: ')
>> 
>> The current workaround is to configure... SLF4JLogPrependSessionID=N
>> 
>> 
>> Would someone please elaborate on the intended use of configuration
>> setting ‘SLF4JLogPrependSessionID’.
>> 
>> Regards - Tommy
>> 
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>> 
>> --
>> Christoph John
>> Software Engineering
>> T +49 241 557080-28
>> chr...@ma...
>> 
>> MACD GmbH
>> Oppenhoffallee 103
>> 52066 Aachen, Germany
>> www.macd.com [1] [1]
>> 
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>> Geschäftsführer: George Macdonald
>> 
>> Links:
>> ------
>> [1] http://www.macd.com/
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> 
> --
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma...
> 
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com [1]
> 
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
> 
> 
> Links:
> ------
> [1] http://www.macd.com

Re: [Quickfixj-users] QuickFIX/J 2.1.1 - SLF4J Log Categories appear incorrect

[Christoph J. <chr...@ma...>]

Hi,

do you mean the "correction" in https://github.com/quickfix-j/quickfixj/pull/232 has to be reverted?

To be honest, until Tommy's mail from a few hours ago I did not understand the original problem.
But when I am not mistaken then we simply need to remove the logPrefix from the Logger again and add 
the prefix it to every message that is logged. Just as it was before (QFJ 1.6.3). Of course you do 
not gain any performance benefit from that but given that it breaks functionality it is pointless.

Sorry for misunderstanding the original problem.

Cheers,
Chris.

On 15.08.19 14:08, philip wrote:
> Hi all,
>
> I had an experiment with this as it's blocking us rolling out the other stuff since this commit.
>
> Just to expand a bit:
>
> Previously you'd have a logging setup that looks like:
>
> log4j.logger.quickfixj.event=INFO
> log4j.appender.stdout.layout.ConversionPattern=%c{1} %p %d{yyyyMMdd-HH:mm:ss.SSS} - %m%m
>
> And with the default configuration you'd get
>
> event INFO 20190102-03:04:05.678 - 8=FIX.4.4....
>
> With the original suffix this gives you nothing because the fully-qualified logger name is:
>
> "quickfixj.eventFIX.4.4:SENDER->TARGET: "
>
> With it as a prefix it gives you nothing because you have to declare something like this for every 
> session.
>
> log4j.logger.FIX.4.4:SENDER->TARGET.quickfixj.event=INFO
>
> because the fully-qualified logger name is:
>
> "FIX.4.4:SENDER->TARGET.quickfixj.event" and so the heirarchical structure fails.
>
> I made a change locally to make the fully-qualified logger name:
>
> "quickfixj.event.FIX.4.4:SENDER->TARGET: "
>
> But even this isn't usable as it gives:
>
> 4:SENDER->TARGET INFO 20190102-03:04:05.678 - 8=FIX.4.4....
>
> I think we have to revert this change completely and rethink it. Maybe we have to extend 
> PatternLayout to provide the session ID as a parameter instead.
>
> Best regards,
>
> On 2019-08-14 19:17, th...@ft... wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>>
>> Hi Chris,
>>
>> Apologies for the delay in response.  I just just saw where a
>> resolution for this issue was entered in JIRA for version 2.2.0.  I do
>> not believe this will solve the problem.  The issue is with the SLF4J
>> logger categories.
>>
>> It appears the original intent of the code change was to eliminate the
>> reallocation of the log message string for each log message
>> (presumably for performance consideration).  The problem is the logger
>> “category” (a.k.a. “name") is also affected by the way it was
>> implemented.
>>
>> For example, when setting the following configuration...
>>
>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>>
>> One would expect “event” log messages to be sent to Log4J with
>> category ’US.THEM.event’ when properties 'SenderCompID=US' and
>> 'TargetCompID=THEM’. However, what occurs is the log category
>> becomes 'US.THEM.eventFIX.4.4:US->THEM: ‘.  The effect of this is to
>> invalidate any Log4J configuration that is applicable to the expected
>> QF/J configured SLF4J log categories.
>>
>> The category should not be changed if the apparent intent is to just
>> prepend the SessionID in the actual log messages (which is the way it
>> worked previously in QF/J 1.6.3 and before).
>>
>> All the current “fix” for QF/J 2.2.0 will do is make the category
>> 'FIX.4.4:US->THEM: US.THEM.event’.  This is still incorrect as the
>> category does not match what is configured by
>> ‘SLF4JLogEventCategory’.
>>
>> A quick review of the way Log4J categories work may be referenced in
>> the “Architecture - Logger Hierarchy” section of the Log4J manual
>> at https://logging.apache.org/log4j/2.x/manual/architecture.html. It
>> is basically a namespace.
>>
>> Regards - Tommy
>>
>>> On Jul 11, 2019, at 3:24 AM, Christoph John
>>> <chr...@ma...> wrote:
>>>
>>> Hi Tommy,
>>>
>>> sorry for the late reply.
>>> Just to understand: you were not specifying the setting
>>> SLF4JLogPrependSessionID=Y before? I mean you shouldn't because its
>>> default seems to be Y.
>>> And when you now put SLF4JLogPrependSessionID=N then the behaviour
>>> is as before, i.e. with QFJ 1.6.3?
>>>
>>> Did the log filenames change or only the content of the files, i.e.
>>> the logger name?
>>>
>>> Cheers,
>>> Chris.
>>>
>>> On 26/04/2019 01:11, th...@ft... wrote:
>>>
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>> Has anybody with authority had a chance to look at this?  The
>>> change introduced has the presumed unintended consequences of
>>> breaking things for some of us using Log4J.
>>>
>>> Regards - Tommy
>>>
>>> On Apr 10, 2019, at 10:26 PM, th...@ft... wrote:
>>>
>>> I recently downloaded QF/J 2.1.1 and am upgrading from QF/J 1.6.3.
>>> Most of the work was converting my application code to utilize the
>>> ‘java.time’ package.  I have everything running and going thru
>>> some tests.  In the process, I noticed log messages in files they
>>> were not in before.  We use Log4J and configure the following QF/J
>>> properties…
>>>
>>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>>>
>> SLF4JLogIncomingMessageCategory=${senderCompID}.${targetCompID}.msg.incoming
>>>
>> SLF4JLogOutgoingMessageCategory=${senderCompID}.${targetCompID}.msg.outgoing
>>>
>>> ..and the recently discovered (not in the user manual)…
>>>
>>>
>> SLF4JLogErrorEventCategory=${senderCompID}.${targetCompID}.errorEvent
>>>
>>> I have spent hours reviewing and comparing QF/J source code trying
>>> to determine what changed.
>>>
>>> It appears there is a configuration setting
>>> 'SLF4JLogPrependSessionID’ described as "Controls whether session
>>> ID is prepended to log message.”
>>>
>>> The behavior of this setting has been completely changed by the
>>> following commit of source file ‘SLF4JLog.java’...
>>>
>>>
>> https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263 
>>
>>>
>>>
>>> It appears the intent of this change did not create the desired
>>> effect as my log “categories" now all have the SessionID appended.
>>> (e.g. 'SENDER.TARGET.eventFIX.4.4:SENDER->TARGET: ')
>>>
>>> The current workaround is to configure... SLF4JLogPrependSessionID=N
>>>
>>>
>>> Would someone please elaborate on the intended use of configuration
>>> setting ‘SLF4JLogPrependSessionID’.
>>>
>>> Regards - Tommy
>>>
>>> _______________________________________________
>>> Quickfixj-users mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>
>> -- 
>> Christoph John
>> Software Engineering
>> T +49 241 557080-28
>> chr...@ma...
>>
>> MACD GmbH
>> Oppenhoffallee 103
>> 52066 Aachen, Germany
>> www.macd.com [1]
>>
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>> Geschäftsführer: George Macdonald
>>
>>
>>
>> Links:
>> ------
>> [1] http://www.macd.com/
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] QuickFIX/J 2.1.1 - SLF4J Log Categories appear incorrect

[philip <ph...@wh...>]

Hi all,

I had an experiment with this as it's blocking us rolling out the other 
stuff since this commit.

Just to expand a bit:

Previously you'd have a logging setup that looks like:

log4j.logger.quickfixj.event=INFO
log4j.appender.stdout.layout.ConversionPattern=%c{1} %p 
%d{yyyyMMdd-HH:mm:ss.SSS} - %m%m

And with the default configuration you'd get

event INFO 20190102-03:04:05.678 - 8=FIX.4.4....

With the original suffix this gives you nothing because the 
fully-qualified logger name is:

"quickfixj.eventFIX.4.4:SENDER->TARGET: "

With it as a prefix it gives you nothing because you have to declare 
something like this for every session.

log4j.logger.FIX.4.4:SENDER->TARGET.quickfixj.event=INFO

because the fully-qualified logger name is:

"FIX.4.4:SENDER->TARGET.quickfixj.event" and so the heirarchical 
structure fails.

I made a change locally to make the fully-qualified logger name:

"quickfixj.event.FIX.4.4:SENDER->TARGET: "

But even this isn't usable as it gives:

4:SENDER->TARGET INFO 20190102-03:04:05.678 - 8=FIX.4.4....

I think we have to revert this change completely and rethink it. Maybe 
we have to extend PatternLayout to provide the session ID as a parameter 
instead.

Best regards,

On 2019-08-14 19:17, th...@ft... wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
> 
> 
> 
> Hi Chris,
> 
> Apologies for the delay in response.  I just just saw where a
> resolution for this issue was entered in JIRA for version 2.2.0.  I do
> not believe this will solve the problem.  The issue is with the SLF4J
> logger categories.
> 
> It appears the original intent of the code change was to eliminate the
> reallocation of the log message string for each log message
> (presumably for performance consideration).  The problem is the logger
> “category” (a.k.a. “name") is also affected by the way it was
> implemented.
> 
> For example, when setting the following configuration...
> 
> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
> 
> One would expect “event” log messages to be sent to Log4J with
> category ’US.THEM.event’ when properties 'SenderCompID=US' and
> 'TargetCompID=THEM’. However, what occurs is the log category
> becomes 'US.THEM.eventFIX.4.4:US->THEM: ‘.  The effect of this is to
> invalidate any Log4J configuration that is applicable to the expected
> QF/J configured SLF4J log categories.
> 
> The category should not be changed if the apparent intent is to just
> prepend the SessionID in the actual log messages (which is the way it
> worked previously in QF/J 1.6.3 and before).
> 
> All the current “fix” for QF/J 2.2.0 will do is make the category
> 'FIX.4.4:US->THEM: US.THEM.event’.  This is still incorrect as the
> category does not match what is configured by
> ‘SLF4JLogEventCategory’.
> 
> A quick review of the way Log4J categories work may be referenced in
> the “Architecture - Logger Hierarchy” section of the Log4J manual
> at https://logging.apache.org/log4j/2.x/manual/architecture.html. It
> is basically a namespace.
> 
> Regards - Tommy
> 
>> On Jul 11, 2019, at 3:24 AM, Christoph John
>> <chr...@ma...> wrote:
>> 
>> Hi Tommy,
>> 
>> sorry for the late reply.
>> Just to understand: you were not specifying the setting
>> SLF4JLogPrependSessionID=Y before? I mean you shouldn't because its
>> default seems to be Y.
>> And when you now put SLF4JLogPrependSessionID=N then the behaviour
>> is as before, i.e. with QFJ 1.6.3?
>> 
>> Did the log filenames change or only the content of the files, i.e.
>> the logger name?
>> 
>> Cheers,
>> Chris.
>> 
>> On 26/04/2019 01:11, th...@ft... wrote:
>> 
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>> 
>> Has anybody with authority had a chance to look at this?  The
>> change introduced has the presumed unintended consequences of
>> breaking things for some of us using Log4J.
>> 
>> Regards - Tommy
>> 
>> On Apr 10, 2019, at 10:26 PM, th...@ft... wrote:
>> 
>> I recently downloaded QF/J 2.1.1 and am upgrading from QF/J 1.6.3.
>> Most of the work was converting my application code to utilize the
>> ‘java.time’ package.  I have everything running and going thru
>> some tests.  In the process, I noticed log messages in files they
>> were not in before.  We use Log4J and configure the following QF/J
>> properties…
>> 
>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>> 
> SLF4JLogIncomingMessageCategory=${senderCompID}.${targetCompID}.msg.incoming
>> 
> SLF4JLogOutgoingMessageCategory=${senderCompID}.${targetCompID}.msg.outgoing
>> 
>> ..and the recently discovered (not in the user manual)…
>> 
>> 
> SLF4JLogErrorEventCategory=${senderCompID}.${targetCompID}.errorEvent
>> 
>> I have spent hours reviewing and comparing QF/J source code trying
>> to determine what changed.
>> 
>> It appears there is a configuration setting
>> 'SLF4JLogPrependSessionID’ described as "Controls whether session
>> ID is prepended to log message.”
>> 
>> The behavior of this setting has been completely changed by the
>> following commit of source file ‘SLF4JLog.java’...
>> 
>> 
> https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263
>> 
>> 
>> It appears the intent of this change did not create the desired
>> effect as my log “categories" now all have the SessionID appended.
>> (e.g. 'SENDER.TARGET.eventFIX.4.4:SENDER->TARGET: ')
>> 
>> The current workaround is to configure... SLF4JLogPrependSessionID=N
>> 
>> 
>> Would someone please elaborate on the intended use of configuration
>> setting ‘SLF4JLogPrependSessionID’.
>> 
>> Regards - Tommy
>> 
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> 
> --
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma...
> 
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com [1]
> 
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
> 
> 
> 
> Links:
> ------
> [1] http://www.macd.com/
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Re: [Quickfixj-users] QuickFIX/J 2.1.1 - SLF4J Log Categories appear incorrect

[<th...@ft...>]

Hi Chris,

Apologies for the delay in response.  I just just saw where a resolution for this issue was entered in JIRA for version 2.2.0.  I do not believe this will solve the problem.  The issue is with the SLF4J logger categories.

It appears the original intent of the code change was to eliminate the reallocation of the log message string for each log message (presumably for performance consideration).  The problem is the logger “category” (a.k.a. “name") is also affected by the way it was implemented.

For example, when setting the following configuration...

SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event

One would expect “event” log messages to be sent to Log4J with category ’US.THEM.event’ when properties 'SenderCompID=US' and 'TargetCompID=THEM’. However, what occurs is the log category becomes 'US.THEM.eventFIX.4.4:US->THEM: ‘.  The effect of this is to invalidate any Log4J configuration that is applicable to the expected QF/J configured SLF4J log categories.

The category should not be changed if the apparent intent is to just prepend the SessionID in the actual log messages (which is the way it worked previously in QF/J 1.6.3 and before).

All the current “fix” for QF/J 2.2.0 will do is make the category 'FIX.4.4:US->THEM: US.THEM.event’.  This is still incorrect as the category does not match what is configured by ‘SLF4JLogEventCategory’.

A quick review of the way Log4J categories work may be referenced in the “Architecture - Logger Hierarchy” section of the Log4J manual at https://logging.apache.org/log4j/2.x/manual/architecture.html <https://logging.apache.org/log4j/2.x/manual/architecture.html>. It is basically a namespace.

Regards - Tommy

> On Jul 11, 2019, at 3:24 AM, Christoph John <chr...@ma...> wrote:
> 
> Hi Tommy,
> 
> sorry for the late reply.
> Just to understand: you were not specifying the setting SLF4JLogPrependSessionID=Y before? I mean you shouldn't because its default seems to be Y.
> And when you now put SLF4JLogPrependSessionID=N then the behaviour is as before, i.e. with QFJ 1.6.3?
> 
> Did the log filenames change or only the content of the files, i.e. the logger name?
> 
> Cheers,
> Chris.
> 
> 
> On 26/04/2019 01:11, th...@ft... <mailto:th...@ft...> wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ <http://www.quickfixj.org/documentation/>
>> QuickFIX/J Support: http://www.quickfixj.org/support/ <http://www.quickfixj.org/support/>
>> 
>> 
>> 
>> Has anybody with authority had a chance to look at this?  The change introduced has the presumed unintended consequences of breaking things for some of us using Log4J.
>> 
>> Regards - Tommy
>> 
>>> On Apr 10, 2019, at 10:26 PM, th...@ft... <mailto:th...@ft...> wrote:
>>> 
>>> I recently downloaded QF/J 2.1.1 and am upgrading from QF/J 1.6.3.  Most of the work was converting my application code to utilize the ‘java.time’ package.  I have everything running and going thru some tests.  In the process, I noticed log messages in files they were not in before.  We use Log4J and configure the following QF/J properties…
>>> 
>>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>>> SLF4JLogIncomingMessageCategory=${senderCompID}.${targetCompID}.msg.incoming
>>> SLF4JLogOutgoingMessageCategory=${senderCompID}.${targetCompID}.msg.outgoing
>>> 
>>> ..and the recently discovered (not in the user manual)…
>>> 
>>> SLF4JLogErrorEventCategory=${senderCompID}.${targetCompID}.errorEvent
>>> 
>>> I have spent hours reviewing and comparing QF/J source code trying to determine what changed.
>>> 
>>> It appears there is a configuration setting 'SLF4JLogPrependSessionID’ described as "Controls whether session ID is prepended to log message.”
>>> 
>>> The behavior of this setting has been completely changed by the following commit of source file ‘SLF4JLog.java’...
>>> 
>>> https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263 <https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263>
>>> 
>>> It appears the intent of this change did not create the desired effect as my log “categories" now all have the SessionID appended. (e.g. 'SENDER.TARGET.eventFIX.4.4:SENDER->TARGET: ')
>>> 
>>> The current workaround is to configure... SLF4JLogPrependSessionID=N
>>> 
>>> Would someone please elaborate on the intended use of configuration setting ‘SLF4JLogPrependSessionID’.
>>> 
>>> Regards - Tommy
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li... <mailto:Qui...@li...>
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users <https://lists.sourceforge.net/lists/listinfo/quickfixj-users>
> 
> -- 
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma... <mailto:chr...@ma...>
> 
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com <http://www.macd.com/>
> 
> Amtsgericht Aachen: HRB 8151 
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald

Re: [Quickfixj-users] QuickFIX/J 2.1.1 - SLF4J Log Categories appear incorrect

[Christoph J. <chr...@ma...>]

FWIW, this has now been corrected in https://www.quickfixj.org/jira/browse/QFJ-983


Cheers,
Chris.

On 11.07.19 10:34, Christoph John via Quickfixj-users wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
> Thinking further about this I believe that you do not need that setting when you already name your 
> categories e.g. "${senderCompID}.${targetCompID}." So setting it to N should be OK.
> The only problem I see now is that the sessionID seems to be appended and not prepended.
>
> Chris.
>
> On 11/07/2019 10:24, Christoph John via Quickfixj-users wrote:
>> QuickFIX/J Documentation:http://www.quickfixj.org/documentation/
>> QuickFIX/J Support:http://www.quickfixj.org/support/
>>
>>
>>
>> Hi Tommy,
>>
>> sorry for the late reply.
>> Just to understand: you were not specifying the setting SLF4JLogPrependSessionID=Y before? I mean 
>> you shouldn't because its default seems to be Y.
>> And when you now put SLF4JLogPrependSessionID=N then the behaviour is as before, i.e. with QFJ 1.6.3?
>>
>> Did the log filenames change or only the content of the files, i.e. the logger name?
>>
>> Cheers,
>> Chris.
>>
>>
>> On 26/04/2019 01:11, th...@ft... wrote:
>>> QuickFIX/J Documentation:http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support:http://www.quickfixj.org/support/
>>>
>>>
>>>
>>> Has anybody with authority had a chance to look at this?  The change introduced has the presumed 
>>> unintended consequences of breaking things for some of us using Log4J.
>>>
>>> Regards - Tommy
>>>
>>>> On Apr 10, 2019, at 10:26 PM, th...@ft... <mailto:th...@ft...> wrote:
>>>>
>>>> I recently downloaded QF/J 2.1.1 and am upgrading from QF/J 1.6.3.  Most of the work was 
>>>> converting my application code to utilize the ‘java.time’ package.  I have everything running 
>>>> and going thru some tests.  In the process, I noticed log messages in files they were not in 
>>>> before.  We use Log4J and configure the following QF/J properties…
>>>>
>>>> SLF4JLogEventCategory=${senderCompID}.${targetCompID}.event
>>>> SLF4JLogIncomingMessageCategory=${senderCompID}.${targetCompID}.msg.incoming
>>>> SLF4JLogOutgoingMessageCategory=${senderCompID}.${targetCompID}.msg.outgoing
>>>>
>>>> ..and the recently discovered (not in the user manual)…
>>>>
>>>> SLF4JLogErrorEventCategory=${senderCompID}.${targetCompID}.errorEvent
>>>>
>>>> I have spent hours reviewing and comparing QF/J source code trying to determine what changed.
>>>>
>>>> It appears there is a configuration setting 'SLF4JLogPrependSessionID’ described as "Controls 
>>>> whether session ID is prepended to log message.”
>>>>
>>>> The behavior of this setting has been completely changed by the following commit of source file 
>>>> ‘SLF4JLog.java’...
>>>>
>>>> https://github.com/quickfix-j/quickfixj/commit/0eb3359059d3551dde588f3cff6c836119141cf6#diff-82a6b6a09255551e3683d8943e3e4263 
>>>>
>>>>
>>>> It appears the intent of this change did not create the desired effect as my log “categories" 
>>>> now all have the SessionID appended. (e.g. 'SENDER.TARGET.eventFIX.4.4:SENDER->TARGET: ')
>>>>
>>>> The current workaround is to configure... SLF4JLogPrependSessionID=N
>>>>
>>>> Would someone please elaborate on the intended use of configuration setting 
>>>> ‘SLF4JLogPrependSessionID’.
>>>>
>>>> Regards - Tommy
>>>
>>>
>
> -- 
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma...
>
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
>
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] Support for UI64 data type

[Christoph J. <chr...@ma...>]

Hmm, is the requirement just to have a six month session without disconnection? Then just send a Logon with ResetSeqNum flag without logging out. That will reset your FIX sequence numbers without dropping the connection. Your 1000-10000 messages that the user might rerequest could be inside your application. To me it sounds like you are using the FIX message store to hold all your application's data. 
Cheers
Chris. 

Am 8. August 2019 12:08:05 MESZ schrieb "Kodippili Arachchige, Asanka" <as...@ls...>:
>The reason we want to exceed I32 max limit is to support continuous
>trading.
>e.g. a FIX user session might continue for more than 6 months, during
>that time I32 limit can be exceeded (this might take more time for
>Trading users, but for DropCopy users with All Firm permissions limit
>will reach earlier)
>For resend, we are not going to send billions of messages, only the
>most recent N (1000 to 10,000) number
>
>The C++ side will be using unsigned integer 64 bit ( 8 bytes) .
>I am adding capability to a java based test tool which acts as a client
>to test the C++ server. I was hoping to submit a pull request for
>changing the data type of sequence number related fields to long.
>Before starting to work on this I wanted to make sure that this change
>is acceptable.
>
>Thanks,
>AsankaK
>
>From: Christoph John [mailto:chr...@ma...]
>Sent: Thursday, August 08, 2019 2:13 PM
>To: Kodippili Arachchige, Asanka <as...@ls...>;
>qui...@li...; Philip Whitehouse
><ph...@wh...>
>Subject: Re: [Quickfixj-users] Support for UI64 data type
>
>Wow, OK. Not indicating a limit in the FIX spec does not mean that one
>should use sequence numbers as large as 18446744073709551615. ;)
>I just realised that you are from LSE and so are on the server side of
>things and should be able to influence the implementation: please think
>about what you are trying to achieve and whether you are using the
>right protocol for it.
>Do you really want to do resends for possibly billions of messages? Why
>are there so many messages anyway? As stated earlier, maybe FIX is not
>the right fit for your purpose?!
>I can only recommend that you consult someone with FIX knowledge before
>building your system.
>
>Apart from that: my C++ skills are a little rusty but isn't an integer
>limited to 4bytes in C++ also? The implementations for C++ and .NET are
>also using an int for sequence numbers.
>Probably it wouldn't be such a big effort to change the sequence number
>to long datatype. But still I think it is not really necessary due to
>my points stated above.
>Of course I am ready to discuss. Maybe you can outline the system that
>you want to build. I am sure people on this mailing list can give some
>advice.
>
>Cheers,
>Chris.
>
>On 08.08.19 09:02, Kodippili Arachchige, Asanka wrote:
>
>Hi,
>
>The developer who is implementing the C++ server side ( FIX GW) of this
>change had posted a question about the maximum value allowed for the
>field according to FIX spec, and the reply he got was that there is no
>limit.
>
>https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507<https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507>
>
>
>
>Sending the value as a string throws a NumberFormatException
>
>
>
>201908220122431.952|ERROR|QFJ Timer|quickfix.SocketInitiator:|    
>|Error during timer processing
>
>quickfix.FieldException: invalid integral value: 18446744073709551615:
>java.lang.NumberFormatException: For input string:
>"18446744073709551615"
>
>  at quickfix.FieldMap.newIncorrectDataException(FieldMap.java:401)
>
>  at quickfix.FieldMap.getInt(FieldMap.java:255)
>
>  at quickfix.Session.sendRaw(Session.java:2500)
>
>  at quickfix.Session.generateLogon(Session.java:1926)
>
>  at quickfix.Session.next(Session.java:1826)
>
>at
>quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:309)
>
>
>
>Thanks,
>
>AsankaK
>
>
>
>-----Original Message-----
>
>From: Philip Whitehouse [mailto:ph...@wh...]
>
>Sent: Wednesday, August 07, 2019 2:07 PM
>
>To:
>qui...@li...<mailto:qui...@li...>
>
>Subject: Re: [Quickfixj-users] Support for UI64 data type
>
>
>
>*** THIS EMAIL ORIGINATED FROM OUTSIDE OF THE ORGANISATION ***
>QuickFIX/J Documentation:
>http://www.quickfixj.org/documentation/<http://www.quickfixj.org/documentation/>
>
>QuickFIX/J Support:
>http://www.quickfixj.org/support/<http://www.quickfixj.org/support/>
>
>
>
>------------------------------------------------------------------------------------------------------------
>
>
>
>Please read these warnings and restrictions:
>
>
>
>This e-mail transmission is strictly confidential and intended solely
>for the ordinary user of the e-mail address to which it was addressed.
>It may contain legally privileged and/or CONFIDENTIAL information.
>
>
>
>The unauthorised use, disclosure, distribution and/or copying of this
>e-mail or any information it contains is prohibited and could, in
>certain circumstances, constitute a criminal offence.
>
>
>
>If you have received this e-mail in error or are not an intended
>recipient please inform the London Stock Exchange immediately by return
>e-mail or telephone 020 7797 1000.
>
>
>
>LSEG may collect, process and retain your personal information for its
>business purposes. For more information please see our Privacy Policy.
>
>
>
>We advise that in keeping with good computing practice the recipient of
>this e-mail should ensure that it is virus free. We do not accept
>responsibility for any virus that may be transferred by way of this
>e-mail.
>
>
>
>E-mail may be susceptible to data corruption, interception and
>unauthorised amendment, and we do not accept liability for any such
>corruption, interception or amendment or any consequences thereof.
>
>
>
>Calls to the London Stock Exchange may be recorded to enable the
>Exchange to carry out its regulatory responsibilities.
>
>
>
>London Stock Exchange plc
>
>10 Paternoster Square
>
>London
>
>EC4M 7LS
>
>Registered in England and Wales No 2075721
>
>
>
>------------------------------------------------------------------------------------------------------------
>
>
>
>
>
>--
>
>Christoph John
>
>Software Engineering
>
>T +49 241 557080-28
>
>chr...@ma...<mailto:chr...@ma...>
>
>
>
>MACD GmbH
>
>Oppenhoffallee 103
>
>52066 Aachen, Germany
>
>www.macd.com<http://www.macd.com>
>
>
>
>Amtsgericht Aachen: HRB 8151
>
>Ust.-Id: DE 813021663
>
>Geschäftsführer: George Macdonald

Re: [Quickfixj-users] Support for UI64 data type

[Kodippili A. A. <as...@ls...>]

The reason we want to exceed I32 max limit is to support continuous trading.
e.g. a FIX user session might continue for more than 6 months, during that time I32 limit can be exceeded (this might take more time for Trading users, but for DropCopy users with All Firm permissions limit will reach earlier)
For resend, we are not going to send billions of messages, only the most recent N (1000 to 10,000) number

The C++ side will be using unsigned integer 64 bit ( 8 bytes) .
I am adding capability to a java based test tool which acts as a client to test the C++ server. I was hoping to submit a pull request for changing the data type of sequence number related fields to long. Before starting to work on this I wanted to make sure that this change is acceptable.

Thanks,
AsankaK

From: Christoph John [mailto:chr...@ma...]
Sent: Thursday, August 08, 2019 2:13 PM
To: Kodippili Arachchige, Asanka <as...@ls...>; qui...@li...; Philip Whitehouse <ph...@wh...>
Subject: Re: [Quickfixj-users] Support for UI64 data type

Wow, OK. Not indicating a limit in the FIX spec does not mean that one should use sequence numbers as large as 18446744073709551615. ;)
I just realised that you are from LSE and so are on the server side of things and should be able to influence the implementation: please think about what you are trying to achieve and whether you are using the right protocol for it.
Do you really want to do resends for possibly billions of messages? Why are there so many messages anyway? As stated earlier, maybe FIX is not the right fit for your purpose?!
I can only recommend that you consult someone with FIX knowledge before building your system.

Apart from that: my C++ skills are a little rusty but isn't an integer limited to 4bytes in C++ also? The implementations for C++ and .NET are also using an int for sequence numbers.
Probably it wouldn't be such a big effort to change the sequence number to long datatype. But still I think it is not really necessary due to my points stated above.
Of course I am ready to discuss. Maybe you can outline the system that you want to build. I am sure people on this mailing list can give some advice.

Cheers,
Chris.

On 08.08.19 09:02, Kodippili Arachchige, Asanka wrote:

Hi,

The developer who is implementing the C++ server side ( FIX GW) of this change had posted a question about the maximum value allowed for the field according to FIX spec, and the reply he got was that there is no limit.

https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507<https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507>



Sending the value as a string throws a NumberFormatException



201908220122431.952|ERROR|QFJ Timer|quickfix.SocketInitiator:|     |Error during timer processing

quickfix.FieldException: invalid integral value: 18446744073709551615: java.lang.NumberFormatException: For input string: "18446744073709551615"

  at quickfix.FieldMap.newIncorrectDataException(FieldMap.java:401)

  at quickfix.FieldMap.getInt(FieldMap.java:255)

  at quickfix.Session.sendRaw(Session.java:2500)

  at quickfix.Session.generateLogon(Session.java:1926)

  at quickfix.Session.next(Session.java:1826)

  at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:309)



Thanks,

AsankaK



-----Original Message-----

From: Philip Whitehouse [mailto:ph...@wh...]

Sent: Wednesday, August 07, 2019 2:07 PM

To: qui...@li...<mailto:qui...@li...>

Subject: Re: [Quickfixj-users] Support for UI64 data type



*** THIS EMAIL ORIGINATED FROM OUTSIDE OF THE ORGANISATION *** QuickFIX/J Documentation: http://www.quickfixj.org/documentation/<http://www.quickfixj.org/documentation/>

QuickFIX/J Support: http://www.quickfixj.org/support/<http://www.quickfixj.org/support/>



------------------------------------------------------------------------------------------------------------



Please read these warnings and restrictions:



This e-mail transmission is strictly confidential and intended solely for the ordinary user of the e-mail address to which it was addressed. It may contain legally privileged and/or CONFIDENTIAL information.



The unauthorised use, disclosure, distribution and/or copying of this e-mail or any information it contains is prohibited and could, in certain circumstances, constitute a criminal offence.



If you have received this e-mail in error or are not an intended recipient please inform the London Stock Exchange immediately by return e-mail or telephone 020 7797 1000.



LSEG may collect, process and retain your personal information for its business purposes. For more information please see our Privacy Policy.



We advise that in keeping with good computing practice the recipient of this e-mail should ensure that it is virus free. We do not accept responsibility for any virus that may be transferred by way of this e-mail.



E-mail may be susceptible to data corruption, interception and unauthorised amendment, and we do not accept liability for any such corruption, interception or amendment or any consequences thereof.



Calls to the London Stock Exchange may be recorded to enable the Exchange to carry out its regulatory responsibilities.



London Stock Exchange plc

10 Paternoster Square

London

EC4M 7LS

Registered in England and Wales No 2075721



------------------------------------------------------------------------------------------------------------





--

Christoph John

Software Engineering

T +49 241 557080-28

chr...@ma...<mailto:chr...@ma...>



MACD GmbH

Oppenhoffallee 103

52066 Aachen, Germany

www.macd.com<http://www.macd.com>



Amtsgericht Aachen: HRB 8151

Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald

Re: [Quickfixj-users] Support for UI64 data type

[Christoph J. <chr...@ma...>]

Wow, OK. Not indicating a limit in the FIX spec does not mean that one should use sequence numbers 
as large as 18446744073709551615. ;)
I just realised that you are from LSE and so are on the server side of things and should be able to 
influence the implementation: please think about what you are trying to achieve and whether you are 
using the right protocol for it.
Do you really want to do resends for possibly billions of messages? Why are there so many messages 
anyway? As stated earlier, maybe FIX is not the right fit for your purpose?!
I can only recommend that you consult someone with FIX knowledge before building your system.

Apart from that: my C++ skills are a little rusty but isn't an integer limited to 4bytes in C++ 
also? The implementations for C++ and .NET are also using an int for sequence numbers.
Probably it wouldn't be such a big effort to change the sequence number to long datatype. But still 
I think it is not really necessary due to my points stated above.
Of course I am ready to discuss. Maybe you can outline the system that you want to build. I am sure 
people on this mailing list can give some advice.

Cheers,
Chris.


On 08.08.19 09:02, Kodippili Arachchige, Asanka wrote:
> Hi,
> The developer who is implementing the C++ server side ( FIX GW) of this change had posted a question about the maximum value allowed for the field according to FIX spec, and the reply he got was that there is no limit.
> https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507
>
> Sending the value as a string throws a NumberFormatException
>
> 201908220122431.952|ERROR|QFJ Timer|quickfix.SocketInitiator:|     |Error during timer processing
> quickfix.FieldException: invalid integral value: 18446744073709551615: java.lang.NumberFormatException: For input string: "18446744073709551615"
> 	at quickfix.FieldMap.newIncorrectDataException(FieldMap.java:401)
> 	at quickfix.FieldMap.getInt(FieldMap.java:255)
> 	at quickfix.Session.sendRaw(Session.java:2500)
> 	at quickfix.Session.generateLogon(Session.java:1926)
> 	at quickfix.Session.next(Session.java:1826)
> 	at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:309)
> 	
> Thanks,
> AsankaK
>
> -----Original Message-----
> From: Philip Whitehouse [mailto:ph...@wh...]
> Sent: Wednesday, August 07, 2019 2:07 PM
> To: qui...@li...
> Subject: Re: [Quickfixj-users] Support for UI64 data type
>
> *** THIS EMAIL ORIGINATED FROM OUTSIDE OF THE ORGANISATION *** QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
> ------------------------------------------------------------------------------------------------------------
>   
> Please read these warnings and restrictions:
>
> This e-mail transmission is strictly confidential and intended solely for the ordinary user of the e-mail address to which it was addressed. It may contain legally privileged and/or CONFIDENTIAL information.
>
> The unauthorised use, disclosure, distribution and/or copying of this e-mail or any information it contains is prohibited and could, in certain circumstances, constitute a criminal offence.
>
> If you have received this e-mail in error or are not an intended recipient please inform the London Stock Exchange immediately by return e-mail or telephone 020 7797 1000.
>
> LSEG may collect, process and retain your personal information for its business purposes. For more information please see our Privacy Policy.
>
> We advise that in keeping with good computing practice the recipient of this e-mail should ensure that it is virus free. We do not accept responsibility for any virus that may be transferred by way of this e-mail.
>
> E-mail may be susceptible to data corruption, interception and unauthorised amendment, and we do not accept liability for any such corruption, interception or amendment or any consequences thereof.
>
> Calls to the London Stock Exchange may be recorded to enable the Exchange to carry out its regulatory responsibilities.
>
> London Stock Exchange plc
> 10 Paternoster Square
> London
> EC4M 7LS
> Registered in England and Wales No 2075721
>   
> ------------------------------------------------------------------------------------------------------------
>

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] Support for UI64 data type

[Kodippili A. A. <as...@ls...>]

Hi,
The developer who is implementing the C++ server side ( FIX GW) of this change had posted a question about the maximum value allowed for the field according to FIX spec, and the reply he got was that there is no limit. 
https://forum.fixtrading.org/t/what-is-the-largest-allowed-value-for-seqnumber-34-unlimited-32-bit-int-64-bit-int/14507

Sending the value as a string throws a NumberFormatException

201908220122431.952|ERROR|QFJ Timer|quickfix.SocketInitiator:|     |Error during timer processing
quickfix.FieldException: invalid integral value: 18446744073709551615: java.lang.NumberFormatException: For input string: "18446744073709551615"
	at quickfix.FieldMap.newIncorrectDataException(FieldMap.java:401)
	at quickfix.FieldMap.getInt(FieldMap.java:255)
	at quickfix.Session.sendRaw(Session.java:2500)
	at quickfix.Session.generateLogon(Session.java:1926)
	at quickfix.Session.next(Session.java:1826)
	at quickfix.mina.SessionConnector$SessionTimerTask.run(SessionConnector.java:309)
	
Thanks,
AsankaK

-----Original Message-----
From: Philip Whitehouse [mailto:ph...@wh...] 
Sent: Wednesday, August 07, 2019 2:07 PM
To: qui...@li...
Subject: Re: [Quickfixj-users] Support for UI64 data type

*** THIS EMAIL ORIGINATED FROM OUTSIDE OF THE ORGANISATION *** QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

------------------------------------------------------------------------------------------------------------
 
Please read these warnings and restrictions:

This e-mail transmission is strictly confidential and intended solely for the ordinary user of the e-mail address to which it was addressed. It may contain legally privileged and/or CONFIDENTIAL information. 

The unauthorised use, disclosure, distribution and/or copying of this e-mail or any information it contains is prohibited and could, in certain circumstances, constitute a criminal offence.

If you have received this e-mail in error or are not an intended recipient please inform the London Stock Exchange immediately by return e-mail or telephone 020 7797 1000.

LSEG may collect, process and retain your personal information for its business purposes. For more information please see our Privacy Policy. 

We advise that in keeping with good computing practice the recipient of this e-mail should ensure that it is virus free. We do not accept responsibility for any virus that may be transferred by way of this e-mail. 

E-mail may be susceptible to data corruption, interception and unauthorised amendment, and we do not accept liability for any such corruption, interception or amendment or any consequences thereof. 

Calls to the London Stock Exchange may be recorded to enable the Exchange to carry out its regulatory responsibilities.

London Stock Exchange plc
10 Paternoster Square
London 
EC4M 7LS
Registered in England and Wales No 2075721 
 
------------------------------------------------------------------------------------------------------------