Menu
▾
▴
quickfixj-users
|
From: imabhi <atm...@gm...> - 2019-08-18 11:23:37
|
We have implemented QuickFIX/J for Trade capture and fetching ICE security definitions. We have allocated 2 GB memory to our ICE Adapter which is responsible for connecting with ICE, flowing deals captured on ICE and fetching securities from ICE for various markets. When we try to fetch ICE Security Definitions for Financial Power market, ICE Adapter is going Out of Memory though we allocated 2 GB. When we increased memory allocation to 3 GB then it worked fine without going OOM. What we observe is, Quickfix engine is taking too much server memory while processing incoming definitions from ICE. If less number of definitions are coming then there is no issue. Do any one had similar issue in past ? If yes then how you fixed this. Is there any option in FIX protocol to make it multi threaded or something like that. Any information will be helpful, waiting for valuable feedback. Thanks, Abhishek Singh -- Sent from: http://quickfix-j.364392.n2.nabble.com/ |
|
From: Christoph J. <chr...@ma...> - 2019-08-18 13:11:42
|
I do not know how multithreading will improve your memory allocation?! What are you doing with the security definitions? I assume you store them in the same application that also uses QuickFIX/J? QFJ itself will not keep incoming messages in memory by itself. Cheers, Chris. Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>: >QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ >QuickFIX/J Support: http://www.quickfixj.org/support/ > > >We have implemented QuickFIX/J for Trade capture and fetching ICE >security >definitions. > >We have allocated 2 GB memory to our ICE Adapter which is responsible >for >connecting with ICE, flowing deals captured on ICE and fetching >securities >from ICE for various markets. > >When we try to fetch ICE Security Definitions for Financial Power >market, >ICE Adapter is going Out of Memory though we allocated 2 GB. >When we increased memory allocation to 3 GB then it worked fine without >going OOM. > >What we observe is, Quickfix engine is taking too much server memory >while >processing incoming definitions from ICE. If less number of definitions >are >coming then there is no issue. > >Do any one had similar issue in past ? If yes then how you fixed this. >Is >there any option in FIX protocol to make it multi threaded or something >like >that. > >Any information will be helpful, waiting for valuable feedback. > >Thanks, >Abhishek Singh > > > >-- >Sent from: http://quickfix-j.364392.n2.nabble.com/ > > >_______________________________________________ >Quickfixj-users mailing list >Qui...@li... >https://lists.sourceforge.net/lists/listinfo/quickfixj-users |
|
From: Abhishek S. <atm...@gm...> - 2019-08-18 16:04:03
|
We are saving security definitions in our DB table. It looks like QFJ keeps whole string in memory, whenever we try to fetch definitions for multiple markets ( e. g. - Oil, Power, Financial Power etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while processing incoming definitions in string format. I am using QFJ 44 (version 1.6.4). On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...> wrote: > I do not know how multithreading will improve your memory allocation?! > > What are you doing with the security definitions? I assume you store them > in the same application that also uses QuickFIX/J? QFJ itself will not keep > incoming messages in memory by itself. > > Cheers, > Chris. > > Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>: >> >> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ >> QuickFIX/J Support: http://www.quickfixj.org/support/ >> >> >> We have implemented QuickFIX/J for Trade capture and fetching ICE security >> definitions. >> >> We have allocated 2 GB memory to our ICE Adapter which is responsible for >> connecting with ICE, flowing deals captured on ICE and fetching securities >> from ICE for various markets. >> >> When we try to fetch ICE Security Definitions for Financial Power market, >> ICE Adapter is going Out of Memory though we allocated 2 GB. >> When we increased memory allocation to 3 GB then it worked fine without >> going OOM. >> >> What we observe is, Quickfix engine is taking too much server memory while >> processing incoming definitions from ICE. If less number of definitions are >> coming then there is no issue. >> >> Do any one had similar issue in past ? If yes then how you fixed this. Is >> there any option in FIX protocol to make it multi threaded or something like >> that. >> >> Any information will be helpful, waiting for valuable feedback. >> >> Thanks, >> Abhishek Singh >> >> >> >> -- >> Sent from: http://quickfix-j.364392.n2.nabble.com/ >> ------------------------------ >> Quickfixj-users mailing list >> Qui...@li... >> https://lists.sourceforge.net/lists/listinfo/quickfixj-users >> >> |
|
From: Abhishek S. <atm...@gm...> - 2019-08-18 17:51:15
|
Hi Chris, my case looks similar to this - http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894 any thought ? On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...> wrote: > We are saving security definitions in our DB table. > > It looks like QFJ keeps whole string in memory, whenever we try to fetch > definitions for multiple markets ( e. g. - Oil, Power, Financial Power > etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while > processing incoming definitions in string format. > > I am using QFJ 44 (version 1.6.4). > > On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...> > wrote: > >> I do not know how multithreading will improve your memory allocation?! >> >> What are you doing with the security definitions? I assume you store them >> in the same application that also uses QuickFIX/J? QFJ itself will not keep >> incoming messages in memory by itself. >> >> Cheers, >> Chris. >> >> Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>: >>> >>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ >>> QuickFIX/J Support: http://www.quickfixj.org/support/ >>> >>> >>> We have implemented QuickFIX/J for Trade capture and fetching ICE security >>> definitions. >>> >>> We have allocated 2 GB memory to our ICE Adapter which is responsible for >>> connecting with ICE, flowing deals captured on ICE and fetching securities >>> from ICE for various markets. >>> >>> When we try to fetch ICE Security Definitions for Financial Power market, >>> ICE Adapter is going Out of Memory though we allocated 2 GB. >>> When we increased memory allocation to 3 GB then it worked fine without >>> going OOM. >>> >>> What we observe is, Quickfix engine is taking too much server memory while >>> processing incoming definitions from ICE. If less number of definitions are >>> coming then there is no issue. >>> >>> Do any one had similar issue in past ? If yes then how you fixed this. Is >>> there any option in FIX protocol to make it multi threaded or something like >>> that. >>> >>> Any information will be helpful, waiting for valuable feedback. >>> >>> Thanks, >>> Abhishek Singh >>> >>> >>> >>> -- >>> Sent from: http://quickfix-j.364392.n2.nabble.com/ >>> ------------------------------ >>> Quickfixj-users mailing list >>> Qui...@li... >>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users >>> >>> |
|
From: Christoph J. <chr...@ma...> - 2019-08-18 18:15:23
|
That case is about sending messages, i.e. a slow consumer. I thought you are receiving the messages? However, could you try with a newer QFJ version? What does your fromApp callback look like? Cheers Chris Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh <atm...@gm...>: >Hi Chris, > >my case looks similar to this - >http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894 > >any thought ? > >On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...> >wrote: > >> We are saving security definitions in our DB table. >> >> It looks like QFJ keeps whole string in memory, whenever we try to >fetch >> definitions for multiple markets ( e. g. - Oil, Power, Financial >Power >> etc.) simultaneously, heap dump shows a lots of memory is used by QFJ >while >> processing incoming definitions in string format. >> >> I am using QFJ 44 (version 1.6.4). >> >> On Sun, Aug 18, 2019 at 6:56 PM Christoph John ><chr...@ma...> >> wrote: >> >>> I do not know how multithreading will improve your memory >allocation?! >>> >>> What are you doing with the security definitions? I assume you store >them >>> in the same application that also uses QuickFIX/J? QFJ itself will >not keep >>> incoming messages in memory by itself. >>> >>> Cheers, >>> Chris. >>> >>> Am 18. August 2019 13:23:30 MESZ schrieb imabhi ><atm...@gm...>: >>>> >>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ >>>> QuickFIX/J Support: http://www.quickfixj.org/support/ >>>> >>>> >>>> We have implemented QuickFIX/J for Trade capture and fetching ICE >security >>>> definitions. >>>> >>>> We have allocated 2 GB memory to our ICE Adapter which is >responsible for >>>> connecting with ICE, flowing deals captured on ICE and fetching >securities >>>> from ICE for various markets. >>>> >>>> When we try to fetch ICE Security Definitions for Financial Power >market, >>>> ICE Adapter is going Out of Memory though we allocated 2 GB. >>>> When we increased memory allocation to 3 GB then it worked fine >without >>>> going OOM. >>>> >>>> What we observe is, Quickfix engine is taking too much server >memory while >>>> processing incoming definitions from ICE. If less number of >definitions are >>>> coming then there is no issue. >>>> >>>> Do any one had similar issue in past ? If yes then how you fixed >this. Is >>>> there any option in FIX protocol to make it multi threaded or >something like >>>> that. >>>> >>>> Any information will be helpful, waiting for valuable feedback. >>>> >>>> Thanks, >>>> Abhishek Singh >>>> >>>> >>>> >>>> -- >>>> Sent from: http://quickfix-j.364392.n2.nabble.com/ >>>> ------------------------------ >>>> Quickfixj-users mailing list >>>> Qui...@li... >>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users >>>> >>>> |
|
From: Abhishek S. <atm...@gm...> - 2019-08-19 03:30:11
|
Yes, I am receiving messages. Which version you are referring to? Is something fixed in newer version? Didnot get your last que about fromApp callback ? On Sun, 18 Aug 2019 at 11:45 PM, Christoph John <chr...@ma...> wrote: > That case is about sending messages, i.e. a slow consumer. > I thought you are receiving the messages? However, could you try with a > newer QFJ version? > What does your fromApp callback look like? > > Cheers > Chris > > Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh < > atm...@gm...>: >> >> Hi Chris, >> >> my case looks similar to this - >> http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894 >> >> any thought ? >> >> On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm...> >> wrote: >> >>> We are saving security definitions in our DB table. >>> >>> It looks like QFJ keeps whole string in memory, whenever we try to fetch >>> definitions for multiple markets ( e. g. - Oil, Power, Financial Power >>> etc.) simultaneously, heap dump shows a lots of memory is used by QFJ while >>> processing incoming definitions in string format. >>> >>> I am using QFJ 44 (version 1.6.4). >>> >>> On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma...> >>> wrote: >>> >>>> I do not know how multithreading will improve your memory allocation?! >>>> >>>> What are you doing with the security definitions? I assume you store >>>> them in the same application that also uses QuickFIX/J? QFJ itself will not >>>> keep incoming messages in memory by itself. >>>> >>>> Cheers, >>>> Chris. >>>> >>>> Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm...>: >>>>> >>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ >>>>> QuickFIX/J Support: http://www.quickfixj.org/support/ >>>>> >>>>> >>>>> We have implemented QuickFIX/J for Trade capture and fetching ICE security >>>>> definitions. >>>>> >>>>> We have allocated 2 GB memory to our ICE Adapter which is responsible for >>>>> connecting with ICE, flowing deals captured on ICE and fetching securities >>>>> from ICE for various markets. >>>>> >>>>> When we try to fetch ICE Security Definitions for Financial Power market, >>>>> ICE Adapter is going Out of Memory though we allocated 2 GB. >>>>> When we increased memory allocation to 3 GB then it worked fine without >>>>> going OOM. >>>>> >>>>> What we observe is, Quickfix engine is taking too much server memory while >>>>> processing incoming definitions from ICE. If less number of definitions are >>>>> coming then there is no issue. >>>>> >>>>> Do any one had similar issue in past ? If yes then how you fixed this. Is >>>>> there any option in FIX protocol to make it multi threaded or something like >>>>> that. >>>>> >>>>> Any information will be helpful, waiting for valuable feedback. >>>>> >>>>> Thanks, >>>>> Abhishek Singh >>>>> >>>>> >>>>> >>>>> -- >>>>> Sent from: http://quickfix-j.364392.n2.nabble.com/ >>>>> ------------------------------ >>>>> Quickfixj-users mailing list >>>>> Qui...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users >>>>> >>>>> |
|
From: Christoph J. <chr...@ma...> - 2019-08-19 09:05:58
|
I am referring to the QuickFIX/J version. Just noticed that the version you are using is rather old. In the fromApp() callback QFJ will notify you of new application messages, i.e. your security definitions. https://github.com/quickfix-j/quickfixj#creating-a-quickfixj-application I just wanted to take a look on how you implemented that method. I cannot imagine that QFJ itself holds onto the message strings unless of course you are getting messages which are hundred megabytes in size that do not fit into the heap. Chris. On 19.08.19 05:29, Abhishek Singh wrote: > Yes, I am receiving messages. > Which version you are referring to? Is something fixed in newer version? > > Didnot get your last que about fromApp callback ? > > On Sun, 18 Aug 2019 at 11:45 PM, Christoph John <chr...@ma... > <mailto:chr...@ma...>> wrote: > > That case is about sending messages, i.e. a slow consumer. > I thought you are receiving the messages? However, could you try with a newer QFJ version? > What does your fromApp callback look like? > > Cheers > Chris > > Am 18. August 2019 19:50:53 MESZ schrieb Abhishek Singh <atm...@gm... > <mailto:atm...@gm...>>: > > Hi Chris, > > my case looks similar to this - > http://quickfix-j.364392.n2.nabble.com/Quickfix-taking-huge-memory-when-network-latency-is-high-td7579878.html#a7579894 > any thought ? > > On Sun, Aug 18, 2019 at 9:48 PM Abhishek Singh <atm...@gm... > <mailto:atm...@gm...>> wrote: > > We are saving security definitions in our DB table. > > It looks like QFJ keeps whole string in memory, whenever we try to fetch definitions > for multiple markets ( e. g. - Oil, Power, Financial Power etc.) simultaneously, heap > dump shows a lots of memory is used by QFJ while processing incoming definitions in > string format. > > I am using QFJ 44 (version > > .4). > > On Sun, Aug 18, 2019 at 6:56 PM Christoph John <chr...@ma... > <mailto:chr...@ma...>> wrote: > > I do not know how multithreading will improve your memory allocation?! > > What are you doing with the security definitions? I assume you store them in the > same application that also uses QuickFIX/J? QFJ itself will not keep incoming > messages in memory by itself. > > Cheers, > Chris. > > Am 18. August 2019 13:23:30 MESZ schrieb imabhi <atm...@gm... > <mailto:atm...@gm...>>: > > QuickFIX/J Documentation:http://www.quickfixj.org/documentation/ > QuickFIX/J Support:http://www.quickfixj.org/support/ > > > We have implemented QuickFIX/J for Trade capture and fetching ICE security > definitions. > > We have allocated 2 GB memory to our ICE Adapter which is responsible for > connecting with ICE, flowing deals captured on ICE and fetching securities > from ICE for various markets. > > When we try to fetch ICE Security Definitions for Financial Power market, > ICE Adapter is going Out of Memory though we allocated 2 GB. > When we increased memory allocation to 3 GB then it worked fine without > going OOM. > > What we observe is, Quickfix engine is taking too much server memory while > processing incoming definitions from ICE. If less number of definitions are > coming then there is no issue. > > Do any one had similar issue in past ? If yes then how you fixed this. Is > there any option in FIX protocol to make it multi threaded or something like > that. > > Any information will be helpful, waiting for valuable feedback. > > Thanks, > Abhishek Singh > > > > -- > Sent from:http://quickfix-j.364392.n2.nabble.com/ > ---------------------------------------------------------------------------------------------------- > Quickfixj-users mailing list > Qui...@li... <mailto:Qui...@li...> > https://lists.sourceforge.net/lists/listinfo/quickfixj-users > -- Christoph John Software Engineering T +49 241 557080-28 chr...@ma... MACD GmbH Oppenhoffallee 103 52066 Aachen, Germany www.macd.com Amtsgericht Aachen: HRB 8151 Ust.-Id: DE 813021663 Geschäftsführer: George Macdonald |
|
From: Colin D. <co...@ma...> - 2019-08-19 13:32:47
|
As a good test, if I were you guys, I would disable all your code in fromApp, ie, don't do anything with the messages you receive from ICE, and see if that helps your memory consumption. What that will tell you is, if the memory consumption issue improves, the problem is in your own code, not QFJ, and the problem is likely somewhere where you're storing these strings in a Map or List, etc. If the memory consumption problem does not improve, then you might be right that there's something to look at in QFJ and it would be wise to use a newer version as the next test, as Chris suggests. - Colin -- Colin DuPlantis Chief Architect, Marketcetera Download, Run, Trade 888.868.4884 https://www.marketcetera.com |
|
From: Robert N. <rob...@gm...> - 2019-08-19 13:41:10
|
So it appears you cannot add them to a database fast enough? If that’s the case then either improve that or queue them on some queue technology providing you are able to process other messages without having read all of these first. Any heap dump should tell you what you’re retaining and why you might be retaining too much. > On Aug 19, 2019, at 8:07 AM, Colin DuPlantis <co...@ma...> wrote: > > QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ > QuickFIX/J Support: http://www.quickfixj.org/support/ > > > As a good test, if I were you guys, I would disable all your code in fromApp, ie, don't do anything with the messages you receive from ICE, and see if that helps your memory consumption. > > What that will tell you is, if the memory consumption issue improves, the problem is in your own code, not QFJ, and the problem is likely somewhere where you're storing these strings in a Map or List, etc. If the memory consumption problem does not improve, then you might be right that there's something to look at in QFJ and it would be wise to use a newer version as the next test, as Chris suggests. > > - Colin > > -- > Colin DuPlantis > Chief Architect, Marketcetera > Download, Run, Trade > 888.868.4884 > https://www.marketcetera.com > > > > _______________________________________________ > Quickfixj-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfixj-users |
|
From: Abhishek S. <atm...@gm...> - 2019-08-20 08:48:42
|
Thanks for input, will try this and lets see what happens. Regards, Abhishek |
Thanks for helping keep SourceForge clean.
X