quickfixj-users

[Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Christoph J. <chr...@ma...>]

Hi,

does anyone have the JDBC MessageStore in use with a larger number of sessions? Let's say 50+ sessions?
I currently struggle to use it because the connection pooling lib Proxool (which QFJ uses) is not 
maintained anymore and if I ran into issues then probably there would be no bug fixes.

Probably it would be not such a big deal to replace it with something better, e.g. HikariCP...?
Did someone already replace Proxool with something different?

Feedback is highly appreciated.

There also is https://github.com/quickfix-j/quickfixj/discussions/373 if you want to leave comments 
on Github.

Thanks,
Chris.

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Colin D. <co...@ma...>]

Not sure if this helps, exactly, but, we replaced the JDBC message store 
with a Hibernate-based one that's fronted by the in-memory message store.

It's fundamentally Spring-based, so allows you to swap in your own pool. 
We've used Hikari and C3P0 successfully.

Here are the pieces:

https://github.com/Marketcetera/marketcetera/blob/master/fix/fix-core/src/main/java/org/marketcetera/fix/store/


On 2/22/21 3:24 AM, Christoph John via Quickfixj-users wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi,
>
> does anyone have the JDBC MessageStore in use with a larger number of 
> sessions? Let's say 50+ sessions?
> I currently struggle to use it because the connection pooling lib 
> Proxool (which QFJ uses) is not maintained anymore and if I ran into 
> issues then probably there would be no bug fixes.
>
> Probably it would be not such a big deal to replace it with something 
> better, e.g. HikariCP...?
> Did someone already replace Proxool with something different?
>
> Feedback is highly appreciated.
>
> There also is https://github.com/quickfix-j/quickfixj/discussions/373 
> if you want to leave comments on Github.
>
> Thanks,
> Chris.
>
-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884
https://www.marketcetera.com

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Christoph J. <chr...@ma...>]

Hi Colin,

thanks for sharing this, it was interesting to see how you implemented the store.

Cheers,
Chris.

On 22.02.21 17:24, Colin DuPlantis wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Not sure if this helps, exactly, but, we replaced the JDBC message store with a Hibernate-based 
> one that's fronted by the in-memory message store.
>
> It's fundamentally Spring-based, so allows you to swap in your own pool. We've used Hikari and 
> C3P0 successfully.
>
> Here are the pieces:
>
> https://github.com/Marketcetera/marketcetera/blob/master/fix/fix-core/src/main/java/org/marketcetera/fix/store/ 
>
>
>
> On 2/22/21 3:24 AM, Christoph John via Quickfixj-users wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi,
>>
>> does anyone have the JDBC MessageStore in use with a larger number of sessions? Let's say 50+ 
>> sessions?
>> I currently struggle to use it because the connection pooling lib Proxool (which QFJ uses) is not 
>> maintained anymore and if I ran into issues then probably there would be no bug fixes.
>>
>> Probably it would be not such a big deal to replace it with something better, e.g. HikariCP...?
>> Did someone already replace Proxool with something different?
>>
>> Feedback is highly appreciated.
>>
>> There also is https://github.com/quickfix-j/quickfixj/discussions/373 if you want to leave 
>> comments on Github.
>>
>> Thanks,
>> Chris.
>>

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Ajay P. <ap...@en...>]

Yes Chris
we use c3p0. https://www.mchange.com/projects/c3p0/
We configure it in our code and manually set it into the factories where we
need it
((JdbcStoreFactory)storeFactory).setDataSource(ourDatasource);
((JdbcLogFactory)jdbcLogFactory).setDataSource(ourDatasource);

We manually run over 500 sessions spread across about 10 processes in prod
between staging and dropcopy connections and have had no issues.


Ajay Patwardhan
Enfusion LLC
125 South Clark Street, Suite 750, Chicago, IL 60603
Main  : 312-253-9800
Direct: 312-253-7659
Fax   : 312-253-9888
email : ap...@en...


On Mon, Feb 22, 2021 at 5:25 AM Christoph John via Quickfixj-users <
qui...@li...> wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J <http://www.quickfixj.org/documentation/QuickFIX/J> Support:
> http://www.quickfixj.org/support/
>
>
> Hi,
>
> does anyone have the JDBC MessageStore in use with a larger number of
> sessions? Let's say 50+ sessions?
> I currently struggle to use it because the connection pooling lib Proxool
> (which QFJ uses) is not
> maintained anymore and if I ran into issues then probably there would be
> no bug fixes.
>
> Probably it would be not such a big deal to replace it with something
> better, e.g. HikariCP...?
> Did someone already replace Proxool with something different?
>
> Feedback is highly appreciated.
>
> There also is https://github.com/quickfix-j/quickfixj/discussions/373 if
> you want to leave comments
> on Github.
>
> Thanks,
> Chris.
>
> --
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma...
>
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com
>
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
>
>
>
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>

-- 
 
<https://www.linkedin.com/uas/login?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenfusion-systems-llc%2Fmycompany> 
 <https://twitter.com/enfusion>
Agility-on-demand. What is it as applied to 
middle and back office operations, when do you need it, and how can you get 
it? Read more here 
<https://www.enfusion.com/how-agile-is-your-asset-management-team/> and 
follow us for future insights.

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Christoph J. <chr...@ma...>]

Hi Ajay,

thanks for this, I totally forgot that it is quite simple to override the built-in Jdbc stuff. Been 
using FileStore most of the time...

Cheers,
Chris.


On 22.02.21 17:05, Ajay Patwardhan wrote:
> Yes Chris
> we use c3p0. https://www.mchange.com/projects/c3p0/ <https://www.mchange.com/projects/c3p0/>
> We configure it in our code and manually set it into the factories where we need it
> ((JdbcStoreFactory)storeFactory).setDataSource(ourDatasource);
> ((JdbcLogFactory)jdbcLogFactory).setDataSource(ourDatasource);
>
> We manually run over 500 sessions spread across about 10 processes in prod between staging and 
> dropcopy connections and have had no issues.
>
>
> Ajay Patwardhan
> Enfusion LLC
> 125 South Clark Street, Suite 750, Chicago, IL 60603
> Main  : 312-253-9800
> Direct: 312-253-7659
> Fax   : 312-253-9888
> email : ap...@en... <mailto:ap...@en...>
>
>
> On Mon, Feb 22, 2021 at 5:25 AM Christoph John via Quickfixj-users 
> <qui...@li... <mailto:qui...@li...>> wrote:
>
>     QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>     QuickFIX/J <http://www.quickfixj.org/documentation/QuickFIX/J> Support:
>     http://www.quickfixj.org/support/ <http://www.quickfixj.org/support/>
>
>
>     Hi,
>
>     does anyone have the JDBC MessageStore in use with a larger number of sessions? Let's say 50+
>     sessions?
>     I currently struggle to use it because the connection pooling lib Proxool (which QFJ uses) is not
>     maintained anymore and if I ran into issues then probably there would be no bug fixes.
>
>     Probably it would be not such a big deal to replace it with something better, e.g. HikariCP...?
>     Did someone already replace Proxool with something different?
>
>     Feedback is highly appreciated.
>
>     There also is https://github.com/quickfix-j/quickfixj/discussions/373
>     <https://github.com/quickfix-j/quickfixj/discussions/373> if you want to leave comments
>     on Github.
>
>     Thanks,
>     Chris.
>
>     -- 
>     Christoph John
>     Software Engineering
>     T +49 241 557080-28
>     chr...@ma... <mailto:chr...@ma...>
>
>     MACD GmbH
>     Oppenhoffallee 103
>     52066 Aachen, Germany
>     www.macd.com <http://www.macd.com>
>
>     Amtsgericht Aachen: HRB 8151
>     Ust.-Id: DE 813021663
>     Geschäftsführer: George Macdonald
>
>
>
>     _______________________________________________
>     Quickfixj-users mailing list
>     Qui...@li... <mailto:Qui...@li...>
>     https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>     <https://lists.sourceforge.net/lists/listinfo/quickfixj-users>
>
>
> <https://www.linkedin.com/uas/login?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenfusion-systems-llc%2Fmycompany> 
> <https://twitter.com/enfusion>
> _Agility-on-demand. What is it as applied to middle and back office operations, when do you need 
> it, and how can you get it? Read more here 
> <https://www.enfusion.com/how-agile-is-your-asset-management-team/> and follow us for future 
> insights. 

-- 
Christoph John
Software Engineering
T +49 241 557080-28
chr...@ma...

MACD GmbH
Oppenhoffallee 103
52066 Aachen, Germany
www.macd.com

Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663
Geschäftsführer: George Macdonald

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Robert N. <rob...@gm...>]

I have a related question but is there anything that allows you to batch these since the current implementation will perform roundtrip per message and that is nowhere near optimal.

ie. think that you have N sessions configured and you want to keep the state in sync separate with one round trip to the database covering multiple sessions.

Is there anything where it will fire on a delayed timer for example where you sacrifice not being completely in sync at all times but don’t have to round trip to the database for every single message?

"We manually run over 500 sessions spread across about 10 processes in prod between staging and dropcopy connections and have had no issues.”

Is that a network write or a database close to where the FIX client is?

> On Feb 22, 2021, at 10:05 AM, Ajay Patwardhan <ap...@en...> wrote:
> 
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
> 
> 
> Yes Chris
> we use c3p0. https://www.mchange.com/projects/c3p0/ <https://www.mchange.com/projects/c3p0/>
> We configure it in our code and manually set it into the factories where we need it
> ((JdbcStoreFactory)storeFactory).setDataSource(ourDatasource);
> ((JdbcLogFactory)jdbcLogFactory).setDataSource(ourDatasource); 
> 
> We manually run over 500 sessions spread across about 10 processes in prod between staging and dropcopy connections and have had no issues.
> 
> 
> Ajay Patwardhan
> Enfusion LLC
> 125 South Clark Street, Suite 750, Chicago, IL 60603
> Main  : 312-253-9800
> Direct: 312-253-7659
> Fax   : 312-253-9888
> email : ap...@en... <mailto:ap...@en...>
> 
> On Mon, Feb 22, 2021 at 5:25 AM Christoph John via Quickfixj-users <qui...@li... <mailto:qui...@li...>> wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J <http://www.quickfixj.org/documentation/QuickFIX/J> Support: http://www.quickfixj.org/support/ <http://www.quickfixj.org/support/>
> 
> 
> Hi,
> 
> does anyone have the JDBC MessageStore in use with a larger number of sessions? Let's say 50+ sessions?
> I currently struggle to use it because the connection pooling lib Proxool (which QFJ uses) is not 
> maintained anymore and if I ran into issues then probably there would be no bug fixes.
> 
> Probably it would be not such a big deal to replace it with something better, e.g. HikariCP...?
> Did someone already replace Proxool with something different?
> 
> Feedback is highly appreciated.
> 
> There also is https://github.com/quickfix-j/quickfixj/discussions/373 <https://github.com/quickfix-j/quickfixj/discussions/373> if you want to leave comments 
> on Github.
> 
> Thanks,
> Chris.
> 
> -- 
> Christoph John
> Software Engineering
> T +49 241 557080-28
> chr...@ma... <mailto:chr...@ma...>
> 
> MACD GmbH
> Oppenhoffallee 103
> 52066 Aachen, Germany
> www.macd.com <http://www.macd.com/>
> 
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
> Geschäftsführer: George Macdonald
> 
> 
> 
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li... <mailto:Qui...@li...>
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users <https://lists.sourceforge.net/lists/listinfo/quickfixj-users>
> 
>  <https://www.linkedin.com/uas/login?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenfusion-systems-llc%2Fmycompany>  <https://twitter.com/enfusion>Agility-on-demand. What is it as applied to middle and back office operations, when do you need it, and how can you get it? Read more here <https://www.enfusion.com/how-agile-is-your-asset-management-team/> and follow us for future insights._______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

Re: [Quickfixj-users] JDBC MessageStore with several number of sessions / connection pooling library

[Colin D. <co...@ma...>]

We do something like this. We have a custom Hibernate message store that 
batches inserts and is fronted by the off-the-shelf in-memory message 
store. While this addresses the performance needs, there's clearly a 
potential consistency problem in case of unexpected shutdown. We address 
this on session creating by looking for a discrepancy between what the 
message store says is the last message and what we show as the last 
message successfully processes by the business system.

Our message store is open-source, so you'd be welcome to have a go at it 
if you want more info.

On 3/2/21 8:05 AM, Robert Nicholson wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
> I have a related question but is there anything that allows you to 
> batch these since the current implementation will perform roundtrip 
> per message and that is nowhere near optimal.
>
> ie. think that you have N sessions configured and you want to keep the 
> state in sync separate with one round trip to the database covering 
> multiple sessions.
>
> Is there anything where it will fire on a delayed timer for example 
> where you sacrifice not being completely in sync at all times but 
> don’t have to round trip to the database for every single message?
>
> "We manually run over 500 sessions spread across about 10 processes in 
> prod between staging and dropcopy connections and have had no issues.”
>
> Is that a network write or a database close to where the FIX client is?
>
>> On Feb 22, 2021, at 10:05 AM, Ajay Patwardhan 
>> <ap...@en... <mailto:ap...@en...>> wrote:
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ 
>> <http://www.quickfixj.org/documentation/>
>> QuickFIX/J Support: http://www.quickfixj.org/support/ 
>> <http://www.quickfixj.org/support/>
>>
>>
>> Yes Chris
>> we use c3p0. https://www.mchange.com/projects/c3p0/ 
>> <https://www.mchange.com/projects/c3p0/>
>> We configure it in our code and manually set it into the factories 
>> where we need it
>> ((JdbcStoreFactory)storeFactory).setDataSource(ourDatasource);
>> ((JdbcLogFactory)jdbcLogFactory).setDataSource(ourDatasource);
>>
>> We manually run over 500 sessions spread across about 10 processes in 
>> prod between staging and dropcopy connections and have had no issues.
>>
>>
>> Ajay Patwardhan
>> Enfusion LLC
>> 125 South Clark Street, Suite 750, Chicago, IL 60603
>> Main  : 312-253-9800
>> Direct: 312-253-7659
>> Fax   : 312-253-9888
>> email : ap...@en... <mailto:ap...@en...>
>>
>>
>> On Mon, Feb 22, 2021 at 5:25 AM Christoph John via Quickfixj-users 
>> <qui...@li... 
>> <mailto:qui...@li...>> wrote:
>>
>>     QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>     QuickFIX/J <http://www.quickfixj.org/documentation/QuickFIX/J>
>>     Support: http://www.quickfixj.org/support/
>>     <http://www.quickfixj.org/support/>
>>
>>
>>     Hi,
>>
>>     does anyone have the JDBC MessageStore in use with a larger
>>     number of sessions? Let's say 50+ sessions?
>>     I currently struggle to use it because the connection pooling lib
>>     Proxool (which QFJ uses) is not
>>     maintained anymore and if I ran into issues then probably there
>>     would be no bug fixes.
>>
>>     Probably it would be not such a big deal to replace it with
>>     something better, e.g. HikariCP...?
>>     Did someone already replace Proxool with something different?
>>
>>     Feedback is highly appreciated.
>>
>>     There also is
>>     https://github.com/quickfix-j/quickfixj/discussions/373
>>     <https://github.com/quickfix-j/quickfixj/discussions/373> if you
>>     want to leave comments
>>     on Github.
>>
>>     Thanks,
>>     Chris.
>>
>>     -- 
>>     Christoph John
>>     Software Engineering
>>     T +49 241 557080-28
>>     chr...@ma... <mailto:chr...@ma...>
>>
>>     MACD GmbH
>>     Oppenhoffallee 103
>>     52066 Aachen, Germany
>>     www.macd.com <http://www.macd.com/>
>>
>>     Amtsgericht Aachen: HRB 8151
>>     Ust.-Id: DE 813021663
>>     Geschäftsführer: George Macdonald
>>
>>
>>
>>     _______________________________________________
>>     Quickfixj-users mailing list
>>     Qui...@li...
>>     <mailto:Qui...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>     <https://lists.sourceforge.net/lists/listinfo/quickfixj-users>
>>
>>
>> <https://www.linkedin.com/uas/login?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenfusion-systems-llc%2Fmycompany> 
>> <https://twitter.com/enfusion>
>> _Agility-on-demand. What is it as applied to middle and back office 
>> operations, when do you need it, and how can you get it? Read more 
>> here 
>> <https://www.enfusion.com/how-agile-is-your-asset-management-team/> 
>> and follow us for future insights. 
>> _______________________________________________
>> Quickfixj-users mailing list
>> Qui...@li... 
>> <mailto:Qui...@li...>
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
>
>
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884
https://www.marketcetera.com