sqlobject-discuss

[SQLObject] URGENT || Python 3.11, MySQL, SQLobject 3.9.1 not working in multiprocessing pool

[Nitin C. <nit...@gm...>]

Hi SQLObject forum,

I have been trying to implement a multi-processed migration job that
parallelly reads mutually exclusive batches of records from a single table
based on the record's ID and sends that data to elasticsearch.

I use the cpu_count in multiprocessing module to at run time how many cores
exists, creates same number of batches and then create a multi processed
pool based on that value of cpu_count that each one of them reads mutually
exclusive records with absolutely no shared state to read the records.But
this doesn't seem to work.

Can anyone please share if SQLObject even supports multiprocessing pools?
If yes, can someone please guide me to an example blog or some code snippet
that is able to successfully handle SQlObject MySQL connection in
multiprocessing pools?

Looking forward to hearing back from you.

Best Regards,
Nitin Chauhan

Re: [SQLObject] SQLobject doesn't work in multiprocessing

[Oleg B. <ph...@ph...>]

Hello!

On Tue, Apr 02, 2024 at 09:37:51AM -0700, Nitin Chauhan <nit...@gm...> wrote:
> Can anyone please share if SQLObject even supports multiprocessing pools?

   I'm sure it doesn't. Every process must open its own connection(s).
For many backends even every thread must open per-thread connection.

> Best Regards,
> Nitin Chauhan

Oleg.
-- 
    Oleg Broytman            https://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] SQLobject doesn't work in multiprocessing

[Nitin C. <nit...@gm...>]

Thanks for the reply Oleg,

Can you please let me know how I can make multiple separate MySQL sqlobject
connections work with each process in a multiprocessing pool?

On Tue, Apr 2, 2024 at 10:10 AM Oleg Broytman <ph...@ph...> wrote:

> Hello!
>
> On Tue, Apr 02, 2024 at 09:37:51AM -0700, Nitin Chauhan <
> nit...@gm...> wrote:
> > Can anyone please share if SQLObject even supports multiprocessing pools?
>
>    I'm sure it doesn't. Every process must open its own connection(s).
> For many backends even every thread must open per-thread connection.
>
> > Best Regards,
> > Nitin Chauhan
>
> Oleg.
> --
>     Oleg Broytman            https://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
>
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>

Re: [SQLObject] SQLobject doesn't work in multiprocessing

[Oleg B. <ph...@ph...>]

Hi!

On Tue, Apr 09, 2024 at 10:09:44AM -0700, Nitin Chauhan <nit...@gm...> wrote:
> Can you please let me know how I can make multiple separate MySQL sqlobject
> connections work with each process in a multiprocessing pool?

   I do not understand the question. Every process just opens a new
connection. Starts a transaction, performs operation(s), commits, closes
the transaction. Before exit closes the connection.

Oleg.
-- 
    Oleg Broytman            https://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.