|
From: Nitin C. <nit...@gm...> - 2024-04-02 16:38:18
|
Hi SQLObject forum, I have been trying to implement a multi-processed migration job that parallelly reads mutually exclusive batches of records from a single table based on the record's ID and sends that data to elasticsearch. I use the cpu_count in multiprocessing module to at run time how many cores exists, creates same number of batches and then create a multi processed pool based on that value of cpu_count that each one of them reads mutually exclusive records with absolutely no shared state to read the records.But this doesn't seem to work. Can anyone please share if SQLObject even supports multiprocessing pools? If yes, can someone please guide me to an example blog or some code snippet that is able to successfully handle SQlObject MySQL connection in multiprocessing pools? Looking forward to hearing back from you. Best Regards, Nitin Chauhan |
