As i understand when i print database connections object and see something like
<_mysql.connection open to 'localhost' at 81f9154>
81f9154 - is an address in memory where database connection object located.. or a pointer.. as you wish, anyway,
i have a multithreaded multiprocess socket server, i know that mysql conn is not inherited my child process, and so
after an socket.accept() is called but before any forking i am closing connection in parent process (in fact it is in thread but i actually wait a thread when it closes a connection) then a fork is called and i reopen connection in parent process using the same thread. While in child process completly new thread and completly new connection to mysql created.
For a convenient work i did my wrapper for _mysql.so, ok now is the question.. when i set up a print in this wrapper right after a connection to mysql established..
for example:
<code>
print str(self.db) + " in " + str(os.getpid())
print str(self.query("select CONNECTION_ID()")) + " in " + str(os.getpid())
</code>
i see in terminal window..
<_mysql.connection open to 'localhost' at 81f9154> in 4936
({'CONNECTION_ID()': 229L},) in 4936
<_mysql.connection open to 'localhost' at 81f9154> in 4940
({'CONNECTION_ID()': 230L},) in 4940
Great thing CONNECTION_ID is different, but thats not a big deal i knew it from "show processlist",
but the annoyng question - sometimes address of an object is the same! As you can see in doifferent processes address of connection object is 81f9154.
What i did wrong? I`m not reimporting any modules in child process, the database connection - is an attribute of an object and is closed before fork and even the attribute itself is deleted.
# a new thread of mysql is created in req_handler object
# and a new connection is opened.
</code>
Maybe i am paranoic, as connection_id() show different but could someone explain me why memory addres is the same?
I mean this is separate processes they do not share memory.
Please help!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks! Great answer :), it is a good suggestion not to close before fork and just to open new in child, but i dont want that mess that would happen if child processes will somehow influence on a parent mysql connection, besides it is not a metter of quick starting a process it matters more a long term running.
I am using sys.exit() which i think almost the same as os._exit() and always closing connections manualy before exit even if any error happens i trace it in a thread run function and trying to quit gracefully closing all manually and flushing all info needed.
I think you a little bit misunderstood me. Conn object point to same memory right after a fork is done and connection reopened in both parent and child. Well there still may be some explanations of what it going on.
Let me tell you as it is done:
1) a fork is done.. there are no connection
2) in a parent it reopens and print address 81f9154
3) at the same time child opens a new connection print an addres of it and is also 81f9154
2 and 3 can happen parrallel - it means events can happen in many varians.. like parent opens conn before child but prints addres after.. if print statement would not be blocking in python thay can event print to terminal by a character from a process.
As i understaned your memory implementation.. you are trying to tell that it looks something like :
1) a fork is done.. there are no connection
2) 81f9154 -> points to a parent conn.. and then some memory reallocation done in parent so 81f9154 addres is freed
3) child opens a conn and it point to 81f9154 but a parent no more point to 81f9154
Thank you for answering so fast :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
sys.exit() will garbage collect all objects, which will mess up your parent's MySQL connection because the shared socket will be closed. os._exit() avoids this.
What I'm saying is, after the fork, the parent and child have the same virtual memory mapping, and the Python memory heap is in the same state (or very nearly), so if both of them allocate objects immediately after the fork, they are very likely to get the same address. However, these addresses are in different address spaces, so they don't point to the same piece of physical memory, due to the copy-on-write nature of the fork implementation.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The socket (used internally by the MySQL client library) can't be used by both the parent and the child after a fork. This is essentially the same problem with sharing a connection between threads: The parent and child will stomp on each other. I think you understand this.
You're deallocating the object, forking, and then allocating a new object. There's no reason why the memory address of the new object can't be the same as the old object; after the fork, the child has the same memory layout as the parent. What you get depends on the implementation of Python object allocation. If it's strategy is to reuse recently freed memory, then it seems likely you will get the same address in the child. So long as the connection ID changes (which it does) you should be OK.
You may want to make use of os._exit() in the child to avoid GC on the database connection from the parent, i.e. don't close/reopen the connection in the parent; just open a new connection in the child (but don't delete the old connection), and use os._exit() to avoid cleanups which would mess up the parent's connection. If you use os._exit(), you should manually close the child's connection first.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As i understand when i print database connections object and see something like
<_mysql.connection open to 'localhost' at 81f9154>
81f9154 - is an address in memory where database connection object located.. or a pointer.. as you wish, anyway,
i have a multithreaded multiprocess socket server, i know that mysql conn is not inherited my child process, and so
after an socket.accept() is called but before any forking i am closing connection in parent process (in fact it is in thread but i actually wait a thread when it closes a connection) then a fork is called and i reopen connection in parent process using the same thread. While in child process completly new thread and completly new connection to mysql created.
For a convenient work i did my wrapper for _mysql.so, ok now is the question.. when i set up a print in this wrapper right after a connection to mysql established..
for example:
<code>
print str(self.db) + " in " + str(os.getpid())
print str(self.query("select CONNECTION_ID()")) + " in " + str(os.getpid())
</code>
i see in terminal window..
<_mysql.connection open to 'localhost' at 81f9154> in 4936
({'CONNECTION_ID()': 229L},) in 4936
<_mysql.connection open to 'localhost' at 81f9154> in 4940
({'CONNECTION_ID()': 230L},) in 4940
Great thing CONNECTION_ID is different, but thats not a big deal i knew it from "show processlist",
but the annoyng question - sometimes address of an object is the same! As you can see in doifferent processes address of connection object is 81f9154.
What i did wrong? I`m not reimporting any modules in child process, the database connection - is an attribute of an object and is closed before fork and even the attribute itself is deleted.
is simple
<code>
def handle_request(...)
self.close_mysql_conn_and_delete_attribute()
pid = os.fork()
if pid:
# parent
...
self.reopen_mysql_conn()
else:
# child
request_handler = req_handler_class()
</code>
Maybe i am paranoic, as connection_id() show different but could someone explain me why memory addres is the same?
I mean this is separate processes they do not share memory.
Please help!
Thanks! Great answer :), it is a good suggestion not to close before fork and just to open new in child, but i dont want that mess that would happen if child processes will somehow influence on a parent mysql connection, besides it is not a metter of quick starting a process it matters more a long term running.
I am using sys.exit() which i think almost the same as os._exit() and always closing connections manualy before exit even if any error happens i trace it in a thread run function and trying to quit gracefully closing all manually and flushing all info needed.
I think you a little bit misunderstood me. Conn object point to same memory right after a fork is done and connection reopened in both parent and child. Well there still may be some explanations of what it going on.
Let me tell you as it is done:
1) a fork is done.. there are no connection
2) in a parent it reopens and print address 81f9154
3) at the same time child opens a new connection print an addres of it and is also 81f9154
2 and 3 can happen parrallel - it means events can happen in many varians.. like parent opens conn before child but prints addres after.. if print statement would not be blocking in python thay can event print to terminal by a character from a process.
As i understaned your memory implementation.. you are trying to tell that it looks something like :
1) a fork is done.. there are no connection
2) 81f9154 -> points to a parent conn.. and then some memory reallocation done in parent so 81f9154 addres is freed
3) child opens a conn and it point to 81f9154 but a parent no more point to 81f9154
Thank you for answering so fast :)
sys.exit() will garbage collect all objects, which will mess up your parent's MySQL connection because the shared socket will be closed. os._exit() avoids this.
What I'm saying is, after the fork, the parent and child have the same virtual memory mapping, and the Python memory heap is in the same state (or very nearly), so if both of them allocate objects immediately after the fork, they are very likely to get the same address. However, these addresses are in different address spaces, so they don't point to the same piece of physical memory, due to the copy-on-write nature of the fork implementation.
The socket (used internally by the MySQL client library) can't be used by both the parent and the child after a fork. This is essentially the same problem with sharing a connection between threads: The parent and child will stomp on each other. I think you understand this.
You're deallocating the object, forking, and then allocating a new object. There's no reason why the memory address of the new object can't be the same as the old object; after the fork, the child has the same memory layout as the parent. What you get depends on the implementation of Python object allocation. If it's strategy is to reuse recently freed memory, then it seems likely you will get the same address in the child. So long as the connection ID changes (which it does) you should be OK.
You may want to make use of os._exit() in the child to avoid GC on the database connection from the parent, i.e. don't close/reopen the connection in the parent; just open a new connection in the child (but don't delete the old connection), and use os._exit() to avoid cleanups which would mess up the parent's connection. If you use os._exit(), you should manually close the child's connection first.