Embedded libmysqld.dll

[George]

Is it possible to to use MySQLdb with libmysqld.dll rather than mysqld.exe?

[George]

This is the compiler output after the usual copying; this is using files from SVN tags 1.2.2b2. I also tried using the files from SVN branches 1.2 (see below)

running build_ext
building '_mysql' extension
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw32\bin\gcc.exe -mno-cygwin -mdll -O -Wall -Id:/mysql/include -ID:\Python
\include -ID:\Python\PC -c _mysql.c -o build\temp.win32-2.5\Release_mysql.o /Dv
ersion_info=(1,2,2,'beta',2) /Dversion=1.2.2b2
gcc: /Dversion_info=(1,2,2,'beta',2): No such file or directory
gcc: /Dversion=1.2.2b2: No such file or directory
In file included from _mysql.c:33:
d:/mysql/include/config-win.h:130:1: warning: "isnan" redefined
In file included from D:/Python/include/pyport.h:200,
from D:/Python/include/Python.h:57,
from pymemcompat.h:9,
from _mysql.c:29:
d:/mingw32/bin/../lib/gcc/mingw32/3.4.2/../../../../include/math.h:380:1: warnin
g: this is the location of the previous definition
In file included from _mysql.c:33:
d:/mysql/include/config-win.h:140:1: warning: "SIZEOF_OFF_T" redefined
In file included from D:/Python/include/Python.h:8,
from pymemcompat.h:9,
from _mysql.c:29:
D:/Python/include/pyconfig.h:312:1: warning: this is the location of the previou
s definition
In file included from _mysql.c:33:
d:/mysql/include/config-win.h:171:1: warning: "finite" redefined
d:/mysql/include/config-win.h:131:1: warning: this is the location of the previo
us definition
d:/mysql/include/config-win.h:284:1: warning: "HAVE_STDDEF_H" redefined
In file included from D:/Python/include/Python.h:8,
from pymemcompat.h:9,
from _mysql.c:29:
D:/Python/include/pyconfig.h:605:1: warning: this is the location of the previou
s definition
error: command 'gcc' failed with exit status 1

The compiler output from branch 1.2 gives the above plus alot like the following:

D:/Python/include/pyconfig.h:605:1: warning: this is the location of the previou
s definition
writing build\temp.win32-2.5\Release_mysql.def
d:\mingw32\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release_mysq
l.o build\temp.win32-2.5\Release_mysql.def -Ld:/mysql/lib/opt -LD:\Python\libs
-LD:\Python\PCBuild -lmysqlclient -lzlib -lmsvcrt -lwsock32 -ladvapi32 -lpython2
5 -lmsvcr71 -o build\lib.win32-2.5_mysql.pyd
d:/mysql/lib/opt\mysqlclient.lib(./release/my_thr_init.obj)(.text[_my_thread_glo
bal_init]+0x1e):C:\cygwin\home\mys: variable '_iob' can't be auto-imported. Plea
se read the documentation for ld's --enable-auto-import for details.
d:/mysql/lib/opt\mysqlclient.lib(./release/default.obj)(.text[_load_defaults]+0x
216):C:\cygwin\home\mys: variable '_iob' can't be auto-imported. Please read the
documentation for ld's --enable-auto-import for details.

[Andy Dustman]

Looks like you are using mingw/gcc and getting Windows C compiler-style switches, which won't work at all. For now you'll have to hack setup.py a bit so it thinks/knows that it is in a POSIX environment.

[George]

I am using mingw in a Windows XP operating system.
I changed setup.py like this:

if os.name == "posix":

flag_prefix = "-"

else: # assume windows

flag_prefix = "/"

and I get lots or warnings ending with this:

Warning: .drectve /DEFAULTLIB:"LIBCMT" /DEFAULTLIB:"OLDNAMES" ' unrecognized Warning: .drectve/DEFAULTLIB:"LIBCMT" /DEFAULTLIB:"OLDNAMES" ' unrecognized
Warning: .drectve /DEFAULTLIB:"LIBCMT" /DEFAULTLIB:"OLDNAMES" ' unrecognized Warning: .drectve/DEFAULTLIB:"LIBCMT" /DEFAULTLIB:"OLDNAMES" ' unrecognized
d:/mysql/lib/opt\mysqlclient.lib(./release/default.obj)(.text[_search_default_fi
le_with_extInfo: resolving iob by linking to imp___iob (auto-import)
]+0x6):C:\cygwin\home\mys: undefined reference to _chkstk' nmth000000.o(.idata$4+0x0): undefined reference to_nm___iob'
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

[Andy Dustman]

There's something screwy with your Python setup. I bet you can't build anything with distutils if it has a C component.

[George]

Is there something that I could show you that would help explain the problem? Do the source files have to be in a certain place in the Python folder tree? I have the source files in an isolated folder with a path set to point to python, mysql, and mingw. This worked with distutils when there was no C involved, but your comment makes me wonder if the source needs to be in a certain place for this build to work.

[Andy Dustman]

mingw/gcc uses POSIX/UNIX-style command-line switches, but something is causing distutils to use Windows/DOS-style command-line switches. I would guess this means you are compiling against the Windows Python and not the Cygwin Python. Maybe you meant to do this, but I don't think it'll work. You'll need to use the Cygwin Python and also MySQL for Cygwin.

[George]

I tried a few more things with minGW32, but it still wouldn't work. Rather than compiling Python and MySQL with minGW32, I decided to go back to Python24 and MySQL 4.0.24 and use David Wood's binary for MySQLdb embedded. I have a statement to import _mysql and Statements like this to get mysql going:
_mysql.server_start()
con = MySQLdb.connect(user="root")
error message is that _mysql doesn't have a 'server_start' attribute.
I also tried _mysql.server.init() which doesn't give the error, but doesn't start the server either. How do I get this embedded server going?

[David Woods]

Here's what I do in Transana, obviously extracted from the code and simplified a bit. In this code, I check the max_allowed_packets variable and increase its size if necessary, but the query could've been anything.

import MySQLdb

databasePath = TransanaGlobal.configData.databaseDir
datadir = "--datadir=" + databasePath
lang = '--language=./share/english' # The MySQL Share folders and files must be available!
MySQLdb.server_init(args=['Transana', datadir, '--basedir=.', lang])

_dbref = MySQLdb.connect(use_unicode=True)

query = "SHOW VARIABLES LIKE 'max_allowed_packe%'"
dbCursor = _dbref.cursor()
dbCursor.execute(query)
for pair in dbCursor.fetchall():
if pair[0] == 'max_allowed_packet':
# Its value comes in different forms depending on what version of MySQL and MySQL for Python we're using.
if type(pair[1]) == array.array:
max_allowed_packet = pair[1].tostring()
else:
max_allowed_packet = pair[1]
if int(max_allowed_packet) < 8388608:
dbCursor.execute('SET GLOBAL max_allowed_packet=8388608')
dbCursor.close()
_dbref.close()
_dbref = None

MySQLdb.server_end()

See Transana's DBInterface.py module (available from our CVS Repository) if you want all the gory details, but this should be sufficient. You probably don't need the array.array check -- you'll probably always get that, but my code has to handle several different MySQL for Python versions on either side of the changes that necessitated that approach to the queries.

David

[George]

Thank you David. I have it up and running which is a great relief.

I am using LOAD DATA LOCAL INFILE which is disabled in the embedded server. If anyone knows how to enable it please let me know. Otherwise I can get the data with python and insert it in the tables.
Robert

[Andy Dustman]

Actually, you can use MySQLdb with libmysqld instead of libmysqlclient. You need to edit site.cfg (set embedded=True) and rebuild. On Windows, you may have to add libmysqld (if that is the correct name on Windows) to libraries in site.cfg and maybe hack setup.py. AFAIK, no one has really tried it on Windows before.

[David Woods]

I got it working on Windows and Mac, but it was quite a long time ago using MySQL 4.0.x on Windows (due to problems with libmysqld.dll in MySQL 4.1.x) and MySQL 4.1.x on Mac OS X, along with MySQL for Python version 1.2.0.

A description of how I did it on Windows is at http://www.transana.org/developers/setup/MySQLPython_Win_SU.htm. A description of what I did on Mac OS X 10.3 is at http://www.transana.org/developers/setup/MySQLPython_Mac.htm. (Follow the instructions for the "single-user version" as that's the version that uses embedded MySQL.)

I'd be interested to know if you can get things working with more recent versions of MySQL and MySQL for Python, and if so, how.

David

[George]

Before tackling embedded, I need to get the compiler to work. I am using the current version of minGW, Python 2.5, and MySQLdb 1.2.2b2.
I edited the compiler section of site.cfg:

[compiler]
mysql_root: D:/mysql
library_dirs: %(mysql_root)s/lib
include_dirs: %(mysql_root)s/include
libraries: mysqlclient zlib msvcrt libcmt
wsock32 advapi32

extra_compile_args:

extra_objects:

The build output looks like this:

D:\mingw32\bin\Src>D:\Python\python setup.py build --compiler=mingw32
D:\Python\lib\distutils\extension.py:133: UserWarning: Unknown Extension options
: 'mysql_root'
warnings.warn(msg)
running build
running build_py
creating build
creating build\lib.win32-2.5
copying _mysql_exceptions.py -> build\lib.win32-2.5
creating build\lib.win32-2.5\MySQLdb
copying MySQLdb__init__.py -> build\lib.win32-2.5\MySQLdb

...normal copying and then:
running build_ext
building '_mysql' extension
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw32\bin\gcc.exe -mno-cygwin -mdll -O -Wall -ID:/mysql/include -ID:\Python
\include -ID:\Python\PC -c _mysql.c -o build\temp.win32-2.5\Release_mysql.o /Dv
ersion_info=(1,2,2,'beta',2) /Dversion=1.2.2b2
gcc: /Dversion_info=(1,2,2,'beta',2): No such file or directory
gcc: /Dversion=1.2.2b2: No such file or directory
_mysql.c:5: error: syntax error before '<' token
_mysql.c:9: error: stray '@' in program
_mysql.c:11:73: too many decimal points in number
_mysql.c:33:9: too many decimal points in number

...more similar errors ending with:

_mysql.c:4904: warning: data definition has no type or storage class
_mysql.c:4905:15: warning: extra tokens at end of #ifdef directive
_mysql.c:4905:1: unterminated #ifdef
error: command 'gcc' failed with exit status 1

Any ideas?

[Andy Dustman]

Use the current MySQLdb-1.2 branch from SVN; it has some build fixes for Windows.