#265 memory leak related to charset='utf8'

[Shannon -jj Behrens]

I'm using MySQLdb 1.2.2, but I can repro the memory leak with today's trunk. Hence, I'm pretty sure this isn't repro-specific.

mysql Ver 14.12 Distrib 5.0.51, for apple-darwin9.2.1 (i686) using EditLine wrapper

Python 2.5.2

I'm using MacPorts on OS X 10.5, but I can repro this on Linux.

Darwin Macintosh-4.local 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386

This program leaks memory rather quickly. Removing the charset

parameter fixes it.

import MySQLdb
import sys

while True:
connection = MySQLdb.connect(
user='user', passwd='password',
host='localhost', db='development',
charset='utf8')
try:
cursor = connection.cursor()
cursor.execute(
'select * from atoms where false')
sys.stdout.write('.')
sys.stdout.flush()
finally:
connection.close()

[Shannon -jj Behrens]

This is a short script that causes a memory leak. Just run it and monitor top.

[Shannon -jj Behrens]

File Added: mysqlleak.py

[Shannon -jj Behrens]

Duh, it looks like this has already been fixed in trunk. I was running it wrong earlier. It definitely leaks in 1.2.2, but it does not appear to leak in trunk. Closing and marking as invalid.

[Shannon -jj Behrens]

Sorry, I'm going to reopen this; I was the one who originally closed it. It turns out that MySQLdb 1.3 on trunk doesn't actually respect charset='utf8'. See http://sourceforge.net/tracker/index.php?func=detail&aid=2127852&group_id=22307&atid=374932. Hence, it's very well possible that MySQLdb might leak memory again after that bug is fixed.

[John Eikenberry]

Any idea what the change was in trunk that fixed this?

[Markus Gritsch]

This program leaks memory rather quickly. Removing the charset

parameter fixes it.

Using the parameters

charset='utf8', use_unicode=False

also shows no leaking and should therefore be used as a temporary workaround for 1.2.2 as mentioned at http://www.sqlalchemy.org/trac/wiki/DatabaseNotes#MySQL

[Kyle VanderBeek]

jj, are you still seeing this? I can't repro it. I've tried on trunk and 1.2br, and my python process stays the exact same size (on Fedora 10).

[Shannon -jj Behrens]

charset='utf8', use_unicode=False

Markus, thanks for the tip. You're right that setting use_unicode=False fixes the memory leak. However, if I set that then MySQLdb gives me UTF-8 strings instead of unicode objects. I'm not using SQLAlchemy, so I'd really much prefer MySQLdb to give me unicode objects.

(Note, I use SQLAlchemy when I have a little bit of very complex data, and I use naked SQL when I have a lot of very simple data. In my current project, I have only three tables, but I have 10s of gigs of data.)

[Shannon -jj Behrens]

jj, are you still seeing this? I can't repro it. I've tried on trunk and
1.2br, and my python process stays the exact same size (on Fedora 10).

Hey, Kyle!

Does trunk actually take care of encodings? The last time I tried it, it didn't https://sourceforge.net/tracker/index.php?func=detail&aid=2127852&group_id=22307&atid=374932. Of course, that was a long time ago.

I haven't tried 1.2br. I'll try that next.

Thanks for the update :)

[Shannon -jj Behrens]

Ugh, trying to build MySQLdb on MacPorts is hard. I'm going to have a hard time testing 1.2br.

[Shannon -jj Behrens]

I tried out MySQL-python-1.2.3b1 on Ubuntu 8.04 with Python 2.5. It still leaks.

[Kyle VanderBeek]

jj, can you give me the complete CREATE TABLE for the "atoms" table referenced by your script? I have a dummy table in my test db and it doesn't leak. I'm mostly interested in the CHARSET and ENGINE you're using.

I've made SVN trunk kind-of work again... Some partially finished work on conversions and encoding is present, so trunk isn't that stable right now.

[Shannon -jj Behrens]

Sure:

CREATE DATABASE development
DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

...

CREATE TABLE atoms (
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,

active TINYINT NOT NULL DEFAULT 1,

url TEXT NOT NULL,
name VARCHAR(255),
blurb VARCHAR(255),
thumbnail TEXT,

UNIQUE INDEX (url(255)),
INDEX (name)

) ENGINE = INNODB;

[Kyle VanderBeek]

I now have a working repro! I have some leads on the leak, but nothing definite yet. Thanks jj.

[Kyle VanderBeek]

Fix several potential leaks during converter iteration.

[Kyle VanderBeek]

JJ - the attached patch should fix the "holy crap it leaks like a sieve" situation. Please test it, and I'll get Andy to put this in 1.2.3b2.

[Kyle VanderBeek]

Ok, the patch is applied to both 1.2br and trunk. Will mark closed once 1.2.3b2 is released.

[Shannon -jj Behrens]

I tried it out on 1.2br. It builds under MacPorts, which is awesome. Better yet, it no longer leaks! Two thumbs way up :-D

[Kyle VanderBeek]

Hrm, thought I marked this pending for 1.2.3b2...

[SourceForge Robot]

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).