#265 memory leak related to charset='utf8'

MySQLdb-1.2
closed
MySQLdb (285)
5
2014-08-19
2008-09-23
No

I'm using MySQLdb 1.2.2, but I can repro the memory leak with today's trunk. Hence, I'm pretty sure this isn't repro-specific.

mysql Ver 14.12 Distrib 5.0.51, for apple-darwin9.2.1 (i686) using EditLine wrapper

Python 2.5.2

I'm using MacPorts on OS X 10.5, but I can repro this on Linux.

Darwin Macintosh-4.local 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386

This program leaks memory rather quickly. Removing the charset

parameter fixes it.

import MySQLdb
import sys

while True:
connection = MySQLdb.connect(
user='user', passwd='password',
host='localhost', db='development',
charset='utf8')
try:
cursor = connection.cursor()
cursor.execute(
'select * from atoms where false')
sys.stdout.write('.')
sys.stdout.flush()
finally:
connection.close()

Related

Bugs: #332

Discussion

  • This is a short script that causes a memory leak. Just run it and monitor top.

     
    Attachments
  • File Added: mysqlleak.py

     
  • Duh, it looks like this has already been fixed in trunk. I was running it wrong earlier. It definitely leaks in 1.2.2, but it does not appear to leak in trunk. Closing and marking as invalid.

     
  • Any idea what the change was in trunk that fixed this?

     
  • Markus Gritsch
    Markus Gritsch
    2009-01-09

    This program leaks memory rather quickly. Removing the charset

    parameter fixes it.

    Using the parameters

    charset='utf8', use_unicode=False

    also shows no leaking and should therefore be used as a temporary workaround for 1.2.2 as mentioned at http://www.sqlalchemy.org/trac/wiki/DatabaseNotes#MySQL

     
  • jj, are you still seeing this? I can't repro it. I've tried on trunk and 1.2br, and my python process stays the exact same size (on Fedora 10).

     
  • charset='utf8', use_unicode=False

    Markus, thanks for the tip. You're right that setting use_unicode=False fixes the memory leak. However, if I set that then MySQLdb gives me UTF-8 strings instead of unicode objects. I'm not using SQLAlchemy, so I'd really much prefer MySQLdb to give me unicode objects.

    (Note, I use SQLAlchemy when I have a little bit of very complex data, and I use naked SQL when I have a lot of very simple data. In my current project, I have only three tables, but I have 10s of gigs of data.)

     
  • jj, are you still seeing this? I can't repro it. I've tried on trunk and
    1.2br, and my python process stays the exact same size (on Fedora 10).

    Hey, Kyle!

    Does trunk actually take care of encodings? The last time I tried it, it didn't https://sourceforge.net/tracker/index.php?func=detail&aid=2127852&group_id=22307&atid=374932. Of course, that was a long time ago.

    I haven't tried 1.2br. I'll try that next.

    Thanks for the update :)

     
  • Ugh, trying to build MySQLdb on MacPorts is hard. I'm going to have a hard time testing 1.2br.

     
  • I tried out MySQL-python-1.2.3b1 on Ubuntu 8.04 with Python 2.5. It still leaks.

     
  • jj, can you give me the complete CREATE TABLE for the "atoms" table referenced by your script? I have a dummy table in my test db and it doesn't leak. I'm mostly interested in the CHARSET and ENGINE you're using.

    I've made SVN trunk kind-of work again... Some partially finished work on conversions and encoding is present, so trunk isn't that stable right now.

     
  • Sure:

    CREATE DATABASE development
    DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

    ...

    CREATE TABLE atoms (
    id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,

    active TINYINT NOT NULL DEFAULT 1,
    
    url TEXT NOT NULL,
    name VARCHAR(255),
    blurb VARCHAR(255),
    thumbnail TEXT,
    
    UNIQUE INDEX (url(255)),
    INDEX (name)
    

    ) ENGINE = INNODB;

     
  • I now have a working repro! I have some leads on the leak, but nothing definite yet. Thanks jj.

     
  • Fix several potential leaks during converter iteration.

     
  • JJ - the attached patch should fix the "holy crap it leaks like a sieve" situation. Please test it, and I'll get Andy to put this in 1.2.3b2.

     
  • Ok, the patch is applied to both 1.2br and trunk. Will mark closed once 1.2.3b2 is released.

     
  • I tried it out on 1.2br. It builds under MacPorts, which is awesome. Better yet, it no longer leaks! Two thumbs way up :-D

     
  • Hrm, thought I marked this pending for 1.2.3b2...

     
  • This Tracker item was closed automatically by the system. It was
    previously set to a Pending status, and the original submitter
    did not respond within 14 days (the time period specified by
    the administrator of this Tracker).