#265 memory leak related to charset='utf8'

MySQLdb-1.2
closed
MySQLdb (285)
5
2014-08-19
2008-09-23
No

I'm using MySQLdb 1.2.2, but I can repro the memory leak with today's trunk. Hence, I'm pretty sure this isn't repro-specific.

mysql Ver 14.12 Distrib 5.0.51, for apple-darwin9.2.1 (i686) using EditLine wrapper

Python 2.5.2

I'm using MacPorts on OS X 10.5, but I can repro this on Linux.

Darwin Macintosh-4.local 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386

This program leaks memory rather quickly. Removing the charset

parameter fixes it.

import MySQLdb
import sys

while True:
connection = MySQLdb.connect(
user='user', passwd='password',
host='localhost', db='development',
charset='utf8')
try:
cursor = connection.cursor()
cursor.execute(
'select * from atoms where false')
sys.stdout.write('.')
sys.stdout.flush()
finally:
connection.close()

Related

Bugs: #332

Discussion

  • Shannon -jj Behrens

    This is a short script that causes a memory leak. Just run it and monitor top.

     
  • Shannon -jj Behrens

    File Added: mysqlleak.py

     
  • Shannon -jj Behrens

    Duh, it looks like this has already been fixed in trunk. I was running it wrong earlier. It definitely leaks in 1.2.2, but it does not appear to leak in trunk. Closing and marking as invalid.

     
  • John Eikenberry

    John Eikenberry - 2009-01-09

    Any idea what the change was in trunk that fixed this?

     
  • Markus Gritsch

    Markus Gritsch - 2009-01-09

    This program leaks memory rather quickly. Removing the charset

    parameter fixes it.

    Using the parameters

    charset='utf8', use_unicode=False

    also shows no leaking and should therefore be used as a temporary workaround for 1.2.2 as mentioned at http://www.sqlalchemy.org/trac/wiki/DatabaseNotes#MySQL

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-05

    jj, are you still seeing this? I can't repro it. I've tried on trunk and 1.2br, and my python process stays the exact same size (on Fedora 10).

     
  • Shannon -jj Behrens

    charset='utf8', use_unicode=False

    Markus, thanks for the tip. You're right that setting use_unicode=False fixes the memory leak. However, if I set that then MySQLdb gives me UTF-8 strings instead of unicode objects. I'm not using SQLAlchemy, so I'd really much prefer MySQLdb to give me unicode objects.

    (Note, I use SQLAlchemy when I have a little bit of very complex data, and I use naked SQL when I have a lot of very simple data. In my current project, I have only three tables, but I have 10s of gigs of data.)

     
  • Shannon -jj Behrens

    jj, are you still seeing this? I can't repro it. I've tried on trunk and
    1.2br, and my python process stays the exact same size (on Fedora 10).

    Hey, Kyle!

    Does trunk actually take care of encodings? The last time I tried it, it didn't https://sourceforge.net/tracker/index.php?func=detail&aid=2127852&group_id=22307&atid=374932. Of course, that was a long time ago.

    I haven't tried 1.2br. I'll try that next.

    Thanks for the update :)

     
  • Shannon -jj Behrens

    Ugh, trying to build MySQLdb on MacPorts is hard. I'm going to have a hard time testing 1.2br.

     
  • Shannon -jj Behrens

    I tried out MySQL-python-1.2.3b1 on Ubuntu 8.04 with Python 2.5. It still leaks.

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-17

    jj, can you give me the complete CREATE TABLE for the "atoms" table referenced by your script? I have a dummy table in my test db and it doesn't leak. I'm mostly interested in the CHARSET and ENGINE you're using.

    I've made SVN trunk kind-of work again... Some partially finished work on conversions and encoding is present, so trunk isn't that stable right now.

     
  • Shannon -jj Behrens

    Sure:

    CREATE DATABASE development
    DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

    ...

    CREATE TABLE atoms (
    id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,

    active TINYINT NOT NULL DEFAULT 1,
    
    url TEXT NOT NULL,
    name VARCHAR(255),
    blurb VARCHAR(255),
    thumbnail TEXT,
    
    UNIQUE INDEX (url(255)),
    INDEX (name)
    

    ) ENGINE = INNODB;

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-18

    I now have a working repro! I have some leads on the leak, but nothing definite yet. Thanks jj.

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-23

    JJ - the attached patch should fix the "holy crap it leaks like a sieve" situation. Please test it, and I'll get Andy to put this in 1.2.3b2.

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-24

    Ok, the patch is applied to both 1.2br and trunk. Will mark closed once 1.2.3b2 is released.

     
  • Shannon -jj Behrens

    I tried it out on 1.2br. It builds under MacPorts, which is awesome. Better yet, it no longer leaks! Two thumbs way up :-D

     
  • Kyle VanderBeek

    Kyle VanderBeek - 2009-02-28

    Hrm, thought I marked this pending for 1.2.3b2...

     
  • SourceForge Robot

    This Tracker item was closed automatically by the system. It was
    previously set to a Pending status, and the original submitter
    did not respond within 14 days (the time period specified by
    the administrator of this Tracker).

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks