#2 performance improving patch

open
nobody
None
5
2006-06-10
2006-06-10
No

Hi,

I'm using easyh10 1.4, cui on Debian Linux.

I had been using 1.2.1, where UTF-8 was the only
supported encoding -- my filenames are encoded in
ISO-8859-1, though.

Due to the encoding problem, easyh10 1.2.1 consumed
huge amounts of memory (> 2GB), because the recursive
listing of files in the Music directory was buggy --
directories it couldn't read due to the wrong
encoding were appended a "/..", thus reading those
directories multiple times.

My ~2500 files became > 140.000 files. The recursion
probably only stopped when MAX_PATH was reached (the
paths grew longer all the time (i.e.
foo/bar/../bar/../bar).

In the beginning, I didn't notice that those dupes
were causing the memory hogging, so I had a look at
the code to find places where some memory could be
saved.

One such place I found is struct
tag_media_target_file -- instead of using arrays with
the full MAX_PATH length, it would suffice to use the
actual filename's length.

Additionally, I noticed, that the array holding all
those file entries is being realloc()ed for every
single file that is added. I.e. 140.000 times for
me...
Doing this in chunks of e.g. 500 entries is much
nicer.

This patch may not be necessary anymore, because I
will probably never get 140.000 real files onto my
20GB H10, but it doesn't hurt either, and nobody
knows the next generation H10 hdd size.

Discussion

  • Carsten Pfeiffer

    patch for less memory usage

     
  • Naoaki Okazaki

    Naoaki Okazaki - 2006-06-13

    Logged In: YES
    user_id=992421

    Thanks again for sending this patch.

    > Due to the encoding problem, easyh10 1.2.1 consumed
    > huge amounts of memory (> 2GB), because the recursive
    > listing of files in the Music directory was buggy --
    > directories it couldn't read due to the wrong
    > encoding were appended a "/..", thus reading those
    > directories multiple times.
    I think we should fix this first. I thought I prevented this
    phenomenon by:
    if (strcmp(entp->d_name, ".") == 0 || strcmp(entp->d_name,
    "..") == 0) {
    continue;
    }
    in filepathutil_posix.c, but it did not work for some reason.

     
  • Carsten Pfeiffer

    Logged In: YES
    user_id=194457

    The checks for '.' and '..' actually work fine -- the
    problem is that if mbstoucs2() fails, there is gibberish
    in the variable "filename" and in my case, that happened
    to be '..'. Therefore "tmp" ended with "..".

     

Log in to post a comment.