#171 Memory leak

v6.0.3
closed
nobody
None
5
2015-12-27
2013-01-21
odinmillion
No

During the analyzing of the D: drive (2Tb total size, 500 Gb free space, 2 000 000 files) udefrag.exe proceess freezes at 100% point:
- memory usage: 800 Mb
- CPU usage: 25% (on 4-core CPU)
I waited 90 min, but nothing happened. So process didn't start to defrag HDD.

1 Attachments

Discussion

<< < 1 2 3 > >> (Page 2 of 3)
  • Stefan Pendl

    Stefan Pendl - 2013-02-09

    I think a fix for this won't make it into the final release of v6.0.0, but we will continue our investigation.

    Could you attach the source code of your utility to create fragmented files, so I might be able to create a test drive in a shorter time, then I can do that now?


    Stefan

     
    Last edit: Stefan Pendl 2013-02-09
    • odinmillion

      odinmillion - 2013-02-10

      Hello! Tomorrow I will attach source code with some comments!

       
      • odinmillion

        odinmillion - 2013-02-11

        Sorry for ugly code, but it works! If you need any comments please feel free to ask me. I haven't got enough time to develop graceful solution.

        P.S. At Win7 my fragmenter utility sometimes crashes. But I found simple approach. I executed big .bat file like that:
        "G:\GenFragmentedFiles\GenFragmentedFiles.exe
        G:\GenFragmentedFiles\GenFragmentedFiles.exe
        G:\GenFragmentedFiles\GenFragmentedFiles.exe
        G:\GenFragmentedFiles\GenFragmentedFiles.exe
        ...
        ...
        G:\GenFragmentedFiles\GenFragmentedFiles.exe"

         
        Last edit: odinmillion 2013-02-12
        • Stefan Pendl

          Stefan Pendl - 2013-02-11

          Thanks for that, I will check it out.


          Stefan

           
  • odinmillion

    odinmillion - 2013-02-13

    Hello! Please tell me if I can do something more!

     
    • Stefan Pendl

      Stefan Pendl - 2013-02-14

      Currently there is nothing you can help with.

      Happy Valentines day.


      Stefan

       
  • Dmitri Arkhangelski

    Hello,

    Sorry for inconvenience, and thanks for log files and screenshots!

    The problem is not related to the out of memory case. As I see the machine has about 3Gb of RAM installed and the program uses about 1Gb and then simply hangs. Hopefully the log shows us that all the major analysis stuff have been completed and the program still failed to enter the defragmentation phase. So, it hanged somewhere in between.

    I'll try to find out what's going wrong there. Maybe some tests will be needed to prove the code correctness.


    Dmitri

     
  • Dmitri Arkhangelski

    Stefan,

    I've realized unexpectedly that the user sent us a lot of information
    on screenshots. It looks weird, but I took attention only to the log
    file before. So, I'll try to find out where the program hangs. It
    would be nice to release the v6 without this bug or at least find out
    that something is wrong with the user's system itself.

    Did you have tried to reproduce the hang-up?


    Dmitri

     
    • Stefan Pendl

      Stefan Pendl - 2013-02-16

      I wasn't able to reproduce it yet, since creating 2000000 files takes some time.

      I have the source code of the file creation utility, so I will try to duplicate it as soon as possible.


      Stefan

       
  • Dmitri Arkhangelski

    Hi guys,

    I simply suspect that in your case all the installed physical memory has been exhausted during the disk analysis and Windows began to use paging file intensively for all the further memory allocations.

    To make the list of fragmented files the program needs to allocate a few bytes of memory more than one million times in your case. In case of the use of the paging file for each allocation, the process can be very very slow taking noticeable amount of time even for a single memory allocation.

    Could you please look at the LED indicating the disk activity. It shouldn't wink if the problem is somewhere else. If it winks then probably you have exhausted physical memory as I've explained above.


    Dmitri

     
    • odinmillion

      odinmillion - 2013-02-16

      Hello! It is a good idea to to release the v6 without this bug))) If we all muck in together, we'll soon fix this bug!
      1. At the Monday (18.02.2013) I will look at the disk activity LED.
      2. Please take into account those screens. Screens demonstrate stepwise nature of this effect: 98,8% : RAM 500Mb : 25% CPU => 100% : RAM 800Mb : 100 %CPU

       
      Last edit: odinmillion 2013-02-16
  • Dmitri Arkhangelski

    Hello,

    It would be nice also to use Sysinternals Process Explorer (it's free) to gather more information about the hanging process. For example the list of running threads may be helpful indeed.


    Dmitri

     
  • Stefan Pendl

    Stefan Pendl - 2013-02-17

    I have adapted the C# project to be more general and it returns some exit codes to be able to handle restarts in a batch script:

    :restart
        GenFragmentedFiles.exe -d:Y -F:6 -n:2000000
    if %ERRORLEVEL% EQU 3 goto :restart
    
    echo.
    pause
    

    Run the exe without arguments to display the usage.

    I am currently preparing a test drive and will hopefully have some more information in a few days.


    Stefan

     
  • Stefan Pendl

    Stefan Pendl - 2013-02-17

    The utility is really fast, thanks for the source code.

    I also noticed that you tried MyFragmenter, which is incredibly slow due to fragmenting the file after its creation.

    The good thing is that the NTFS driver of Windows is so inefficient, that creating fragmented files is so easy.

    Creating 200000 fragmented files takes about 1.25 hours on my system, so I will do some testing during the next week.


    Stefan

     
    • odinmillion

      odinmillion - 2013-02-18

      Yes, MyFragmenter was too slow)))

      UPD. Inefficiency of NTFS driver is warrantor of udefrag popularity)))

       
      Last edit: odinmillion 2013-02-18
  • Dmitri Arkhangelski

    Hi guys,

    I suppose we've found one more bottleneck in the program. For each of 1.000.000 fragmented files it compares the file with others to put it at the certain position to the list of fragmented files to keep the list sorted by number of fragments. So, each insertion to the list needs not only a single memory allocation, but also up to 1 million comparisons which isn't fast indeed.

    To prove this suspicion I've added a bit more debugging output to the program (installer in attachment). I expect that the program will add first thousands of files to the list very swiftly and then will add further portions of files slower, slower and slower.


    Dmitri

     
    • odinmillion

      odinmillion - 2013-02-18

      Maybe we should use counting sort (http://en.wikipedia.org/wiki/Counting_sort) or another non-trivial sorting algorithm?

      UPD. Maybe we should start sorting once time after 100% complete analyzing stage?

       
      Last edit: odinmillion 2013-02-18
  • odinmillion

    odinmillion - 2013-02-18

    Hello! I used "ultradefrag-6.0.0-rc2-18.02.2013.bin.i386.exe" version and captured some logs. HDD Led didn't wink.

    UPD. My test stend is x64. So I wasn't able to install "
    ultradefrag-6.0.0-rc2-18.02.2013.bin.i386.exe". I installed software on the another PC, then I copied folder to the test stend. And I copied dll's from previous version. If changes situated in the dll's please post portable package.

    UPD2. Please take into account thread 1148 (udefrag.dll+0xb8d4) - 14.png.

     
    Last edit: odinmillion 2013-02-18
  • Stefan Pendl

    Stefan Pendl - 2013-02-18

    I have now been able to do some initial testing using my test drive on my x64 dual CPU machine.

    Thread ultradefrag.exe!StartJobsThreadProc is running at about 30% CPU usage while analysis phase is below 100%.
    Memory usage goes up to about 800MB steadily.

    Thread udefrag.dll!start_job is running at 50% CPU usage after analysis phase reaches 100%, previously it was around 5%.
    Memory usage jumps up from 800MB to 1GB and stays there.

    Attached is the latest x64 installer.


    Stefan

     
  • Stefan Pendl

    Stefan Pendl - 2013-02-18

    Attached is the x64 and x86 portable release.


    Stefan

     
    Last edit: Stefan Pendl 2013-02-18
    • odinmillion

      odinmillion - 2013-02-18

      Thank you for the attach but I asked for "ultradefrag-6.0.0-rc2-18.02.2013.bin.i386.exe" portable version with some additional debug output from
      Dmitri Arkhangelski.

      PS. It is very good that you was able to reproduce hang-up!

       
      • Stefan Pendl

        Stefan Pendl - 2013-02-18

        These are compiled from the latest sources including the additional output.

        I just didn't include the date in the file name, since this doesn't change the contents of the ZIP archive ;-)


        Stefan

         
        • odinmillion

          odinmillion - 2013-02-18

          Thank you for the comment) Should I make some additional tests after your progress with reproducing hang-up?

           
          • Stefan Pendl

            Stefan Pendl - 2013-02-18

            The more results we have the better we will be able to resolve the issue.


            Stefan

             
  • Dmitri Arkhangelski

    Hi,

    Please test using either x86 portable package or with x64 installer. Both of 'em posted above by Stefan.

    If the problem is really with the sorting, I'll rewrite all the code with use of binary trees - they're much more efficient than double linked lists when you need some kind of sorting.

    The v6 code is quite well designed, so it would be not difficult to adjust it for use of the binary trees if tests will prove the problem existence.


    Dmitri

     
<< < 1 2 3 > >> (Page 2 of 3)

Log in to post a comment.