Migrate from GitHub to SourceForge with this tool. Check out all of SourceForge's recent improvements.
Close

#2204 Use two threads for folder compare

Trunk
closed-accepted
5
2007-02-17
2007-02-16
Kimmo Varis
No

Depending on folders compared and type of processor (having hyper-threading or multiple cores) this may make folder compare clearly faster.

Currently (before the patch) we have folder compare in two phases:
1. phase is the collection phase when we find and collect all items we'll compare into itemlist
2. phase is the compare phase when we compare all items in the list

These phases are run serialized, first phase is run into end (all items found) before second phase is started, in the same thread.

This patch makes these two phases run in two different threads at the same time.

So in best case we can start comparing items in second phase after collect thread has found few items. And can do the compare and collecting at the same time.

I think in most cases both phases are disk-intensive tasks, so the disk is the bottleneck. And the effect of this patch is relatively small. But be there few large files, found at begin of collect phase, those files can be compared in memory while other thread finds more files.

Another scenario which gains from this patch is filtering - as it is done in memory.

So while this patch makes compare code a little bit more complex, it certainly causes some speedups for some compare scenarios. I'm using critical sections for locking list access, so its added overhead is minimal.

Discussion

  • Kimmo Varis

    Kimmo Varis - 2007-02-16

    Original and altered files

     
  • Kimmo Varis

    Kimmo Varis - 2007-02-16

    Logged In: YES
    user_id=631874
    Originator: YES

    I don't have large folders to test right now, but probably compare by time/size will gain from this patch - their disk use is minimal in compare phase so compare can advance almost the collect phase speed.

     
  • Kimmo Varis

    Kimmo Varis - 2007-02-17

    Logged In: YES
    user_id=631874
    Originator: YES

    Attaching updated patch. This patch handles special case I found while testing:
    - compare by size/time can be so fast that we must sometimes wait that new items are added to the list

    I'll apply this patch in few minutes.

    File Added: SmarterThreading2.zip

     
  • Kimmo Varis

    Kimmo Varis - 2007-02-17
    • assigned_to: nobody --> kimmov
    • status: open --> closed-accepted
     
  • Kimmo Varis

    Kimmo Varis - 2007-02-17

    Logged In: YES
    user_id=631874
    Originator: YES

    Committed to SVN.
    Completed: At revision: 4110

     

Log in to post a comment.