Menu

#141 sort with multiple thread support to improve performance

Terminator
open
nobody
None
3
2016-07-06
2016-07-05
wangyugui
No

sort command is used in terminator, but it need multiple thread support to improve performance.

sort command is used by 'XXXX|sort -k2.5,2 -k4,4n -k5,5n' in terminator.

'sort -k2.5,2 -k4,4n -k5,5n ref.fa >x.fa' will have the default max 8 thread support.
'sort --parallel=32 -k2.5,2 -k4,4n -k5,5n ref.fa >x.fa' will have the default 32 thread support.

but when used a pipe input, there will be no mutiple thread support in sort event with --parallel=32.
cat ref.fa |sort -k2.5,2 -k4,4n -k5,5n >x.fa ==>single thread
cat ref.fa |sort --parallel=32 -k2.5,2 -k4,4n -k5,5n >x.fa ==>single thread

it will need a lit fix only, but I can't fix perl source.

Discussion

  • wangyugui

    wangyugui - 2016-07-05

    and the sort in terminator will use 1279m or more time, so we need to improve the performance of it.

     
  • wangyugui

    wangyugui - 2016-07-05

    my friend give a dirty fix. i will test it.

    ```
    Index: bin/caqc.pl
    --- bin/caqc.pl (revision 676)
    +++ bin/caqc.pl (working copy)
    @@ -1751,8 +1751,9 @@

     # sort on UID of UTG, or CCO, then begin position, then end
     # UTGs should only be unplaced surrogates
    
    - open( BADMATES, " sort -k2.5,2 -k4,4n -k5,5n > $badMateFile" )
    + #open( BADMATES, " sort -k2.5,2 -k4,4n -k5,5n > $badMateFile" )
    + # die "Couldn't write to $badMateFile.";
    + open( BADMATES, ">","$badMateFile.tmp" );
    local $\ = "\n";
    local $, = "\t";
    while ( my ( $badId, $coord ) = each %bad_mates ) {
    @@ -1764,6 +1765,8 @@
    }
    }
    close BADMATES;
    +
    + system("sort --parallel=32 -k2.5,2 -k4,4n -k5,5n $badMateFile.tmp > $badMateFile");
     exit(0);
    

    }
    ```

     

    Last edit: wangyugui 2016-07-05
  • wangyugui

    wangyugui - 2016-07-06

    the dirty fix seems work. please close this ticket.

     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.