svtoolkit-help — bug reports and trouble shooting

[svtoolkit-help] Error running genotyping Genome Strip.

[Dani M. <dan...@bs...>]

Hello, im running Genome Strip and i had some error in genotype stage.
1st I run Preprocess and Discovery stages with some error but they are
solved when i re-run the script.
My problem starts when I run the genotype script, i dont know if the
org.broad... is bugged or what is happening. Can you help me??

This is my command line:

export SV_DIR=/gpfs/projects/bsc05/dani/Genome_Strip/svtoolkit
classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
java -Xmx4g -cp ${classpath} \
org.broadinstitute.gatk.queue.QCommandLine \
-S ${SV_DIR}/qscript/SVGenotyper.q \
-S ${SV_DIR}/qscript/SVQScript.q \
-cp ${classpath} \
-gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
--disableJobReport \
-configFile ${SV_DIR}/conf/genstrip_parameters.txt \
-R /gpfs/projects/bsc05/jordivalls/GCAT/human_ref_PANCANCER/genome.fa \
-I
/gpfs/projects/bsc05/jordivalls/new_insilico/insilico_3/fastq_bam/insilico_tumor/BAM/bam_mem/bam_merge_all/bam_recalibrator/insilico3_recalib.bam
\
-genderMapFile /gpfs/projects/bsc05/jordivalls/GCAT/GENOME_STRIP/gender.txt
\
-md
/gpfs/scratch/bsc05/bsc05832/GCAT/GCAT/GENOME_STRIP/metadata_insilico_3_nord3
\
-runDirectory
/gpfs/scratch/bsc05/bsc05832/GCAT/GCAT/GENOME_STRIP/genotyping \
-jobLogDir
/gpfs/scratch/bsc05/bsc05832/GCAT/GCAT/GENOME_STRIP/genotyping/logs \
-vcf
/gpfs/scratch/bsc05/bsc05832/GCAT/GCAT/GENOME_STRIP/svdiscovery.dels.vcf
-O
/gpfs/scratch/bsc05/bsc05832/GCAT/GCAT/GENOME_STRIP/genotyping/insilico3.genotypes.vcf
-parallelRecords 100 \
-disableGATKTraversal \
-run

And this is my error:

INFO  16:21:57,799 QCommandLine - Done with errors
##### ERROR --
##### ERROR stack trace
org.broadinstitute.gatk.queue.QException: Error adding function:
org.broadinstitute.sv.qscript.SVQScript$SVGenotyper@20e19ea
        at org.broadinstitute.gatk.queue.engine.QGraph.add(QGraph.scala:141)
        at
org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5$$anonfun$apply$2.apply(QCommandLine.scala:166)
        at
org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5$$anonfun$apply$2.apply(QCommandLine.scala:166)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at
org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5.apply(QCommandLine.scala:166)
        at
org.broadinstitute.gatk.queue.QCommandLine$$anonfun$execute$5.apply(QCommandLine.scala:146)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at
org.broadinstitute.gatk.queue.QCommandLine.execute(QCommandLine.scala:146)
        at
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:256)
        at
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
        at
org.broadinstitute.gatk.queue.QCommandLine$.main(QCommandLine.scala:61)
        at
org.broadinstitute.gatk.queue.QCommandLine.main(QCommandLine.scala)
Caused by: java.lang.IllegalStateException: Function should have at least
one output: CommandLineGATK
        at
org.broadinstitute.gatk.queue.function.QFunction$class.isFail(QFunction.scala:175)
        at
org.broadinstitute.gatk.queue.extensions.gatk.CommandLineGATK.isFail(CommandLineGATK.scala:9)
        at
org.broadinstitute.gatk.queue.engine.FunctionEdge.<init>(FunctionEdge.scala:61)
        at org.broadinstitute.gatk.queue.engine.QGraph.add(QGraph.scala:136)
        ... 14 more
##### ERROR
------------------------------------------------------------------------------------------
##### ERROR A GATK RUNTIME ERROR has occurred (version
3.7.GS-r1748-0-g74bfe0b):
##### ERROR
##### ERROR This might be a bug. Please check the documentation guide to
see if this is a known problem.
##### ERROR If not, please post the error message, with stack trace, to the
GATK forum.
##### ERROR Visit our website and forum for extensive documentation and
answers to
##### ERROR commonly asked questions
https://software.broadinstitute.org/gatk
##### ERROR
##### ERROR MESSAGE: Error adding function:
org.broadinstitute.sv.qscript.SVQScript$SVGenotyper@20e19ea
##### ERROR
------------------------------------------------------------------------------------------
INFO  16:21:57,806 QCommandLine - Shutting down jobs. Please wait...
/home/bsc05/bsc05832/.lsbatch/1540218102.3326608.shell: line 34: -O:
command not found
/home/bsc05/bsc05832/.lsbatch/1540218102.3326608.shell: line 35:
-parallelRecords: command not found
"Nord3_Genomestrip_genotyping_insilico3_SVGenotyper2_.3326608.err" 57L,
5441C


Thanks for help!!!
Greetings


http://bsc.es/disclaimer

[svtoolkit-help] problem using the slurm-drmaa bridge

[Sergiu N. <ser...@sc...>]

Greetings,


I am using the slurm-drmaa bridge, and the pipeline has trouble 
submitting jobs. Is there a way to see the actual cluster job submission 
and figure out what parameters are rejected by the SLURM system?

Submitting with:
java -Xmx4g -cp 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/SVToolkit.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/Queue.jar 
org.broadinstitute.gatk.queue.QCommandLine -S 
/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/qscript/SVPreprocess.q 
-S 
/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/qscript/SVQScript.q 
-gatk 
/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/GenomeAnalysisTK.jar 
-configFile 
/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/conf/genstrip_parameters.txt 
-R /sw/data/uppnex/GATK/2.8/b37/human_g1k_v37.fasta -I 
/proj/sens2016011/nobackup/melt/data/bam_links/00028292.sorted.bam -md 
meta -bamFilesAreDisjoint true -jobLogDir 
/proj/sens2016011/nobackup/genomestrip/tests/batch/logs -run -jobRunner 
Drmaa -gatkJobRunner Drmaa -jobNative -t 4:0:0 -jobNative -p node 
-jobNative -A sens2016011-bianca


Error goes like this:

$ cat logs/SVPreprocess-1.out
  'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC' 
'-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50' 
'-XX:GCHeapFreeLimit=10' 
'-Djava.io.tmpdir=/castor/project/proj_nobackup/genomestrip/tests/batch/.queue/tmp' 
  '-cp' 
'/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/SVToolkit.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/Queue.jar' 
  'org.broadinstitute.sv.apps.CreateMetaDataDirectory'  '-md' 'meta' 
'-configFile' 
'/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/conf/genstrip_parameters.txt' 

org.broadinstitute.gatk.queue.util.RetryException: Gave up after 4 attempts.
	at org.broadinstitute.gatk.queue.util.Retry$.attempt(Retry.scala:67)
	at 
org.broadinstitute.gatk.queue.engine.drmaa.DrmaaJobRunner.start(DrmaaJobRunner.scala:91)
	at 
org.broadinstitute.gatk.queue.engine.FunctionEdge.start(FunctionEdge.scala:101)
.........
	at 
org.broadinstitute.gatk.utils.jna.drmaa.v1_0.JnaSession.checkError(JnaSession.java:392)
	at 
org.broadinstitute.gatk.utils.jna.drmaa.v1_0.JnaSession.runJob(JnaSession.java:79)
	at 
org.broadinstitute.gatk.queue.engine.drmaa.DrmaaJobRunner.runJob(DrmaaJobRunner.scala:115)
	at 
org.broadinstitute.gatk.queue.engine.drmaa.DrmaaJobRunner$$anonfun$start$1.apply$mcV$sp(DrmaaJobRunner.scala:93)
	at 
org.broadinstitute.gatk.queue.engine.drmaa.DrmaaJobRunner$$anonfun$start$1.apply(DrmaaJobRunner.scala:91)
	at 
org.broadinstitute.gatk.queue.engine.drmaa.DrmaaJobRunner$$anonfun$start$1.apply(DrmaaJobRunner.scala:91)
	at org.broadinstitute.gatk.queue.util.Retry$.attempt(Retry.scala:50)
	... 10 more


Thanks!
-- 
---
Sergiu Netotea, PhD
Researcher at NBIS
(National Bioinformatics Infrastructure Sweden)
Department of Biology and Biological Engineering
Chalmers University of Technology
Kemivägen 10, SE-412 96 Göteborg
ser...@sc...
skype: sergiunetotea
+46 (0)70 28 36 306

[svtoolkit-help] cannot submit sample batches for processing

[Sergiu N. <ser...@sc...>]

SV_preprocess requires a full node in order to run the processing of a 
single sample. However it doesn't accept a list of files as input, or am 
I doing something wrong with the -I parameter?

I tried using the file paths separated by space and a file with the same 
paths separated by newlines. The docs are rather vague about it.

The bellow command works, but it requires a full node rather than a 
core. I can't afford it.

java -Xmx4g -cp 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/SVToolkit.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/Queue.jar 
             org.broadinstitute.gatk.queue.QCommandLine             -S 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/qscript/SVPreprocess.q 
             -S 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/qscript/SVQScript.q 
             -gatk 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/lib/gatk/GenomeAnalysisTK.jar 
             -configFile 
$/proj/sens2016011/nobackup/genomestrip/lib/svtoolkit/conf/genstrip_parameters.txt 
             -R /sw/data/uppnex/GATK/2.8/b37/human_g1k_v37.fasta 
     -I 
/proj/sens2016011/nobackup/melt/data/bam_links/00028285.sorted.bam 
       -md meta             -bamFilesAreDisjoint true 
-jobLogDir /proj/sens2016011/nobackup/genomestrip/tests/logs 
-run -jobRunner Drmaa -gatkJobRunner Drmaa -jobNative '-t 4:0:0' 
-jobNative '-p node' -jobNative '-A sens2016011-bianca'


Thanks!
-- 
---
Sergiu Netotea, PhD
Researcher at NBIS
(National Bioinformatics Infrastructure Sweden)
Department of Biology and Biological Engineering
Chalmers University of Technology
Kemivägen 10, SE-412 96 Göteborg
ser...@sc...
skype: sergiunetotea
+46 (0)70 28 36 306

Re: [svtoolkit-help] Samtool

[Wusheng Z. <min...@gm...>]

Hi Daren,

Did you try "which samtools" and what shows?

Best regards,
Wusheng

Liu, Daren T. <dl...@cu...> 于2018年8月13日周一 上午8:49写道：

> Hi User Support for Genomestrip,
>
>
>
> I am currently running the preprocessing step of the pipeline. However, I
> am getting an error message saying: samtools not found. I have provided the
> path for samtools in the global environment as well as added the path to my
> Bashrc Script. Have you encountered problems like this before? Our workflow
> manager is in the style of SGE and we are using the latest version of
> samtools. Attached are the error messages and wrapper script.
>
>
>
>
> Best,
>
> Daren
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>

[svtoolkit-help] Samtool

[Liu, D. T. <dl...@cu...>]

Hi User Support for Genomestrip,

I am currently running the preprocessing step of the pipeline. However, I am getting an error message saying: samtools not found. I have provided the path for samtools in the global environment as well as added the path to my Bashrc Script. Have you encountered problems like this before? Our workflow manager is in the style of SGE and we are using the latest version of samtools. Attached are the error messages and wrapper script.


Best,
Daren

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Bob H. <han...@br...>]

In general, I prefer the forum as more people read it.

I didn't get Thomas' email for some reason.

There is a Queue parameter -maxConcurrentRun N that limits the number of 
concurrent jobs from Queue's point of view.  In a cluster environment, 
this is the maximum number submitted to the cluster at once.  In a 
parallel shell environment, it's the number of concurrent shell jobs.

I don't now how the slurm drmaa api would interact with .sbatch (i.e. 
whether it reads it, which takes precedence). The -jobNative parameters 
are passed through the drmaa api.

-Bob

On 6/28/18 11:41 AM, Wusheng Zhang wrote:
> Hi Thoma,
>
> Thank you very much for sharing your experience. I am working on a 
> cluster with slurm 17.11.7 but without Drma, and I am trying to 
> persuade the administrator to install it. If is it possible, could you 
> share how you setup the corresponding flag? For example, is the 
> following correct?
>
>
> …
>
> -jobRunner Drmaa
>
> -gatkJobRunner 
> <http://software.broadinstitute.org/software/genomestrip/org_broadinstitute_sv_qscript_QCommandLine.html#--gatkJobRunner>Drmaa
>
> …
>
>
> And do I need to setup -jobNative? Because based on my understanding, 
> this flag is used to passing the parameters such as the 
> nodes_per_tasks, memory, walltime, and so on to the system — is this 
> correct? But since in the slurm, we will use the .sbatch script to 
> submit the job and setup the parameters above, I am not sure how to 
> make -jobNative consistent with the sbatch script. May I have your 
> suggestion?
>
>
> Thank you very much.
>
>
> Best regards,
>
> Wusheng
>
>
>
> Thomas Faraut <Tho...@in... <mailto:Tho...@in...>> 
> 于2018年5月22日周二 上午4:29写道：
>
>     Dear Wusheng,
>
>     I have been using genome STRIP on two clusters with SLURM.
>
>     On the first one the Drmaa interface was available and
>     very easy to use. You simply have to make sure that the
>     Drmaa library file (libdrmaa.so) is in your LD_LIBRARY_PATH.
>
>     On the second cluster there is no Drmaa interface (not
>     implemented in part because of the scheduling policy).
>     On this cluster I found no solution except to run Genome
>     STRP as a single process which is clearly not convenient.
>     If you find a solution for this I would be very much interested.
>
>     By the way, it would be nice, when using the Queue workflow
>     engine to control the maximum number of (simultaneous) submissions
>     to reduce our footprint on the cluster Queue.
>     I don't know if it is currently possible.
>
>     Best regards,
>     Thomas
>
>
>     Le 18/05/2018 à 18:38, Wusheng Zhang a écrit :
>>     Dear Genome STRiP users,
>>
>>     I am new to this software and intend to install this software on
>>     my university HPC cluster. But the cluster management software
>>     the university is using is the SLURM rather than LSF or SGE. In
>>     this sense, is it still possible to install Genome STRiP on my
>>     university cluster and is there anything I need to change based
>>     on the original installation steps? Thank you very much.
>>
>>     Best regards,
>>     Wusheng
>>
>>
>>     ------------------------------------------------------------------------------
>>     Check out the vibrant tech community on one of the world's most
>>     engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>
>>
>>     _______________________________________________
>>     svtoolkit-help mailing list
>>     svt...@li...
>>     <mailto:svt...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
>     -- 
>     Thomas Faraut
>
>     New email add...@in... <mailto:Tho...@in...>
>
>     UMR 1388 INRA-INPT GenPhySE
>     Tel: +33 (0)5 61 28 54 57
>     http://genphyse.toulouse.inra.fr
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org!
>     http://sdm.link/slashdot_______________________________________________
>     svtoolkit-help mailing list
>     svt...@li...
>     <mailto:svt...@li...>
>     https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Thomas F. <Tho...@in...>]

Hi Wusheng,

The options I use for running SVDiscovery (and CNVDiscoveryPipeline) on
our cluster with slurm and Drmaa are :

   -jobRunner Drmaa
   -gatkJobRunner Drmaa
   -jobNative "--mem-per-cpu={cluster.mem}000 --time={cluster.time}"
   -jobQueue workq

where indeed -jobNative specifies the options to be passed to slurm.

This is how genomeSTRIP with interact with the cluster.
Putting the genomeSTRIP command in a batch script is a different
issue, it will simply delegate the genomeSTRIP master process
to a job send on the cluster (so both things are independent)

A simple reminder, SVDiscovery can work as a single process, while
the CNVDiscoveryPipeline will send jobs to the cluster without
asking for permission. So the latter will not work with a slurm
cluster without Drmaa.

Best regards,
Thomas


Le 28/06/2018 à 17:41, Wusheng Zhang a écrit :
> Hi Thoma,
>
> Thank you very much for sharing your experience. I am working on a 
> cluster with slurm 17.11.7 but without Drma, and I am trying to 
> persuade the administrator to install it. If is it possible, could you 
> share how you setup the corresponding flag? For example, is the 
> following correct?
>
>
> …
>
> -jobRunner Drmaa
>
> -gatkJobRunner 
> <http://software.broadinstitute.org/software/genomestrip/org_broadinstitute_sv_qscript_QCommandLine.html#--gatkJobRunner>Drmaa
>
> …
>
>
> And do I need to setup -jobNative? Because based on my understanding, 
> this flag is used to passing the parameters such as the 
> nodes_per_tasks, memory, walltime, and so on to the system — is this 
> correct? But since in the slurm, we will use the .sbatch script to 
> submit the job and setup the parameters above, I am not sure how to 
> make -jobNative consistent with the sbatch script. May I have your 
> suggestion?
>
>
> Thank you very much.
>
>
> Best regards,
>
> Wusheng
>
>
>
> Thomas Faraut <Tho...@in... <mailto:Tho...@in...>> 
> 于2018年5月22日周二 上午4:29写道：
>
>     Dear Wusheng,
>
>     I have been using genome STRIP on two clusters with SLURM.
>
>     On the first one the Drmaa interface was available and
>     very easy to use. You simply have to make sure that the
>     Drmaa library file (libdrmaa.so) is in your LD_LIBRARY_PATH.
>
>     On the second cluster there is no Drmaa interface (not
>     implemented in part because of the scheduling policy).
>     On this cluster I found no solution except to run Genome
>     STRP as a single process which is clearly not convenient.
>     If you find a solution for this I would be very much interested.
>
>     By the way, it would be nice, when using the Queue workflow
>     engine to control the maximum number of (simultaneous) submissions
>     to reduce our footprint on the cluster Queue.
>     I don't know if it is currently possible.
>
>     Best regards,
>     Thomas
>
>
>     Le 18/05/2018 à 18:38, Wusheng Zhang a écrit :
>>     Dear Genome STRiP users,
>>
>>     I am new to this software and intend to install this software on
>>     my university HPC cluster. But the cluster management software
>>     the university is using is the SLURM rather than LSF or SGE. In
>>     this sense, is it still possible to install Genome STRiP on my
>>     university cluster and is there anything I need to change based
>>     on the original installation steps? Thank you very much.
>>
>>     Best regards,
>>     Wusheng
>>
>>
>>     ------------------------------------------------------------------------------
>>     Check out the vibrant tech community on one of the world's most
>>     engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>
>>
>>     _______________________________________________
>>     svtoolkit-help mailing list
>>     svt...@li...
>>     <mailto:svt...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
>     -- 
>     Thomas Faraut
>
>     New email add...@in... <mailto:Tho...@in...>
>
>     UMR 1388 INRA-INPT GenPhySE
>     Tel: +33 (0)5 61 28 54 57
>     http://genphyse.toulouse.inra.fr
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org!
>     http://sdm.link/slashdot_______________________________________________
>     svtoolkit-help mailing list
>     svt...@li...
>     <mailto:svt...@li...>
>     https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>

-- 
Thomas Faraut

New email address Tho...@in...

UMR 1388 INRA-INPT GenPhySE
Tel: +33 (0)5 61 28 54 57
http://genphyse.toulouse.inra.fr

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Wusheng Z. <min...@gm...>]

Hi Thoma,

Thank you very much for sharing your experience. I am working on a cluster
with slurm 17.11.7 but without Drma, and I am trying to persuade the
administrator to install it. If is it possible, could you share how you
setup the corresponding flag? For example, is the following correct?


…

-jobRunner Drmaa

-gatkJobRunner
<http://software.broadinstitute.org/software/genomestrip/org_broadinstitute_sv_qscript_QCommandLine.html#--gatkJobRunner>
Drmaa

…


And do I need to setup -jobNative? Because based on my understanding, this
flag is used to passing the parameters such as the nodes_per_tasks, memory,
walltime, and so on to the system — is this correct? But since in the
slurm, we will use the .sbatch script to submit the job and setup the
parameters above, I am not sure how to make -jobNative consistent with the
sbatch script. May I have your suggestion?


Thank you very much.


Best regards,

Wusheng


Thomas Faraut <Tho...@in...> 于2018年5月22日周二 上午4:29写道：

> Dear Wusheng,
>
> I have been using genome STRIP on two clusters with SLURM.
>
> On the first one the Drmaa interface was available and
> very easy to use. You simply have to make sure that the
> Drmaa library file (libdrmaa.so) is in your LD_LIBRARY_PATH.
>
> On the second cluster there is no Drmaa interface (not
> implemented in part because of the scheduling policy).
> On this cluster I found no solution except to run Genome
> STRP as a single process which is clearly not convenient.
> If you find a solution for this I would be very much interested.
>
> By the way, it would be nice, when using the Queue workflow
> engine to control the maximum number of (simultaneous) submissions
> to reduce our footprint on the cluster Queue.
> I don't know if it is currently possible.
>
> Best regards,
> Thomas
>
>
> Le 18/05/2018 à 18:38, Wusheng Zhang a écrit :
>
> Dear Genome STRiP users,
>
> I am new to this software and intend to install this software on my
> university HPC cluster. But the cluster management software the university
> is using is the SLURM rather than LSF or SGE. In this sense, is it still
> possible to install Genome STRiP on my university cluster and is there
> anything I need to change based on the original installation steps? Thank
> you very much.
>
> Best regards,
> Wusheng
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> svtoolkit-help mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
>
> --
> Thomas Faraut
>
> New email address Tho...@in...
>
> UMR 1388 INRA-INPT GenPhySE
> Tel: +33 (0)5 61 28 54 57http://genphyse.toulouse.inra.fr
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Wusheng Z. <min...@gm...>]

Hi Bob,


Thank you very much for your help! After reading your answer, I found I am
too ignorance to the genomestrip last month, and now I think I have enough
knowledge for asking help. Since the cluster I am using is slurm 17.11.7, I
hope that I can directly use Drmaa without the patches. But if it does not
work, it will be very helpful if you can share your pathces. And I am
trying to persuade the administrator to install the Drmaa now.


But before that, is it possible to run CNVDiscovery sequentially with the
--jobRunner Shell \ --gatkRunner Shell \ (without any --jobnative flag) on
the cluster? BTW, based on my understanding, only to the CNVDiscovery step,
specifying the --jobRunner and --gatkRunner is needed — is this correct?


And should I ask questions on the GATK forum rather than the mailing list?


Thank you very much.


Best regards,

Wusheng


Bob Handsaker <han...@br...> 于2018年5月18日周五 下午5:36写道：

> Hi, Wusheng,
>
> I have gotten Genome STRiP (GS) to run on slurm, but the one time I did
> this we had to make a patch to the DRMAA library. This was done for a
> relatively old version of slurm (that was running on the cluster of
> interest) and although I sent some emails I never got any feedback as to
> whether the patches could be contributed back to the slurm code base.
>
> If you want to try to get it running, I am happy to share the slurm
> patches we had to make with you, especially if you would help get the drmaa
> API fixes to be generally supported in slurm.
>
> -Bob Handsaker
> On 5/18/18 12:38 PM, Wusheng Zhang wrote:
>
> Dear Genome STRiP users,
>
> I am new to this software and intend to install this software on my
> university HPC cluster. But the cluster management software the university
> is using is the SLURM rather than LSF or SGE. In this sense, is it still
> possible to install Genome STRiP on my university cluster and is there
> anything I need to change based on the original installation steps? Thank
> you very much.
>
> Best regards,
> Wusheng
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> svtoolkit-help mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
>
>

Re: [svtoolkit-help] Error message when running SVPreprocess

[Wusheng Z. <min...@gm...>]

I found that there is such kind of error

INFO  10:53:37,168 HelpFormatter -
-------------------------------------------------------------
INFO  10:53:37,175 HelpFormatter - Program Name:
org.broadinstitute.sv.apps.ComputeDepthProfiles
INFO  10:53:37,180 HelpFormatter - Program Args: -O
/proj/yunligrp/users/minzhi/md_tempdir/profiles_100Kb/profile_seq_chr16_100000.dat.gz
-I md_tempdir/headers.bam -configFile
/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt -R
/proj/yunligrp/users/yesu/hs38DH/hs38DH.chr16.fasta/test.fa -L chr16:0-0
-md md_tempdir -profileBinSize 100000 -maximumReferenceGapLength 10000
INFO  10:53:37,192 HelpFormatter - Executing as mi...@b1... on
Linux 3.10.0-693.11.6.el7.x86_64 amd64; OpenJDK 64-Bit Server VM
1.8.0_171-b10.
INFO  10:53:37,192 HelpFormatter - Date/Time: 2018/06/14 10:53:37
INFO  10:53:37,193 HelpFormatter -
-------------------------------------------------------------
INFO  10:53:37,193 HelpFormatter -
-------------------------------------------------------------
Exception in thread "main"
org.broadinstitute.sv.commandline.ArgumentException: Alignment file does
not exist: md_tempdir/headers.bam
at org.broadinstitute.sv.dataset.SAMLocation.create(SAMLocation.java:99)
at
org.broadinstitute.sv.commandline.CommandLineParser.createSAMLocation(CommandLineParser.java:256)
at
org.broadinstitute.sv.commandline.CommandLineParser.parseSAMLocations(CommandLineParser.java:236)
at
org.broadinstitute.sv.commandline.CommandLineParser.parseSAMLocations(CommandLineParser.java:220)
at
org.broadinstitute.sv.apps.ComputeDepthProfiles.run(ComputeDepthProfiles.java:117)
at
org.broadinstitute.sv.commandline.CommandLineProgram.execute(CommandLineProgram.java:54)
at
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:256)
at
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
at
org.broadinstitute.sv.commandline.CommandLineProgram.runAndReturnResult(CommandLineProgram.java:29)
at
org.broadinstitute.sv.commandline.CommandLineProgram.run(CommandLineProgram.java:25)
at
org.broadinstitute.sv.apps.ComputeDepthProfiles.main(ComputeDepthProfiles.java:109)

The "md_tempdir" is the directory of -md flag. My confusion is that the
data in this files should be generated during the running rather than I
prepare at first. Besides, other .out files did not generate such kind of
problem. I used our own reference reference data rather than the data
provided by the genome strip, so will this lead to the error? May I have
your suggestions?

Thank you in advance.

Best regards,
Minzhi

2018-06-14 10:10 GMT-04:00 Wusheng Zhang <min...@gm...>:

> Dear Genome STRiP users,
>
> I am running SVPreprocess on the cluster with SLURM as the schedule
> management software. I started an interactive session with 8 cores, and
> then executed the following lines on the terminal:
>
> module purge
> module load r/3.5.0
> module load samtools/1.8
> module load tabix/0.2.6
>
> classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/
> GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
> SV_DIR="/proj/yunligrp/users/minzhi/svtoolkit"
> java -Xmx4g -cp ${classpath}\
>      org.broadinstitute.gatk.queue.QCommandLine\
>      -S ${SV_DIR}/qscript/SVPreprocess.q\
>      -S ${SV_DIR}/qscript/SVQScript.q\
>      -cp ${classpath}\
>      -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
>      -configFile ${SV_DIR}/conf/genstrip_parameters.txt \
>      -R /proj/yunligrp/users/yesu/hs38DH/hs38DH.chr16.fasta/test.fa \
>      -I /proj/yunligrp/users/minzhi/JHS_30.list \
>      -md md_tempdir \
>      -ploidyMapFile standard_ploidy.map \
>      -bamFilesAreDisjoint true \
>      -jobLogDir logs \
>      -run
>
> The version of JAVA is here
>
> [minzhi@longleaf-login1 minzhi]$ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
>
> I got NO errors when the first time I executed the above command lines,
> but when I run it again, it returns errors shown as below
>
> ...
> *ERROR* 20:51:55,597 FunctionEdge - Error:  'java'  '-Xmx2048m'
> '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'
> '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/.queue/tmp'
> '-cp' ':/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar::/
> proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysis
> TK.jar::/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-cp'
> '/proj/yunligrp/users/minzhi:/proj/yunligrp/users/minzhi/svt
> oolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi:/proj/
> yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.
> jar:/proj/yunligrp/users/minzhi:/proj/yunligrp/users/
> minzhi/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.sv.apps.ComputeGCProfiles'
> '-O' '/proj/yunligrp/users/minzhi/md_tempdir/gcprofile/reference.gcprof.zip'
> '-R' '/proj/yunligrp/users/yesu/hs38DH/hs38DH.chr16.fasta/test.fa'  '-md'
> 'md_tempdir'  '-writeReferenceProfile' 'true'  '-configFile'
> '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'
> *ERROR* 20:51:55,600 FunctionEdge - Contents of
> /proj/yunligrp/users/minzhi/logs/*SVPreprocess-6.out*:
> INFO  20:46:16,772 HelpFormatter - ------------------------------
> ----------------------------
> INFO  20:46:16,775 HelpFormatter - Program Name:
> org.broadinstitute.sv.apps.ComputeGCProfiles
> ...
> INFO  20:56:10,398 QGraph - 92 Pend, 1 Run, 1 Fail, 67 Done
> ...
> INFO  20:56:35,371 QGraph - 92 Pend, 0 Run, 1 Fail, 68 Done
> ...
> INFO  20:56:38,945 QCommandLine - Done with errors
>
> And then I re-run the svpreprocess with regards to the same .bam files.
> However, there are two errors but happened at different indexes (*SVPreprocess-73.out
> and SVPreprocess-80.out*).
>
> I am confused about two parts:
>
> 1. Does anyone meet such kind of error before, and may I have your
> suggestions about this error? I checked the corresponding log file such as SVPreprocess-6.out
> and the SVPreprocess.jobreport.txt, but there is no error message in
> those files.
> 2. My log file started from index 6 rather than 0 which means that the
> first log file is SVPreprocess-6.out but not SVPreprocess-0.out -- does
> it mean that I made anything wrong?
>
> Thank you in advance.
>
> Best regards,
> Wusheng
>
>

[svtoolkit-help] Error message when running SVPreprocess

[Wusheng Z. <min...@gm...>]

Dear Genome STRiP users,

I am running SVPreprocess on the cluster with SLURM as the schedule
management software. I started an interactive session with 8 cores, and
then executed the following lines on the terminal:

module purge
module load r/3.5.0
module load samtools/1.8
module load tabix/0.2.6

classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/
gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
SV_DIR="/proj/yunligrp/users/minzhi/svtoolkit"
java -Xmx4g -cp ${classpath}\
     org.broadinstitute.gatk.queue.QCommandLine\
     -S ${SV_DIR}/qscript/SVPreprocess.q\
     -S ${SV_DIR}/qscript/SVQScript.q\
     -cp ${classpath}\
     -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
     -configFile ${SV_DIR}/conf/genstrip_parameters.txt \
     -R /proj/yunligrp/users/yesu/hs38DH/hs38DH.chr16.fasta/test.fa \
     -I /proj/yunligrp/users/minzhi/JHS_30.list \
     -md md_tempdir \
     -ploidyMapFile standard_ploidy.map \
     -bamFilesAreDisjoint true \
     -jobLogDir logs \
     -run

The version of JAVA is here

[minzhi@longleaf-login1 minzhi]$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

I got NO errors when the first time I executed the above command lines, but
when I run it again, it returns errors shown as below

...
*ERROR* 20:51:55,597 FunctionEdge - Error:  'java'  '-Xmx2048m'
'-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'
'-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/
yunligrp/users/minzhi/.queue/tmp'  '-cp' ':/proj/yunligrp/users/minzhi/
svtoolkit/lib/SVToolkit.jar::/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/
GenomeAnalysisTK.jar::/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'
'-cp' '/proj/yunligrp/users/minzhi:/proj/yunligrp/users/minzhi/
svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi:/
proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/
yunligrp/users/minzhi:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'
'org.broadinstitute.sv.apps.ComputeGCProfiles'  '-O'
'/proj/yunligrp/users/minzhi/md_tempdir/gcprofile/reference.gcprof.zip'
'-R' '/proj/yunligrp/users/yesu/hs38DH/hs38DH.chr16.fasta/test.fa'  '-md'
'md_tempdir'  '-writeReferenceProfile' 'true'  '-configFile'
'/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'
*ERROR* 20:51:55,600 FunctionEdge - Contents of /proj/yunligrp/users/minzhi/
logs/*SVPreprocess-6.out*:
INFO  20:46:16,772 HelpFormatter - ------------------------------
----------------------------
INFO  20:46:16,775 HelpFormatter - Program Name: org.broadinstitute.sv.apps.
ComputeGCProfiles
...
INFO  20:56:10,398 QGraph - 92 Pend, 1 Run, 1 Fail, 67 Done
...
INFO  20:56:35,371 QGraph - 92 Pend, 0 Run, 1 Fail, 68 Done
...
INFO  20:56:38,945 QCommandLine - Done with errors

And then I re-run the svpreprocess with regards to the same .bam files.
However, there are two errors but happened at different indexes
(*SVPreprocess-73.out
and SVPreprocess-80.out*).

I am confused about two parts:

1. Does anyone meet such kind of error before, and may I have your
suggestions about this error? I checked the corresponding log file
such as SVPreprocess-6.out
and the SVPreprocess.jobreport.txt, but there is no error message in those
files.
2. My log file started from index 6 rather than 0 which means that the
first log file is SVPreprocess-6.out but not SVPreprocess-0.out -- does it
mean that I made anything wrong?

Thank you in advance.

Best regards,
Wusheng

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Bob H. <han...@br...>]

Thanks so much for sharing your experience. If you want to post this on 
the forum, that would be great.

It is great to know that Queue runs OK on a slurm system with drmaa 
support. My only experience with slurm was with a rather old version and 
we had to make modifications to the slurm code to fix problems in the 
drmaa API in order to get Genome STRiP to run.

There is a flag to Queue to do what I think you want: -maxConcurrentRun N

Limits the number of jobs that will be sent to the job scheduler 
simultaneously.

-Bob

On 5/22/18 4:28 AM, Thomas Faraut wrote:
> Dear Wusheng,
>
> I have been using genome STRIP on two clusters with SLURM.
>
> On the first one the Drmaa interface was available and
> very easy to use. You simply have to make sure that the
> Drmaa library file (libdrmaa.so) is in your LD_LIBRARY_PATH.
>
> On the second cluster there is no Drmaa interface (not
> implemented in part because of the scheduling policy).
> On this cluster I found no solution except to run Genome
> STRP as a single process which is clearly not convenient.
> If you find a solution for this I would be very much interested.
>
> By the way, it would be nice, when using the Queue workflow
> engine to control the maximum number of (simultaneous) submissions
> to reduce our footprint on the cluster Queue.
> I don't know if it is currently possible.
>
> Best regards,
> Thomas
>
>
> Le 18/05/2018 à 18:38, Wusheng Zhang a écrit :
>> Dear Genome STRiP users,
>>
>> I am new to this software and intend to install this software on my 
>> university HPC cluster. But the cluster management software the 
>> university is using is the SLURM rather than LSF or SGE. In this 
>> sense, is it still possible to install Genome STRiP on my university 
>> cluster and is there anything I need to change based on the original 
>> installation steps? Thank you very much.
>>
>> Best regards,
>> Wusheng
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>
>>
>> _______________________________________________
>> svtoolkit-help mailing list
>> svt...@li...
>> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help
>
> -- 
> Thomas Faraut
>
> New email add...@in...
>
> UMR 1388 INRA-INPT GenPhySE
> Tel: +33 (0)5 61 28 54 57
> http://genphyse.toulouse.inra.fr
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help

Re: [svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Thomas F. <Tho...@in...>]

Dear Wusheng,

I have been using genome STRIP on two clusters with SLURM.

On the first one the Drmaa interface was available and
very easy to use. You simply have to make sure that the
Drmaa library file (libdrmaa.so) is in your LD_LIBRARY_PATH.

On the second cluster there is no Drmaa interface (not
implemented in part because of the scheduling policy).
On this cluster I found no solution except to run Genome
STRP as a single process which is clearly not convenient.
If you find a solution for this I would be very much interested.

By the way, it would be nice, when using the Queue workflow
engine to control the maximum number of (simultaneous) submissions
to reduce our footprint on the cluster Queue.
I don't know if it is currently possible.

Best regards,
Thomas


Le 18/05/2018 à 18:38, Wusheng Zhang a écrit :
> Dear Genome STRiP users,
>
> I am new to this software and intend to install this software on my 
> university HPC cluster. But the cluster management software the 
> university is using is the SLURM rather than LSF or SGE. In this 
> sense, is it still possible to install Genome STRiP on my university 
> cluster and is there anything I need to change based on the original 
> installation steps? Thank you very much.
>
> Best regards,
> Wusheng
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help

-- 
Thomas Faraut

New email address Tho...@in...

UMR 1388 INRA-INPT GenPhySE
Tel: +33 (0)5 61 28 54 57
http://genphyse.toulouse.inra.fr

[svtoolkit-help] Is it possible to install Genome STRiP on an HPC cluster with SLURM as the management software?

[Wusheng Z. <min...@gm...>]

Dear Genome STRiP users,

I am new to this software and intend to install this software on my
university HPC cluster. But the cluster management software the university
is using is the SLURM rather than LSF or SGE. In this sense, is it still
possible to install Genome STRiP on my university cluster and is there
anything I need to change based on the original installation steps? Thank
you very much.

Best regards,
Wusheng

[svtoolkit-help] Genome Strip 2 using wrong bin size

[Wakeling, M. <M.W...@ex...>]

Hi.

I'm trying to run the LCNV detection pipeline from Genome Strip 2 on a set of 237 whole genome samples. I have performed the PreProcess and GenerateDepthProfiles stages, and am trying to run the LCNVDiscoveryPipeline stage. When I run queue, it reports that it has created 19993 jobs, however all the jobs then fail. For instance, the first job is:

INFO  08:31:37,845 FunctionEdge - Starting:    'Rscript'  '/gpfs/ts0/home/mw501/Research_Project-MRC147594/genome_sequencing/genome_strip2/svtoolkit/R/lcnv/lcnv_scan.R'  '--profileFile' 'profiles_10000/profile_seq_1_728.dat.gz'  '--targetSample' 'WG0007'  '--maxDepth' '60'  '--ploidyMapFile' '/gpfs/ts0/home/mw501/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.ploidymap.txt'  '--genderMapFile' 'output_metadata_directory/sample_gender.report.txt'  '--outputFile' 'lcnv_output/seq_1/1_WG0007.dat'

and this fails because there is no file named profiles_10000/profile_seq_1_728.dat.gz - but there is a file named profiles_10000/profile_seq_1_10000.dat.gz. I'm not sure where it is getting the 728 from.

The arguments for the three stages are:

export SV_DIR=~/Research_Project-MRC147594/genome_sequencing/genome_strip2/svtoolkit
classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
java -Xmx4g -cp ${classpath} \
     org.broadinstitute.gatk.queue.QCommandLine \
     -S ${SV_DIR}/qscript/SVPreprocess.q \
     -S ${SV_DIR}/qscript/SVQScript.q \
     -cp ${classpath} \
     -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
     -configFile ${SV_DIR}/conf/genstrip_parameters.txt \
     -R ~/Research_Project-MRC147594/resources/grch37/human_g1k_v37.fasta \
     -I bams.list \
     -md output_metadata_directory \
     -bamFilesAreDisjoint true \
     -copyNumberMaskFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.gcmask.fasta \
     -genderMaskBedFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.gendermask.bed \
     -genomeMaskFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.svmask.fasta \
     -ploidyMapFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.ploidymap.txt \
     -readDepthMaskFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.rdmask.bed \
     -jobLogDir logDir \
     -jobRunner ParallelShell \
     -maxConcurrentRun 16 \
     -run

export SV_DIR=~/Research_Project-MRC147594/genome_sequencing/genome_strip2/svtoolkit
classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
java -Xmx4g -cp ${classpath} \
     org.broadinstitute.gatk.queue.QCommandLine \
     -S ${SV_DIR}/qscript/profiles/GenerateDepthProfiles.q \
     -S ${SV_DIR}/qscript/SVQScript.q \
     -cp ${classpath} \
     -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
     -R ~/Research_Project-MRC147594/resources/grch37/human_g1k_v37.fasta \
     -md output_metadata_directory \
     -profileBinSize 10000 \
     -maximumReferenceGapLength 1000 \
     -runDirectory profiles_10000 \
     -genomeMaskFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.svmask.fasta \
     -ploidyMapFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.ploidymap.txt \
     -jobLogDir profiles_10000/logDir \
     -jobRunner ParallelShell \
     -maxConcurrentRun 16 \
     -run

export SV_DIR=~/Research_Project-MRC147594/genome_sequencing/genome_strip2/svtoolkit
classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
java -Xmx4g -cp ${classpath} \
     org.broadinstitute.gatk.queue.QCommandLine \
     -S ${SV_DIR}/qscript/discovery/lcnv/LCNVDiscoveryPipeline.q \
     -S ${SV_DIR}/qscript/SVQScript.q \
     -cp ${classpath} \
     -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
     -R ~/Research_Project-MRC147594/resources/grch37/human_g1k_v37.fasta \
     -md output_metadata_directory \
     -profilesDir profiles_10000 \
     -runDirectory lcnv_output \
     -maxDepth 60 \
     -genomeMaskFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.svmask.fasta \
     -ploidyMapFile ~/Research_Project-MRC147594/resources/hg19/Homo_sapiens_assembly19/Homo_sapiens_assembly19.ploidymap.txt \
     -genderMapFile output_metadata_directory/sample_gender.report.txt \
     -jobLogDir lcnv_output/logDir \
     -jobRunner ParallelShell \
     -maxConcurrentRun 16 \
     -run

Any assistance would be greatly appreciated.

Matthew

[svtoolkit-help] Genome STRiP software qustion

[王四华 <ws...@12...>]

Dear experts：
I have some doubt about Genome STRiP software.
As we all know:
To run discovery or genotyping on a single sequenced genome or a small set of genomes, you need to call your data against a background population, 
such as a set of genomes from the 1000 Genomes Project.  The background population does not need to be matched to the target individuals.
But how could I use the background population?  If I hava a single sequenced genome sample (named: Sample AA),and how can I use the background population? 
If I have 20 background sample.I did not find the parameter for the background population. I only know the parameter (- I) for the input bam file for target sample AA. 
Expect your reply! Thank you very much!

[svtoolkit-help] ArgumentMatchStringValue error for MergeInsertSizeHistograms during running installtest

[Yige Wu <yi...@wu...>]

Hi All,

I was running the discovery.sh under the directory 
/svtoolkit/installtest and got the following error during SVPreprocess. 
I think it says the input file "installtest.hist.bin" has the 
inappropriate data type, although there's no error or warning for the 
previous step that generated this file (also pasted below). I'm clueless 
where should I begin to check for this error. (tried the latest 2 
versions of svtoolkit, got the same error; java version "1.8.0_45"; 
samtools version 1.6)

Error for MergeInsertSizeHistograms:

ERROR 19:54:21,182 FunctionEdge - Error:  'java'  '-Xmx2048m' 
'-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4' '-XX:GCTimeLimit=50'  
'-XX:GCHeapFreeLimit=10' '-Djava.io.tmpdir=/home/yigewu2012/tmpdir'  
'-cp' 
'/home/yigewu2012/svtoolkit/lib/SVToolkit.jar:/home/yigewu2012/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/home/yigewu2012/svtoolkit/lib/gatk/Queue.jar' 
'-cp' 
'/home/yigewu2012/svtoolkit/lib/SVToolkit.jar:/home/yigewu2012/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/home/yigewu2012/svtoolkit/lib/gatk/Queue.jar' 
'org.broadinstitute.sv.apps.MergeInsertSizeHistograms'  '-I' 
'/home/yigewu2012/test1/metadata/isd/installtest.hist.bin'  '-O' 
'/home/yigewu2012/test1/metadata/isd.hist.bin'
ERROR 19:54:21,184 FunctionEdge - Contents of 
/home/yigewu2012/test1/logs/SVPreprocess-8.out:
Exception in thread "main" 
org.broadinstitute.gatk.utils.exceptions.UserException$CommandLineException: 
Invalid command line: Failed to parse value 
org.broadinstitute.gatk.utils.commandline.ArgumentMatchStringValue@60615d72 
for argument mInputFiles.  This is most commonly caused by providing an 
incorrect data type (e.g. a double when an int is required)
         at 
org.broadinstitute.gatk.utils.commandline.SimpleArgumentTypeDescriptor.parse(ArgumentTypeDescriptor.java:756)
         at 
org.broadinstitute.gatk.utils.commandline.CompoundArgumentTypeDescriptor.parse(ArgumentTypeDescriptor.java:834)
         at 
org.broadinstitute.gatk.utils.commandline.ArgumentTypeDescriptor.parse(ArgumentTypeDescriptor.java:137)
         at 
org.broadinstitute.gatk.utils.commandline.ArgumentSource.parse(ArgumentSource.java:119)
         at 
org.broadinstitute.gatk.utils.commandline.ParsingEngine.loadValueIntoObject(ParsingEngine.java:509)
         at 
org.broadinstitute.gatk.utils.commandline.ParsingEngine.loadArgumentsIntoObject(ParsingEngine.java:429)
         at 
org.broadinstitute.gatk.utils.commandline.ParsingEngine.loadArgumentsIntoObject(ParsingEngine.java:403)
         at 
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:235)
         at 
org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
         at 
org.broadinstitute.sv.commandline.CommandLineProgram.runAndReturnResult(CommandLineProgram.java:29)
         at 
org.broadinstitute.sv.commandline.CommandLineProgram.run(CommandLineProgram.java:25)
         at 
org.broadinstitute.sv.apps.MergeInsertSizeHistograms.main(MergeInsertSizeHistograms.java:40) 


The log for generating "installtest.hist.bin":

INFO  19:53:51,177 FunctionEdge - Done:  'java'  '-Xmx2048m' 
'-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4' '-XX:GCTimeLimit=50'  
'-XX:GCHeapFreeLimit=10' '-Djava.io.tmpdir=/home/yigewu2012/tmpdir'  
'-cp' 
'/home/yigewu2012/svtoolkit/lib/SVToolkit.jar:/home/yigewu2012/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/home/yigewu2012/svtoolkit/lib/gatk/Queue.jar' 
org.broadinstitute.sv.main.SVCommandLine '-T' 
'ComputeInsertSizeHistogramsWalker'  '-R' 
'/home/yigewu2012/svtoolkit/installtest/data/human_b36_chr1.fasta' '-I' 
'/home/yigewu2012/svtoolkit/installtest/data/installtest.bam' '-O' 
'/home/yigewu2012/test1/metadata/isd/installtest.hist.bin' 
'-disableGATKTraversal' 'true'  '-md' 'test1/metadata' '-genomeInterval' 
'1:61700000-61900000'  '-configFile' 
'/home/yigewu2012/svtoolkit/installtest/conf/genstrip_installtest_parameters.txt' 
'-chimerismFile' 'test1/metadata/isd/installtest.chimer.dat' 
'-createHistogramFile' 'true'  -createEmpty

/svtoolkit/installtest.

/svtoolkit/installtest

[svtoolkit-help] New User to GenomeSTRiP Questions

[John B. <joh...@bu...>]

To whom it may concern,

I am new to GenomeSTRiP and fairly new to using computers in a research
fashion (running terminal, properly formatting file paths).

My question is, the necessary parts I have downloaded (BWA, samtools, R,
picard) came in .jar or .zip files. Do I need to extract them before being
able to "use" them in the file pathways?

Thank you,
John Binkowski

Re: [svtoolkit-help] RedundancyAnnotator

[Bob H. <han...@br...>]

Hi, Thomas,

The RedundancyAnnotator is designed to work with VCFs generated from 
multiple tools (it was originally developed to help merge SVs from 
multiple callers are part of the 1000 Genomes project). It does require 
consistent annotations (e.g. all input files have genotype likelihoods) 
and the quality of the results will depend on how well-calibrated the 
likelihoods are. I haven't used it with lumpy, so your mileage may vary.

There are a couple of different approaches/modes you can use. The 
defaults are more designed for multiple calling methods on the same 
samples, which is what it sounds like you have. The tool compares all 
"nearby" variants pairwise, where "nearby" is usually determined by 
degree of overlap (default is 50% I believe, but you can change this). 
For the pairwise comparison, the default mode calculates that likelihood 
that any sample is more likely to have a different genotype than the 
same genotype, basically on-diagonal vs. off-diagonal (again, you can 
adjust the threshold). If no samples have sufficiently confidently 
different genotypes, then the two variants are deemed redundant. In this 
case, we want to filter one of the two redundant variants, and this is 
done by setting a filter on the variant with the smallest posterior 
genotype likelihoods (least confident genotype calls). The method 
attempts to compute a stable dominance order so that if there are 
multiple overlapping calls the minimal set is removed.

The default settings tend to produce rather "light" filtering, erring on 
the side of leaving calls unfiltered and only filtering those that are 
confidently quite similar. This may be appropriate for an association 
study, where you don't mind a few extra tests. If you are trying to 
create a reference map, you may get better results by turning up the 
thresholds. You can evaluate, for example, by looking at how many 
overlapping calls remain and the degree of overlap. If you turn the 
thresholds high enough, you can force the output to have no overlapping 
variants.

As an aside, the other application we use this for is to combine calls 
across disjoint sets of samples. In this case, we need more aggressive 
merging, so we change the settings to ignore the genotype likelihoods in 
the pairwise comparisons and just use the hard genotype calls and set a 
threshold on the allowable number of genotype discordances.

-Bob

On 10/3/17 8:09 AM, Thomas Faraut wrote:
> Dear GenomeSTRIP team,
>
> We used successfully genomeSTRIP to detect medium to large deletions 
> in goats.
> For smaller deletions, we use another variant detection tool (lumpy) but
> would like to be able to use the RedundancyAnnotator from the 
> svtoolkit to
> detect duplicate calls.
>
> Is it possible to use the RedundancyAnnotator with a vcf file provided by
> another SV genotyping tool provided that the genotype likelihoods are
> available ?
> Or is this redundancy score calculation described in one of the 
> genomeSTRIP
> paper ?
>
> Thank you in advance for your help.
>
> Best regards,
> Thomas Faraut
>

[svtoolkit-help] RedundancyAnnotator

[Thomas F. <Tho...@in...>]

Dear GenomeSTRIP team,

We used successfully genomeSTRIP to detect medium to large deletions in 
goats.
For smaller deletions, we use another variant detection tool (lumpy) but
would like to be able to use the RedundancyAnnotator from the svtoolkit to
detect duplicate calls.

Is it possible to use the RedundancyAnnotator with a vcf file provided by
another SV genotyping tool provided that the genotype likelihoods are
available ?
Or is this redundancy score calculation described in one of the genomeSTRIP
paper ?

Thank you in advance for your help.

Best regards,
Thomas Faraut

-- 
Thomas Faraut

New email address Tho...@in...

UMR 1388 INRA-INPT GenPhySE
Tel: +33 (0)5 61 28 54 57
http://genphyse.toulouse.inra.fr

Re: [svtoolkit-help] Error without description

[Bob H. <han...@br...>]

You shouldn't follow installtest as the example to run.  Remove:

-genomeMaskFile, -copyNumberMaskFile, -genderMapFile, 
-disableGATKTraversal, -useMultiStep, -reduceInsertSizeDistributions, 
-computeGCProfiles, -computeReadCounts and -P 
chimerism.use.correction:false.

Let these all default as appropriate (the mask files default based on -R).

Change -configFile to "-configFile ${SV_DIR}/conf/genstrip_parameters.txt".

The problem you reported may still remain, however, as it is likely 
something in your LSF environment such that Queue thinks some of the 
individual batch jobs have failed when in fact they have succeeded. This 
may be because the job is returning a non-zero exit status for some 
reason, or some quirk of the DRMAA api in your environment.
These problems can be hard to debug (sometimes this seems to happen more 
often with jobs that complete very quickly). Often, a sufficient 
workaround is to simply run the command again (it will retry just the 
jobs that previously failed). The jobs may run fine the second time around.

-Bob

On 9/25/17 1:42 AM, Elbay Aliyev wrote:
>
> Dear GenomeStrip team,
>
> My name is Elbay Aliyev. I am working as a Research Specialist at 
> Sidra Medical and Research Center. We have huge project of 3000 Qatari 
> Genome Project and we want to use genomestrip in our studies. But we 
> are facing unexplainable error without any stacktrace during 
> Preprocess step without description.
>
> **Preprocess script:
> **
> java -cp ${classpath} ${mx} \
> org.broadinstitute.gatk.queue.QCommandLine \
> -S ${SV_DIR}/qscript/SVPreprocess.q \
> -S ${SV_DIR}/qscript/SVQScript.q \
> -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
> --disableJobReport \
> -cp ${classpath} \
> -configFile conf/genstrip_installtest_parameters.txt \
> -tempDir ${SV_TMPDIR} \
> -R data/Homo_sapiens_assembly19.fasta \
> -genomeMaskFile data/Homo_sapiens_assembly19.svmask.fasta \
> -copyNumberMaskFile data/Homo_sapiens_assembly19.gcmask.fasta \
> -genderMapFile data/installtest_gender.map \
> -runDirectory ${runDir} \
> -md ${runDir}/metadata \
> -disableGATKTraversal \
> -useMultiStep \
> -reduceInsertSizeDistributions false \
> -computeGCProfiles true \
> -computeReadCounts true \
> -jobLogDir ${runDir}/logs \
> -I ${inputFile} \
> -P chimerism.use.correction:false \
> -run \
> || exit 1
>
> and a on a couple of steps we have errors like that.
>
>
> ERROR 16:09:20,504 FunctionEdge - Error: 'java' '-Xmx2048m' 
> '-XX:+UseParallelOldGC' '-XX:ParallelGCThreads=4' '-XX:GCTimeLimit=50' 
> '-XX:GCHeapFreeLimit=10' 
> '-Djava.io.tmpdir=/gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/tmpdir' 
> '-cp' 
> '/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/SVToolkit.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/GenomeAnalysisTK.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/Queue.jar' 
> '-cp' 
> '/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/SVToolkit.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/GenomeAnalysisTK.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/Queue.jar' 
> 'org.broadinstitute.sv.apps.ComputeDepthProfiles' '-O' 
> '/gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/metadata/profiles_100Kb/profile_seq_GL000232.1_100000.dat.gz' 
> '-I' 'PMC01/metadata/headers.bam' '-configFile' 
> 'conf/genstrip_installtest_parameters.txt' '-P' 
> 'chimerism.use.correction:false' '-R' 
> 'data/Homo_sapiens_assembly19.fasta' '-L' 'GL000232.1:0-0' 
> '-genomeMaskFile' 'data/Homo_sapiens_assembly19.svmask.fasta' '-md' 
> 'PMC01/metadata' '-profileBinSize' '100000' 
> '-maximumReferenceGapLength' '10000'
> ERROR 16:09:20,509 FunctionEdge - Contents of 
> /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/logs/SVPreprocess-111.out:
> INFO 16:04:10,442 HelpFormatter - 
> -------------------------------------------------------------
> INFO 16:04:10,445 HelpFormatter - Program Name: 
> org.broadinstitute.sv.apps.ComputeDepthProfiles
> INFO 16:04:10,449 HelpFormatter - Program Args: -O 
> /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/metadata/profiles_100Kb/profile_seq_GL000232.1_100000.dat.gz 
> -I PMC01/metadata/headers.bam -configFile 
> conf/genstrip_installtest_parameters.txt -P 
> chimerism.use.correction:false -R data/Homo_sapiens_assembly19.fasta 
> -L GL000232.1:0-0 -genomeMaskFile 
> data/Homo_sapiens_assembly19.svmask.fasta -md PMC01/metadata 
> -profileBinSize 100000 -maximumReferenceGapLength 10000
> INFO 16:04:10,453 HelpFormatter - Executing as 
> ea...@hp...dra.local on Linux 
> 3.10.0-229.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 
> 1.8.0_121-b13.
> INFO 16:04:10,454 HelpFormatter - Date/Time: 2017/09/24 16:04:10
> INFO 16:04:10,454 HelpFormatter - 
> -------------------------------------------------------------
> INFO 16:04:10,454 HelpFormatter - 
> -------------------------------------------------------------
> INFO 16:04:10,461 ComputeDepthProfiles - Opening reference sequence ...
> INFO 16:04:10,462 ComputeDepthProfiles - Opened reference sequence.
> INFO 16:04:10,462 ComputeDepthProfiles - Opening genome mask ...
> INFO 16:04:10,463 ComputeDepthProfiles - Opened genome mask.
> INFO 16:04:10,465 MetaData - Opening metadata ...
> INFO 16:04:10,466 MetaData - Adding metadata location PMC01/metadata ...
> INFO 16:04:10,476 MetaData - Opened metadata.
> INFO 16:04:10,476 ComputeDepthProfiles - Opened metadata.
> INFO 16:04:10,476 ComputeDepthProfiles - Initializing input data set ...
> INFO 16:04:10,513 ComputeDepthProfiles - Initialized data set: 1 file, 
> 1 read group, 1 sample.
> INFO 16:04:10,518 MetaData - Loading insert size histograms ...
> INFO 16:04:11,940 ReadCountCache - Initializing read count cache with 
> 1 file.
>
> INFO 16:04:12,010 CommandLineProgram - Program completed.
>
> /Done. There were no warn messages./
>
> /Looks like script did his job perfect but still gives error. Same 
> issue with SVPreprocess 6 and 11 :(/
>
> echo "./discovery.sh" | bsub -n 16 -e 
> /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/test.err -o 
> /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/test.out -P PMC01.test
>
> We are submitting to our LSF system like that. our LSF version is 
> 10.1. Java version is 1.8.21. GATK we are using internal one provided 
> with GATK.
>
> Thanks in advance.
>
> *Elbay Aliyev*
>
> Software Engineer
>
> *Sidra Medical and Research Center***
>
> PO Box 26999 *| *Doha, Qatar
>
> Direct: 40037393
>
> ea...@si... <mailto:ea...@si...>*|* www.sidra.org 
> <http://www.sidra.org/>
>
> cid:image001.jpg@01CFB54A.69FD9EB0
>
> *P****Please consider the environment before printing this email.*
>
> Disclaimer: This email and its attachments may be confidential and are 
> intended solely for the use of the individual to whom it is addressed. 
> If you are not the intended recipient, any reading, printing, storage, 
> disclosure, copying or any other action taken in respect of this 
> e-mail is prohibited and may be unlawful. If you are not the intended 
> recipient, please notify the sender immediately by using the reply 
> function and then permanently delete what you have received. Any views 
> or opinions expressed are solely those of the author and do not 
> necessarily represent those of Sidra Medical and Research Center.
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> svtoolkit-help mailing list
> svt...@li...
> https://lists.sourceforge.net/lists/listinfo/svtoolkit-help

[svtoolkit-help] Error without description

[Elbay A. <ea...@si...>]

Dear GenomeStrip team,
My name is Elbay Aliyev. I am working as a Research Specialist at Sidra Medical and Research Center. We have huge project of 3000 Qatari Genome Project and we want to use genomestrip in our studies. But we are facing unexplainable error without any stacktrace during Preprocess step without description.
**Preprocess script:
**
java -cp ${classpath} ${mx} \
org.broadinstitute.gatk.queue.QCommandLine \
-S ${SV_DIR}/qscript/SVPreprocess.q \
-S ${SV_DIR}/qscript/SVQScript.q \
-gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
--disableJobReport \
-cp ${classpath} \
-configFile conf/genstrip_installtest_parameters.txt \
-tempDir ${SV_TMPDIR} \
-R data/Homo_sapiens_assembly19.fasta \
-genomeMaskFile data/Homo_sapiens_assembly19.svmask.fasta \
-copyNumberMaskFile data/Homo_sapiens_assembly19.gcmask.fasta \
-genderMapFile data/installtest_gender.map \
-runDirectory ${runDir} \
-md ${runDir}/metadata \
-disableGATKTraversal \
-useMultiStep \
-reduceInsertSizeDistributions false \
-computeGCProfiles true \
-computeReadCounts true \
-jobLogDir ${runDir}/logs \
-I ${inputFile} \
-P chimerism.use.correction:false \
-run \
|| exit 1
and a on a couple of steps we have errors like that.

ERROR 16:09:20,504 FunctionEdge - Error: 'java' '-Xmx2048m' '-XX:+UseParallelOldGC' '-XX:ParallelGCThreads=4' '-XX:GCTimeLimit=50' '-XX:GCHeapFreeLimit=10' '-Djava.io.tmpdir=/gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/tmpdir' '-cp' '/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/SVToolkit.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/GenomeAnalysisTK.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/Queue.jar' '-cp' '/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/SVToolkit.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/GenomeAnalysisTK.jar:/gpfs/projects/tmedicine/ealiyev/genomestrip/lib/gatk/Queue.jar' 'org.broadinstitute.sv.apps.ComputeDepthProfiles' '-O' '/gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/metadata/profiles_100Kb/profile_seq_GL000232.1_100000.dat.gz' '-I' 'PMC01/metadata/headers.bam' '-configFile' 'conf/genstrip_installtest_parameters.txt' '-P' 'chimerism.use.correction:false' '-R' 'data/Homo_sapiens_assembly19.fasta' '-L' 'GL000232.1:0-0' '-genomeMaskFile' 'data/Homo_sapiens_assembly19.svmask.fasta' '-md' 'PMC01/metadata' '-profileBinSize' '100000' '-maximumReferenceGapLength' '10000'
ERROR 16:09:20,509 FunctionEdge - Contents of /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/logs/SVPreprocess-111.out:
INFO 16:04:10,442 HelpFormatter - -------------------------------------------------------------
INFO 16:04:10,445 HelpFormatter - Program Name: org.broadinstitute.sv.apps.ComputeDepthProfiles
INFO 16:04:10,449 HelpFormatter - Program Args: -O /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/PMC01/metadata/profiles_100Kb/profile_seq_GL000232.1_100000.dat.gz -I PMC01/metadata/headers.bam -configFile conf/genstrip_installtest_parameters.txt -P chimerism.use.correction:false -R data/Homo_sapiens_assembly19.fasta -L GL000232.1:0-0 -genomeMaskFile data/Homo_sapiens_assembly19.svmask.fasta -md PMC01/metadata -profileBinSize 100000 -maximumReferenceGapLength 10000
INFO 16:04:10,453 HelpFormatter - Executing as ea...@hp...dra.local on Linux 3.10.0-229.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13.
INFO 16:04:10,454 HelpFormatter - Date/Time: 2017/09/24 16:04:10
INFO 16:04:10,454 HelpFormatter - -------------------------------------------------------------
INFO 16:04:10,454 HelpFormatter - -------------------------------------------------------------
INFO 16:04:10,461 ComputeDepthProfiles - Opening reference sequence ...
INFO 16:04:10,462 ComputeDepthProfiles - Opened reference sequence.
INFO 16:04:10,462 ComputeDepthProfiles - Opening genome mask ...
INFO 16:04:10,463 ComputeDepthProfiles - Opened genome mask.
INFO 16:04:10,465 MetaData - Opening metadata ...
INFO 16:04:10,466 MetaData - Adding metadata location PMC01/metadata ...
INFO 16:04:10,476 MetaData - Opened metadata.
INFO 16:04:10,476 ComputeDepthProfiles - Opened metadata.
INFO 16:04:10,476 ComputeDepthProfiles - Initializing input data set ...
INFO 16:04:10,513 ComputeDepthProfiles - Initialized data set: 1 file, 1 read group, 1 sample.
INFO 16:04:10,518 MetaData - Loading insert size histograms ...
INFO 16:04:11,940 ReadCountCache - Initializing read count cache with 1 file.
INFO 16:04:12,010 CommandLineProgram - Program completed.
Done. There were no warn messages.
Looks like script did his job perfect but still gives error. Same issue with SVPreprocess 6 and 11 [:(]
echo "./discovery.sh" | bsub -n 16 -e /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/test.err -o /gpfs/projects/tmedicine/ealiyev/genomestrip/PMC01/test.out -P PMC01.test
We are submitting to our LSF system like that. our LSF version is 10.1. Java version is 1.8.21. GATK we are using internal one provided with GATK.
Thanks in advance.

Elbay Aliyev
Software Engineer

Sidra Medical and Research Center
PO Box 26999 | Doha, Qatar
Direct: 40037393
ea...@si...<mailto:ea...@si...> | www.sidra.org<http://www.sidra.org/>

[cid:image001.jpg@01CFB54A.69FD9EB0]
P Please consider the environment before printing this email.

Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.

[svtoolkit-help] svtoolkit with prefix java

[<mc...@na...>]

Hi Bob,
I'm running svtoolkit on linux server machine where I cannot be a root.
So I need to use a java installed at another path.
The machine has java1.6 already installed, and it seems svtoolkit uses this java instead of what I installed customely by setting java as just java.
Which is not the java path I put in.
INFO  10:37:53,469 QGraph - Failed:   'java'  '-Xmx2048m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/scratch2/x1477hyk/SVanalysis/.queue/tmp'  '-cp' '/scratch2/x1477hyk/svtoolkit/lib/SVToolkit.jar:/scratch2/x1477hyk/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/scratch2/x1477hyk/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/scratch2/x1477hyk/svtoolkit/lib/SVToolkit.jar:/scratch2/x1477hyk/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/scratch2/x1477hyk/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.sv.apps.CreateMetaDataDirectory'  '-md' 'output_metadata_directory'  '-configFile' '/scratch2/x1477hyk/svtoolkit/conf/genstrip_parameters.txt'

so I get 
INFO  10:37:53,470 QCommandLine - Script failed: 6752 Pend, 0 Run, 2 Fail, 0 Done
is there any way to force svtoolkit to use my java? or other solution to fix this problem?
Thanks,Seungmin Kim

Re: [svtoolkit-help] GenomeStrip Error Invalid Interval HLA-A*01:01:01:01 SVPreprocess

[Seva K. <sk...@br...>]

Hi Martin,

  Can you post the script that invokes the preprocessing, and also the full
stack trace of the exception you are getting?
  And what is in your file interval.list? Is it in the same directory as
the shell script?

Seva

[svtoolkit-help] GenomeStrip Error Invalid Interval HLA-A*01:01:01:01 SVPreprocess

[mschwarz <m.s...@me...>]

Hi,

i try to run the lattest version of SVtools with the lattest 
reference/mask for HG38 from your server. When I start the 
SVPreprocessing script I get the following error:

java.lang.IllegalArgumentException: Invalid Interval HLA-A*01:01:01:01

It seems that the SVPreprocess-Script have problems with the additional 
contigs in HG38.

I try a workaround with the -L interval.list argument but struggle into 
an other error latter in the script.
(java.lang.RuntimeException: Unrecognized sequence: intervals.list)

Install-Test runs fine.

Hope you can help me.

Thanks

Martin Schwarz