Menu
▾
▴
wgs-assembler-users — Discussion about Celera Assembler
You can subscribe to this list here.
| 2012 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
(29) |
May
(8) |
Jun
(5) |
Jul
(46) |
Aug
(16) |
Sep
(5) |
Oct
(6) |
Nov
(17) |
Dec
(7) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2013 |
Jan
(5) |
Feb
(2) |
Mar
(10) |
Apr
(13) |
May
(20) |
Jun
(7) |
Jul
(6) |
Aug
(14) |
Sep
(9) |
Oct
(19) |
Nov
(17) |
Dec
(3) |
| 2014 |
Jan
(3) |
Feb
|
Mar
(7) |
Apr
(1) |
May
(1) |
Jun
(30) |
Jul
(10) |
Aug
(2) |
Sep
(18) |
Oct
(3) |
Nov
(4) |
Dec
(13) |
| 2015 |
Jan
(27) |
Feb
|
Mar
(19) |
Apr
(12) |
May
(10) |
Jun
(18) |
Jul
(4) |
Aug
(2) |
Sep
(2) |
Oct
|
Nov
(1) |
Dec
(9) |
| 2016 |
Jan
(6) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
(1) |
Sep
(1) |
Oct
|
Nov
|
Dec
|
|
From: Brian W. <th...@gm...> - 2014-06-19 00:16:20
|
Unfortunately, I'm on vacation at the moment, and finding little time to spend helping you. "Too many open files" is a limit imposed by the OS. Can you increase this? We've set our large memory machines to allow 100,000 open files. The output files sizes -- and the problem you're suffering from -- are all caused by the way overlaps are created. Correction asked for only overlaps between Illumina and PacBio reads. All the illumina reads are 'first' in the store, and all the pacbio reads are at the end. Overlap jobs will find overlaps between 'other' reads and some subset of the store - e.g., the first overlap job will process the first 10% of the reads, the second will do the second 10% of the reads, etc. Since the pacbio are last, the last job found all the overlaps, so only the last file is of significant size. This also breaks the partitioning scheme used when sorting overlaps. It assumes overlaps are distributed randomly, but yours are all piled up at the end. I don't see an easy fix here, but I think I can come up with a one-off hack to get your store built. Are you comfortable working with C code and compiling? Send the output of 'gatekeeper -dumpinfo *gkpStore' so I can see the number of reads per library. On Tue, Jun 17, 2014 at 6:45 PM, Santiago Revale <san...@gm...> wrote: > Hi Brian, > > When using 1024, it said the OS wasn't able to handle it, and it > recommended using 1008. > When using 1008, CA ended arguing "Failed to open output file... Too many > open files". > > Now I'm trying with fewer parts, but I don't think this would solve the > problem. > > Do you have any more ideas? > > Thanks again in advance. > > Regards, > Santiago > > > On Sun, Jun 15, 2014 at 10:10 PM, Santiago Revale < > san...@gm...> wrote: > >> Hi Brian, >> >> Thanks for your reply. In regards of your suggestions: >> >> 1) the PBcR process generates OVB files without zipping them; just to be >> sure, I've tried to unzip some of them just in case the extension were >> missing; >> >> 2) I've re-launched the process with the suggested parameters, but using >> 512 instead of 1024; the result was exactly the same: same error in the >> same step. Also, again 511 out of 512 files had a size of 2.3Gb while the >> last file was 1.2Tb long. Do you know why does this happens? >> >> I'm trying one last time using 1024 instead. >> >> Thanks again for your reply. I'm open to some more suggestions. >> >> Regards, >> Santiago >> >> >> On Fri, Jun 13, 2014 at 4:25 PM, Brian Walenz <th...@gm...> wrote: >> >>> Hi- >>> >>> This is a flaw in gzip, where it doesn't report the uncompressed size >>> correctly for files larger than 2gb. I'm not intimately familiar with this >>> pipeline, so don't know exactly how to implement the fixes below. >>> >>> Fix with either: >>> >>> 1) gzip -d the *gz files before building the overlap store. The 'find' >>> command in the log indicates the pipeline will pick up the uncompressed >>> files. You might need to remove the 'asm.ovlStore.list' file before >>> restarting (this has the list of inputs to overlapStoreBuild). >>> >>> 2) Set ovlStoreMemory to (exactly) "0 -f 1024". This will tell it to >>> use 0MB memory, and instead use 1024 files regardless of the size. 512 >>> files will also work, and is a little safer (not near some Linux 'number of >>> open files' limits). >>> >>> 3) Build the overlap store by hand (with either the uncompressed input, >>> or the -f instead of -M option), outside the script, and then restart the >>> script. The script will notice there is an overlap store already present, >>> and skip the build. The command is in the log file -- make sure the final >>> store is called 'asm.ovlStore', and not 'asm.ovlStore.BUILDING'. >>> >>> Option 1 should work, but option 2 is the easiest to try. I wouldn't >>> try option 3 until Sergey speaks up. >>> >>> b >>> >>> >>> >>> >>> On Fri, Jun 13, 2014 at 12:33 PM, Santiago Revale < >>> san...@gm...> wrote: >>> >>>> Dear CA community, >>>> >>>> I'm running the correction of some PacBio reads with high-identity >>>> Illumina reads, in a high memory server, for a 750 Mbp genome. I've >>>> considered the known issues addressed in the website when starting the >>>> correction. >>>> >>>> When executing the pipeline, I've reached to the overlapStoreBuild step >>>> with 48 ovb files, size 26 Gb each (totaling 1.2Tb). ovls files have >>>> already been deleted by the script. The error happened while executing >>>> overlapStoreBuild: >>>> >>>> ... >>>> bucketizing DONE! >>>> overlaps skipped: >>>> 0 OBT - low quality >>>> 0 DUP - non-duplicate overlap >>>> 0 DUP - different library >>>> 0 DUP - dedup not requested >>>> terminate called after throwing an instance of 'std::bad_alloc' >>>> what(): std::bad_alloc >>>> >>>> Failed with 'Aborted' >>>> ... >>>> >>>> >>>> I ran this step twice: the first one having set ovlStoreMemory to 8192 >>>> Mb, but the second one, set it on 160000 (160 Gb). In the "Overlap store >>>> failure" FAQ, it mentioned as possible causes "Out of disk space" (which is >>>> not my case) and "Corrupt gzip files / too many fragments". I don't have >>>> gzip files and I have only 15 fragments. Also, bucketizing step finishes OK. >>>> >>>> Also, some odd thing I've noticed (at least odd for me) is that 14 of >>>> the 15 temp files (tmp.sort.XXX) of the asm.ovlStore.BUILDING folder have a >>>> size 79Gb while the last one size is 1.2Tb. >>>> >>>> Could anybody tell me what could be the cause of this error and how to >>>> solve it? >>>> >>>> I'm attaching the asm.ovlStore.err and the pacBioToCA log files for >>>> complete descriptions of the error and the executed commands. >>>> >>>> Thank you very much in advance. >>>> >>>> Regards, >>>> Santiago >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk >>>> Solutions >>>> Find What Matters Most in Your Big Data with HPCC Systems >>>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. >>>> Leverages Graph Analysis for Fast Processing & Easy Data Exploration >>>> http://p.sf.net/sfu/hpccsystems >>>> _______________________________________________ >>>> wgs-assembler-users mailing list >>>> wgs...@li... >>>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users >>>> >>>> >>> >> > |
|
From: Santiago R. <san...@gm...> - 2014-06-17 22:45:30
|
Hi Brian, When using 1024, it said the OS wasn't able to handle it, and it recommended using 1008. When using 1008, CA ended arguing "Failed to open output file... Too many open files". Now I'm trying with fewer parts, but I don't think this would solve the problem. Do you have any more ideas? Thanks again in advance. Regards, Santiago On Sun, Jun 15, 2014 at 10:10 PM, Santiago Revale <san...@gm...> wrote: > Hi Brian, > > Thanks for your reply. In regards of your suggestions: > > 1) the PBcR process generates OVB files without zipping them; just to be > sure, I've tried to unzip some of them just in case the extension were > missing; > > 2) I've re-launched the process with the suggested parameters, but using > 512 instead of 1024; the result was exactly the same: same error in the > same step. Also, again 511 out of 512 files had a size of 2.3Gb while the > last file was 1.2Tb long. Do you know why does this happens? > > I'm trying one last time using 1024 instead. > > Thanks again for your reply. I'm open to some more suggestions. > > Regards, > Santiago > > > On Fri, Jun 13, 2014 at 4:25 PM, Brian Walenz <th...@gm...> wrote: > >> Hi- >> >> This is a flaw in gzip, where it doesn't report the uncompressed size >> correctly for files larger than 2gb. I'm not intimately familiar with this >> pipeline, so don't know exactly how to implement the fixes below. >> >> Fix with either: >> >> 1) gzip -d the *gz files before building the overlap store. The 'find' >> command in the log indicates the pipeline will pick up the uncompressed >> files. You might need to remove the 'asm.ovlStore.list' file before >> restarting (this has the list of inputs to overlapStoreBuild). >> >> 2) Set ovlStoreMemory to (exactly) "0 -f 1024". This will tell it to use >> 0MB memory, and instead use 1024 files regardless of the size. 512 files >> will also work, and is a little safer (not near some Linux 'number of open >> files' limits). >> >> 3) Build the overlap store by hand (with either the uncompressed input, >> or the -f instead of -M option), outside the script, and then restart the >> script. The script will notice there is an overlap store already present, >> and skip the build. The command is in the log file -- make sure the final >> store is called 'asm.ovlStore', and not 'asm.ovlStore.BUILDING'. >> >> Option 1 should work, but option 2 is the easiest to try. I wouldn't try >> option 3 until Sergey speaks up. >> >> b >> >> >> >> >> On Fri, Jun 13, 2014 at 12:33 PM, Santiago Revale < >> san...@gm...> wrote: >> >>> Dear CA community, >>> >>> I'm running the correction of some PacBio reads with high-identity >>> Illumina reads, in a high memory server, for a 750 Mbp genome. I've >>> considered the known issues addressed in the website when starting the >>> correction. >>> >>> When executing the pipeline, I've reached to the overlapStoreBuild step >>> with 48 ovb files, size 26 Gb each (totaling 1.2Tb). ovls files have >>> already been deleted by the script. The error happened while executing >>> overlapStoreBuild: >>> >>> ... >>> bucketizing DONE! >>> overlaps skipped: >>> 0 OBT - low quality >>> 0 DUP - non-duplicate overlap >>> 0 DUP - different library >>> 0 DUP - dedup not requested >>> terminate called after throwing an instance of 'std::bad_alloc' >>> what(): std::bad_alloc >>> >>> Failed with 'Aborted' >>> ... >>> >>> >>> I ran this step twice: the first one having set ovlStoreMemory to 8192 >>> Mb, but the second one, set it on 160000 (160 Gb). In the "Overlap store >>> failure" FAQ, it mentioned as possible causes "Out of disk space" (which is >>> not my case) and "Corrupt gzip files / too many fragments". I don't have >>> gzip files and I have only 15 fragments. Also, bucketizing step finishes OK. >>> >>> Also, some odd thing I've noticed (at least odd for me) is that 14 of >>> the 15 temp files (tmp.sort.XXX) of the asm.ovlStore.BUILDING folder have a >>> size 79Gb while the last one size is 1.2Tb. >>> >>> Could anybody tell me what could be the cause of this error and how to >>> solve it? >>> >>> I'm attaching the asm.ovlStore.err and the pacBioToCA log files for >>> complete descriptions of the error and the executed commands. >>> >>> Thank you very much in advance. >>> >>> Regards, >>> Santiago >>> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions >>> Find What Matters Most in Your Big Data with HPCC Systems >>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. >>> Leverages Graph Analysis for Fast Processing & Easy Data Exploration >>> http://p.sf.net/sfu/hpccsystems >>> _______________________________________________ >>> wgs-assembler-users mailing list >>> wgs...@li... >>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users >>> >>> >> > |
|
From: Miguel G. <mi...@uj...> - 2014-06-16 03:18:10
|
Hi Serge, As you said, seems that we don't have enough overlaps to create the fasta output file. I have checked the coverage from Illumina and Pacbio and there is an average of 20% or less. Thanks for your help! On 2014?06?12? 00:35, Serge Koren wrote: > It looks like almost no overlaps were computed for your dataset. You > have 1M reads and only 280K overlaps so less than 1 overlap per read. > Is this data you are able to share to help diagnose what is going on? > Otherwise, can you give more details on your dataset, how much > coverage of Illumina data and PacBio data do you have? > On Jun 10, 2014, at 10:46 PM, Miguel Grau <mi...@uj... > <mailto:mi...@uj...>> wrote: > >> Computed mean 1.250417 sd 0.612309 using min value 1 (1.000000) max 1 >> (1.000000) > |
|
From: Santiago R. <san...@gm...> - 2014-06-16 01:11:06
|
Hi Brian, Thanks for your reply. In regards of your suggestions: 1) the PBcR process generates OVB files without zipping them; just to be sure, I've tried to unzip some of them just in case the extension were missing; 2) I've re-launched the process with the suggested parameters, but using 512 instead of 1024; the result was exactly the same: same error in the same step. Also, again 511 out of 512 files had a size of 2.3Gb while the last file was 1.2Tb long. Do you know why does this happens? I'm trying one last time using 1024 instead. Thanks again for your reply. I'm open to some more suggestions. Regards, Santiago Slds Santiago On Fri, Jun 13, 2014 at 4:25 PM, Brian Walenz <th...@gm...> wrote: > Hi- > > This is a flaw in gzip, where it doesn't report the uncompressed size > correctly for files larger than 2gb. I'm not intimately familiar with this > pipeline, so don't know exactly how to implement the fixes below. > > Fix with either: > > 1) gzip -d the *gz files before building the overlap store. The 'find' > command in the log indicates the pipeline will pick up the uncompressed > files. You might need to remove the 'asm.ovlStore.list' file before > restarting (this has the list of inputs to overlapStoreBuild). > > 2) Set ovlStoreMemory to (exactly) "0 -f 1024". This will tell it to use > 0MB memory, and instead use 1024 files regardless of the size. 512 files > will also work, and is a little safer (not near some Linux 'number of open > files' limits). > > 3) Build the overlap store by hand (with either the uncompressed input, or > the -f instead of -M option), outside the script, and then restart the > script. The script will notice there is an overlap store already present, > and skip the build. The command is in the log file -- make sure the final > store is called 'asm.ovlStore', and not 'asm.ovlStore.BUILDING'. > > Option 1 should work, but option 2 is the easiest to try. I wouldn't try > option 3 until Sergey speaks up. > > b > > > > > On Fri, Jun 13, 2014 at 12:33 PM, Santiago Revale < > san...@gm...> wrote: > >> Dear CA community, >> >> I'm running the correction of some PacBio reads with high-identity >> Illumina reads, in a high memory server, for a 750 Mbp genome. I've >> considered the known issues addressed in the website when starting the >> correction. >> >> When executing the pipeline, I've reached to the overlapStoreBuild step >> with 48 ovb files, size 26 Gb each (totaling 1.2Tb). ovls files have >> already been deleted by the script. The error happened while executing >> overlapStoreBuild: >> >> ... >> bucketizing DONE! >> overlaps skipped: >> 0 OBT - low quality >> 0 DUP - non-duplicate overlap >> 0 DUP - different library >> 0 DUP - dedup not requested >> terminate called after throwing an instance of 'std::bad_alloc' >> what(): std::bad_alloc >> >> Failed with 'Aborted' >> ... >> >> >> I ran this step twice: the first one having set ovlStoreMemory to 8192 >> Mb, but the second one, set it on 160000 (160 Gb). In the "Overlap store >> failure" FAQ, it mentioned as possible causes "Out of disk space" (which is >> not my case) and "Corrupt gzip files / too many fragments". I don't have >> gzip files and I have only 15 fragments. Also, bucketizing step finishes OK. >> >> Also, some odd thing I've noticed (at least odd for me) is that 14 of the >> 15 temp files (tmp.sort.XXX) of the asm.ovlStore.BUILDING folder have a >> size 79Gb while the last one size is 1.2Tb. >> >> Could anybody tell me what could be the cause of this error and how to >> solve it? >> >> I'm attaching the asm.ovlStore.err and the pacBioToCA log files for >> complete descriptions of the error and the executed commands. >> >> Thank you very much in advance. >> >> Regards, >> Santiago >> >> >> >> >> ------------------------------------------------------------------------------ >> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions >> Find What Matters Most in Your Big Data with HPCC Systems >> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. >> Leverages Graph Analysis for Fast Processing & Easy Data Exploration >> http://p.sf.net/sfu/hpccsystems >> _______________________________________________ >> wgs-assembler-users mailing list >> wgs...@li... >> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users >> >> > |
|
From: Brian W. <th...@gm...> - 2014-06-13 19:25:44
|
Hi- This is a flaw in gzip, where it doesn't report the uncompressed size correctly for files larger than 2gb. I'm not intimately familiar with this pipeline, so don't know exactly how to implement the fixes below. Fix with either: 1) gzip -d the *gz files before building the overlap store. The 'find' command in the log indicates the pipeline will pick up the uncompressed files. You might need to remove the 'asm.ovlStore.list' file before restarting (this has the list of inputs to overlapStoreBuild). 2) Set ovlStoreMemory to (exactly) "0 -f 1024". This will tell it to use 0MB memory, and instead use 1024 files regardless of the size. 512 files will also work, and is a little safer (not near some Linux 'number of open files' limits). 3) Build the overlap store by hand (with either the uncompressed input, or the -f instead of -M option), outside the script, and then restart the script. The script will notice there is an overlap store already present, and skip the build. The command is in the log file -- make sure the final store is called 'asm.ovlStore', and not 'asm.ovlStore.BUILDING'. Option 1 should work, but option 2 is the easiest to try. I wouldn't try option 3 until Sergey speaks up. b On Fri, Jun 13, 2014 at 12:33 PM, Santiago Revale <san...@gm...> wrote: > Dear CA community, > > I'm running the correction of some PacBio reads with high-identity > Illumina reads, in a high memory server, for a 750 Mbp genome. I've > considered the known issues addressed in the website when starting the > correction. > > When executing the pipeline, I've reached to the overlapStoreBuild step > with 48 ovb files, size 26 Gb each (totaling 1.2Tb). ovls files have > already been deleted by the script. The error happened while executing > overlapStoreBuild: > > ... > bucketizing DONE! > overlaps skipped: > 0 OBT - low quality > 0 DUP - non-duplicate overlap > 0 DUP - different library > 0 DUP - dedup not requested > terminate called after throwing an instance of 'std::bad_alloc' > what(): std::bad_alloc > > Failed with 'Aborted' > ... > > > I ran this step twice: the first one having set ovlStoreMemory to 8192 Mb, > but the second one, set it on 160000 (160 Gb). In the "Overlap store > failure" FAQ, it mentioned as possible causes "Out of disk space" (which is > not my case) and "Corrupt gzip files / too many fragments". I don't have > gzip files and I have only 15 fragments. Also, bucketizing step finishes OK. > > Also, some odd thing I've noticed (at least odd for me) is that 14 of the > 15 temp files (tmp.sort.XXX) of the asm.ovlStore.BUILDING folder have a > size 79Gb while the last one size is 1.2Tb. > > Could anybody tell me what could be the cause of this error and how to > solve it? > > I'm attaching the asm.ovlStore.err and the pacBioToCA log files for > complete descriptions of the error and the executed commands. > > Thank you very much in advance. > > Regards, > Santiago > > > > > ------------------------------------------------------------------------------ > HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions > Find What Matters Most in Your Big Data with HPCC Systems > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. > Leverages Graph Analysis for Fast Processing & Easy Data Exploration > http://p.sf.net/sfu/hpccsystems > _______________________________________________ > wgs-assembler-users mailing list > wgs...@li... > https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users > > |
|
From: Santiago R. <san...@gm...> - 2014-06-13 16:34:09
|
Dear CA community,
I'm running the correction of some PacBio reads with high-identity Illumina
reads, in a high memory server, for a 750 Mbp genome. I've considered the
known issues addressed in the website when starting the correction.
When executing the pipeline, I've reached to the overlapStoreBuild step
with 48 ovb files, size 26 Gb each (totaling 1.2Tb). ovls files have
already been deleted by the script. The error happened while executing
overlapStoreBuild:
...
bucketizing DONE!
overlaps skipped:
0 OBT - low quality
0 DUP - non-duplicate overlap
0 DUP - different library
0 DUP - dedup not requested
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Failed with 'Aborted'
...
I ran this step twice: the first one having set ovlStoreMemory to 8192 Mb,
but the second one, set it on 160000 (160 Gb). In the "Overlap store
failure" FAQ, it mentioned as possible causes "Out of disk space" (which is
not my case) and "Corrupt gzip files / too many fragments". I don't have
gzip files and I have only 15 fragments. Also, bucketizing step finishes OK.
Also, some odd thing I've noticed (at least odd for me) is that 14 of the
15 temp files (tmp.sort.XXX) of the asm.ovlStore.BUILDING folder have a
size 79Gb while the last one size is 1.2Tb.
Could anybody tell me what could be the cause of this error and how to
solve it?
I'm attaching the asm.ovlStore.err and the pacBioToCA log files for
complete descriptions of the error and the executed commands.
Thank you very much in advance.
Regards,
Santiago
|
|
From: Serge K. <ser...@gm...> - 2014-06-11 15:35:48
|
It looks like almost no overlaps were computed for your dataset. You have 1M reads and only 280K overlaps so less than 1 overlap per read. Is this data you are able to share to help diagnose what is going on? Otherwise, can you give more details on your dataset, how much coverage of Illumina data and PacBio data do you have? On Jun 10, 2014, at 10:46 PM, Miguel Grau <mi...@uj...> wrote: > Computed mean 1.250417 sd 0.612309 using min value 1 (1.000000) max 1 (1.000000) |
|
From: Miguel G. <mi...@uj...> - 2014-06-11 02:53:02
|
Hi Sergey,
This is the content of asm.layout.err:
Parsing arguments
Starting correctPacBio, running from /apps/wgs-8.1/Linux-amd64/bin/
The cutoff is set to be 0
Opening stores
Loading library information
Correcting fragments
Streaming fragments
Each thread responsible for 145 (16 threads) and each file 12 of 2335
fragments
Starting in thread 3 from 1000435 to 1000579
Starting in thread 2 from 1000290 to 1000434
Starting in thread 1 from 1000145 to 1000289
Starting in thread 4 from 1000580 to 1000724
Starting in thread 5 from 1000725 to 1000869
Starting in thread 6 from 1000870 to 1001014
Starting in thread 7 from 1001015 to 1001159
Starting in thread 8 from 1001160 to 1001304
Starting in thread 9 from 1001305 to 1001449
Starting in thread 10 from 1001450 to 1001594
In thread 4 going to output files 49-60 with 12
In thread 5 going to output files 61-72 with 12
Starting in thread 11 from 1001595 to 1001739
Starting in thread 12 from 1001740 to 1001884
Starting in thread 13 from 1001885 to 1002029
Starting in thread 14 from 1002030 to 1002174
In thread 3 going to output files 37-48 with 12
In thread 2 going to output files 25-36 with 12
Starting in thread 0 from 1000001 to 1000144
Starting in thread 15 from 1002175 to 1002335
In thread 1 going to output files 13-24 with 12
In thread 7 going to output files 85-96 with 12
In thread 8 going to output files 97-108 with 12
In thread 0 going to output files 1-12 with 12
In thread 6 going to output files 73-84 with 12
In thread 10 going to output files 121-132 with 12
In thread 9 going to output files 109-120 with 12
In thread 13 going to output files 158-169 with 12
In thread 11 going to output files 133-144 with 12
In thread 12 going to output files 145-157 with 12
In thread 14 going to output files 170-181 with 12
In thread 15 going to output files 182-194 with 12
Computed mean 1.250417 sd 0.612309 using min value 1 (1.000000) max 1
(1.000000)
After quartile adjustment mean is 1.000000 sd: 0.000000
Picking cutoff as 1 mean would be 1.000000 +- 0.000000 (1)
Filtering repeats
Storing 5244 fragments out of 1000002
Loaded 140803 overlaps
Cleaning up
And asm. ovlStore.err:
Scanning overlap files to count the number of overlaps.
Found 0.282 million overlaps.
Memory limit 8192MB supplied. I'll put 1528731753 IIDs (429.50 million
overlaps) into each of 1 buck$
bucketizing /apps/wgs-8.1/DATA/tempburnin2/./1-overlapper/001/000001.ovb.gz
bucketizing DONE!
overlaps skipped:
0 OBT - low quality
0 DUP - non-duplicate overlap
0 DUP - different library
0 DUP - dedup not requested
reading
/apps/wgs-8.1/DATA/tempburnin2/./asm.ovlStore.BUILDING/tmp.sort.000 (0)
sorting
/apps/wgs-8.1/DATA/tempburnin2/./asm.ovlStore.BUILDING/tmp.sort.000 (0)
writing
/apps/wgs-8.1/DATA/tempburnin2/./asm.ovlStore.BUILDING/tmp.sort.000 (0)
Closing the new store:
ovs->ovs.ovsMagic = 0x65726f74536c766f
ovs->ovs.ovsVersion = 2
ovs->ovs.numOverlapsPerFile = 67108864
ovs->ovs.smallestIID = 3
ovs->ovs.largestIID = 1002325
ovs->ovs.numOverlapsTotal = 281606
ovs->ovs.highestFileIndex = 1
ovs->ovs.maxReadLenInBits = 16
The other log files seems correct.
Thanks,
Miquel
On 2014?06?11? 06:01, Serge Koren wrote:
> Hi,
>
> It looks like from your output that no corrected sequences were
> generated. Do you have the full output of the pipeline along with the
> asm.layout.err, asm.gkpStore.err*, and asm.ovlStore.err in the
> temppacbio directory?
>
> Sergey
>
> On Jun 9, 2014, at 9:41 PM, Miguel Grau <mi...@uj...
> <mailto:mi...@uj...>> wrote:
>
>> Hi wgs-users,
>> I am using PBcR, following the wiki tutorial, "Correction Using
>> Complementary High-Identity Sequences
>> <http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=PacBioToCA#Self-Correction_With_C2_Sequences_.28or_newer.29>"
>> When I run pacBioToca to generate corrected pacbio sequences, the
>> process stop suddenly:
>>
>> <Linux-amd64>/bin/fastqToCA -libraryname illumina -technology
>> illumina -type sanger -innie -mates
>> illumina_forward_paired.fastq,illumina_reverse_paired.fastq >
>> illumina.frg
>>
>> <Linux-amd64>/bin/pacBioToCA -length 500 -partitions 200 -l pacbio -t
>> 8 -s pacbio.spec -fastq pacbio.filtered_subreads.fastq illumina.frg >
>> run.out 2>&1
>>
>> The last log lines are (without error messages before):
>>
>> cp asm.layout.err /apps/wgs-8.1/DATA//pacbio.correction.err
>> Total split bases is 0 vs 9188565 so ratio is 0
>> ----------------------------------------END Mon Jun 9 18:29:42 2014
>> (0 seconds)
>> ----------------------------------------START Mon Jun 9 18:29:42 2014
>> cp asm.layout.hist /apps/wgs-8.1/DATA//pacbio.correction.hist
>> ----------------------------------------END Mon Jun 9 18:29:42 2014
>> (0 seconds)
>> ----------------------------------------START Mon Jun 9 18:29:42 2014
>> cat `ls [0-9]*.fasta |grep trim |sort -T . -rnk1` >
>> /apps/wgs-8.1/DATA//pacbio.fasta
>>
>> I am not sure which parameter I have to edit and why the total split
>> base are 0.
>> CA 8.1 is correctly installed, if I use the sample example data from
>> the tutorial, it works.
>>
>> Thanks,
>>
>>
>>
>> ------------------------------------------------------------------------------
>> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
>> Find What Matters Most in Your Big Data with HPCC Systems
>> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
>> http://p.sf.net/sfu/hpccsystems_______________________________________________
>> wgs-assembler-users mailing list
>> wgs...@li...
>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
>
|
|
From: Serge K. <ser...@gm...> - 2014-06-10 21:01:59
|
Hi, It looks like from your output that no corrected sequences were generated. Do you have the full output of the pipeline along with the asm.layout.err, asm.gkpStore.err*, and asm.ovlStore.err in the temppacbio directory? Sergey On Jun 9, 2014, at 9:41 PM, Miguel Grau <mi...@uj...> wrote: > Hi wgs-users, > I am using PBcR, following the wiki tutorial, "Correction Using Complementary High-Identity Sequences" > When I run pacBioToca to generate corrected pacbio sequences, the process stop suddenly: > > <Linux-amd64>/bin/fastqToCA -libraryname illumina -technology illumina -type sanger -innie -mates illumina_forward_paired.fastq,illumina_reverse_paired.fastq > illumina.frg > > <Linux-amd64>/bin/pacBioToCA -length 500 -partitions 200 -l pacbio -t 8 -s pacbio.spec -fastq pacbio.filtered_subreads.fastq illumina.frg > run.out 2>&1 > > The last log lines are (without error messages before): > > cp asm.layout.err /apps/wgs-8.1/DATA//pacbio.correction.err > Total split bases is 0 vs 9188565 so ratio is 0 > ----------------------------------------END Mon Jun 9 18:29:42 2014 (0 seconds) > ----------------------------------------START Mon Jun 9 18:29:42 2014 > cp asm.layout.hist /apps/wgs-8.1/DATA//pacbio.correction.hist > ----------------------------------------END Mon Jun 9 18:29:42 2014 (0 seconds) > ----------------------------------------START Mon Jun 9 18:29:42 2014 > cat `ls [0-9]*.fasta |grep trim |sort -T . -rnk1` > /apps/wgs-8.1/DATA//pacbio.fasta > > I am not sure which parameter I have to edit and why the total split base are 0. > CA 8.1 is correctly installed, if I use the sample example data from the tutorial, it works. > > Thanks, > > > > ------------------------------------------------------------------------------ > HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions > Find What Matters Most in Your Big Data with HPCC Systems > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. > Leverages Graph Analysis for Fast Processing & Easy Data Exploration > http://p.sf.net/sfu/hpccsystems_______________________________________________ > wgs-assembler-users mailing list > wgs...@li... > https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users |
|
From: Corey W. <cor...@gm...> - 2014-06-10 16:02:23
|
Brian, That email really helped me out! It turns out an older version of wgs was hanging out on the server and was in the system-wide .bashrc. After using sed to remove it from my PATH when I log in I set out to recompile the kmer package. It turns out the problem may be with our system because I kept getting this 'cp cannot preserve permissions' error and then compilation would stop. So as you suggested I tried compiling somewhere else (a VM). After working through the dependencies I got a working installation. I then copied the working kmer directory back to the original machine and wgs installed without issue. Thanks, Brian, for getting back to me so quickly and getting me on the right track. -Corey On Tue, Jun 10, 2014 at 8:21 AM, Brian Walenz <th...@gm...> wrote: > I think it is, somehow, finding an older version of 'kmer' and maybe > samtools. Are those installed system-wide? If there are other > installations nearby, try compiling from a different directory. kmer r1960 > is definitely included in wgs-8.1. The symbol EXISTDB_H_VERSION is defined > in libkmer/existDB.H. > > Which OS is this on? > > b > > > > > On Mon, Jun 9, 2014 at 4:39 PM, Corey Wischmeyer < > cor...@gm...> wrote: > >> Downloading: >> >> >> http://sourceforge.net/projects/wgs-assembler/files/wgs-assembler/wgs-8.1/wgs-8.1.tar.bz2/download >> >> then executing these commands >> >> bzip2 -dc wgs-8.1.tar.bz2 | tar -xf - cd wgs-8.1 cd kmer && make install >> && cd .. cd samtools && make && cd .. cd src && make && cd .. cd .. >> >> Gives an error about samtools no longer being included in the package. >> >> When I then skip to the next bit and run the make and cd I get this >> message: >> >> merTrim.C:46:2: error: #error kmer needs to be updated to at least r1960 >> >> merTrim.C:47:2: error: #error note that the kmer svn url changed in mid >> December 2012, the old url does not have r1960 >> >> make[1]: *** No rule to make target `bio++.H', needed by `meryl.o'. Stop. >> >> make: *** [objs] Error 1 >> >> Has anyone else run into this behavior? Should I make a ticket? >> >> -- >> -Corey Wischmeyer >> >> -- -Corey Wischmeyer |
|
From: Brian W. <th...@gm...> - 2014-06-10 13:22:08
|
I think it is, somehow, finding an older version of 'kmer' and maybe samtools. Are those installed system-wide? If there are other installations nearby, try compiling from a different directory. kmer r1960 is definitely included in wgs-8.1. The symbol EXISTDB_H_VERSION is defined in libkmer/existDB.H. Which OS is this on? b On Mon, Jun 9, 2014 at 4:39 PM, Corey Wischmeyer <cor...@gm... > wrote: > Downloading: > > > http://sourceforge.net/projects/wgs-assembler/files/wgs-assembler/wgs-8.1/wgs-8.1.tar.bz2/download > > then executing these commands > > bzip2 -dc wgs-8.1.tar.bz2 | tar -xf - cd wgs-8.1 cd kmer && make install > && cd .. cd samtools && make && cd .. cd src && make && cd .. cd .. > > Gives an error about samtools no longer being included in the package. > > When I then skip to the next bit and run the make and cd I get this > message: > > merTrim.C:46:2: error: #error kmer needs to be updated to at least r1960 > > merTrim.C:47:2: error: #error note that the kmer svn url changed in mid > December 2012, the old url does not have r1960 > > make[1]: *** No rule to make target `bio++.H', needed by `meryl.o'. Stop. > > make: *** [objs] Error 1 > > Has anyone else run into this behavior? Should I make a ticket? > > -- > -Corey Wischmeyer > > |
|
From: Miguel G. <mi...@uj...> - 2014-06-10 02:04:15
|
Hi wgs-users, I am using PBcR, following the wiki tutorial, "Correction Using Complementary High-Identity Sequences <http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=PacBioToCA#Self-Correction_With_C2_Sequences_.28or_newer.29>" When I run pacBioToca to generate corrected pacbio sequences, the process stop suddenly: <Linux-amd64>/bin/fastqToCA -libraryname illumina -technology illumina -type sanger -innie -mates illumina_forward_paired.fastq,illumina_reverse_paired.fastq > illumina.frg <Linux-amd64>/bin/pacBioToCA -length 500 -partitions 200 -l pacbio -t 8 -s pacbio.spec -fastq pacbio.filtered_subreads.fastq illumina.frg > run.out 2>&1 The last log lines are (without error messages before): cp asm.layout.err /apps/wgs-8.1/DATA//pacbio.correction.err Total split bases is 0 vs 9188565 so ratio is 0 ----------------------------------------END Mon Jun 9 18:29:42 2014 (0 seconds) ----------------------------------------START Mon Jun 9 18:29:42 2014 cp asm.layout.hist /apps/wgs-8.1/DATA//pacbio.correction.hist ----------------------------------------END Mon Jun 9 18:29:42 2014 (0 seconds) ----------------------------------------START Mon Jun 9 18:29:42 2014 cat `ls [0-9]*.fasta |grep trim |sort -T . -rnk1` > /apps/wgs-8.1/DATA//pacbio.fasta I am not sure which parameter I have to edit and why the total split base are 0. CA 8.1 is correctly installed, if I use the sample example data from the tutorial, it works. Thanks, |
|
From: Corey W. <cor...@gm...> - 2014-06-09 20:39:58
|
Downloading: http://sourceforge.net/projects/wgs-assembler/files/wgs-assembler/wgs-8.1/wgs-8.1.tar.bz2/download then executing these commands bzip2 -dc wgs-8.1.tar.bz2 | tar -xf - cd wgs-8.1 cd kmer && make install && cd .. cd samtools && make && cd .. cd src && make && cd .. cd .. Gives an error about samtools no longer being included in the package. When I then skip to the next bit and run the make and cd I get this message: merTrim.C:46:2: error: #error kmer needs to be updated to at least r1960 merTrim.C:47:2: error: #error note that the kmer svn url changed in mid December 2012, the old url does not have r1960 make[1]: *** No rule to make target `bio++.H', needed by `meryl.o'. Stop. make: *** [objs] Error 1 Has anyone else run into this behavior? Should I make a ticket? -- -Corey Wischmeyer |
|
From: Fields, C. J <cjf...@il...> - 2014-06-09 16:28:36
|
Has anyone tried using runCA using PBS/Torque? I know it is set up for SGE/LSF but I wanted to know if others have ventured down this path before I try modifying runCA to work with PBS commands. chris |
|
From: Walenz, B. <bw...@jc...> - 2014-05-02 17:51:20
|
Hi- [this ended up in my spam folder, sorry] I don't see any problems here, but can't explain what it is doing in the pauses. There are some steps in scaffold merging that are not interesting and don't log anything - the fill gaps case was probably this. In scaffold merging, after an iteration of merging it done, it needs to slog through the scaffolds and find more work to do. There isn't much logging here. It is possible that the last case is from this. There is one problem in cgw worth mentioning here. It can get stuck in an infinite loop where it merges two scaffolds, decides the new scaffold is flawed, breaks the new scaffold into two scaffolds, and repeats. If this happens, the logs will show repeated merges between scaffolds number A and A+1. The sizes of these two scaffolds is constant between iterations. Is the machine otherwise healthy - is the CGW process size below physical memory size? b On 4/29/14 7:28 AM, "kuhl" <ku...@mo...> wrote: > Dear CA developers, > > I am currently running an assembly of a mammalian genome (2.1Gbp) using > illumina paired ends and mate pairs and 454 reads. I recently switched > from > using BOG to BOGART, which doubled the N50 of the unitigs. Thus, I was > quite > euphoric about the results, but now there is a problem in cgw. At some > scaffolds the assembler runs for hours and hours without any output to the > logfile. Is this a known problem? > > Find below three examples from the cgw logfile. > > Regards, Heiner > > example 1(during Fill_Gaps iteration): > > CreateAContigInScaffold()-- new contig 9629698 in scaffold 4689 > CreateAContigInScaffold()-- new contig 9629699 in scaffold 4689 > WARNING: Negative variance (379508.35 +- -46149.23) from edgeOrient I CI > 9571434 F (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI > 68466 R (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING: > Negative variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F > (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R > (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING: Negative > variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F > (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R > (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING: Negative > variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F > (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R > (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)trf failed with > info=11; no solution found, giving up. > RecomputeOffsetsInScaffold() returned RECOMPUTE_LAPACK on scaffold 4701; > no more work will help. > > Here it took several hours until cgw proceeded > > MergeMultiAlignsFast_new()-- failed to find overlap between contigs > 9612721 and 9571434. > CreateAContigInScaffold()-- MergeMultiAlignsFast_new() failed. > CreateAContigInScaffold()-- contig 9612721 (original 9612721) > 847501-614216 > CreateAContigInScaffold()-- contig 9571434 (original 9571434) > 1179469-799558 > CreateAContigInScaffold()-- contig 9612720 (original 9612720) > 1047392-847481 > CreateAContigInScaffold()-- contig 9612719 (original 9612719) > 1054076-1047372 > CreateAContigInScaffold()-- contig 9578740 (original 9578740) > 1059537-1054056 > CreateAContigInScaffold()-- contig 9612718 (original 9612718) > 1084352-1059517 > CreateAContigInScaffold()-- contig 9591744 (original 9591744) > 1103685-1084332 > CreateAContigInScaffold()-- contig 9578739 (original 9578739) > 1132833-1103665 > CreateAContigInScaffold()-- contig 212091 (original 212091) > 1132813-1140087 > CreateAContigInScaffold()-- contig 68466 (original 68466) 1140067-1179067 > > * CIScaffold 4701 numCI:19 (a:75856 b:68466) length: 1179469 >> Includes CIs > 0: CI 75856 sid 4701 len 1256, 32.7 ends > 0, 1256 var 0.0, 32.7 orient F > 1: CI 169716 sid 4701 len 11486, 298.6 ends > 1236, 12722 var 11663292.9,11663591.6 orient F > 2: CI 244409 sid 4701 len 2610, 67.9 ends > 12702, 15312 var 12466586.4,12466654.3 orient F > 3: CI 78245 sid 4701 len 21682, 563.7 ends > 36974, 15292 var 14686531.7,14685968.0 orient R > 4: CI 9578742 sid 4701 len 239019, 6214.5 ends > 277524, 38505 var 15137021.4,15130806.9 orient R > 5: CI 9612724 sid 4701 len 76477, 1988.4 ends > 354054, 277577 var 15143369.0,15141380.6 orient R > 6: CI 9612723 sid 4701 len 96573, 2510.9 ends > 450889, 354316 var 15148845.4,15146334.6 orient R > 7: CI 9612722 sid 4701 len 129181, 3358.7 ends > 580050, 450869 var 15166458.9,15163100.2 orient R > 8: CI 9591745 sid 4701 len 31856, 828.3 ends > 611886, 580030 var 15826952.4,15826124.1 orient R > 9: CI 9612721 sid 4701 len 233285, 6065.4 ends > 847501, 614216 var 16123024.1,16116958.7 orient R > 10: CI 9571434 sid 4701 len 379911, 9877.7 ends > 1179469, 799558 var 17520656.7,17510779.0 orient R > 11: CI 9612720 sid 4701 len 199911, 5197.7 ends > 1047392, 847481 var 16161345.7,16156148.0 orient R > 12: CI 9612719 sid 4701 len 6704, 174.3 ends > 1054076,1047372 var 16696287.7,16696113.4 orient R > 13: CI 9578740 sid 4701 len 5481, 142.5 ends > 1059537,1054056 var 16931871.5,16931729.0 orient R > 14: CI 9612718 sid 4701 len 24835, 645.7 ends > 1084352,1059517 var 17365259.4,17364613.7 orient R > 15: CI 9591744 sid 4701 len 19353, 503.2 ends > 1103685,1084332 var 17370622.7,17370119.5 orient R > 16: CI 9578739 sid 4701 len 29168, 758.4 ends > 1132833,1103665 var 17391605.6,17390847.2 orient R > 17: CI 212091 sid 4701 len 7274, 189.1 ends > 1132813,1140087 var 17414697.6,17414886.7 orient F > ______________________________________________________________________________ > ________________________________________ > > example 2(during Fill_Gaps iteration): > > * CIScaffold 4670 numCI:2 (a:75476 b:9629697) length: 3296 >> Includes CIs > 0: CI 75476 sid 4670 len 1480, 38.5 ends > 1480, 0 var 38.5, 0.0 orient R > 1: CI 9629697 sid 4670 len 2692, 70.0 ends > 604, 3296 var 9114.7, 9184.7 orient F >> M Edges A > cidA:4670 cidB:5344 weight:2 ori:O con:0 distance:-1668 > stddev:150.706 (144551240,144551241) > cidA:4670 cidB:7729 weight:2 ori:A con:0 distance:43 > stddev:134.686 (119529581,119529580) > cidA:4670 cidB:8383 weight:1 ori:O con:0 distance:-177 > stddev:190.49 (143054256,143054255) > cidA:4670 cidB:8383 weight:7 ori:O con:1 distance:-1387 > stddev:80.549 (148871104,148871105) > cidA:4670 cidB:14282 weight:2 ori:A con:0 distance:-184 > stddev:134.697 (159606342,159606343) > cidA:4670 cidB:39401 weight:1 ori:O con:0 distance:-1032 > stddev:190.549 (10056130,10056129) > cidA:4670 cidB:47012 weight:1 ori:A con:0 distance:-270 > stddev:190.497 (163142964,163142963) > cidA:4670 cidB:47012 weight:1 ori:A con:0 distance:-1831 > stddev:213.14 (40610662,40610663) > cidA:4670 cidB:47012 weight:4 ori:A con:0 distance:-971 > stddev:100.436 (150398128,150398127) > cidA:4670 cidB:47525 weight:2 ori:A con:0 distance:-470 > stddev:134.711 (64096137,64096138) > cidA:4670 cidB:48709 weight:3 ori:A con:0 distance:-1828 > stddev:109.954 (3540936,3540935) >> M Edges B > cidA:4670 cidB:30766 weight:1 ori:N con:0 distance:279 > stddev:830.551 (200817167,200817166) > cidA:4670 cidB:40711 weight:5 ori:N con:0 distance:-754 > stddev:85.2075 (68316430,68316429) > cidA:4670 cidB:43356 weight:2 ori:N con:0 distance:-3106 > stddev:150.768 (152889486,152889487) > > <---- Here cgw stays for several hours without any message and but using > 100% CPU > > I stopped Fill gaps and restarted cgw at merge scaffolds aggressive > ________________________________________________________________________ > > example 3(during merge scaffolds aggressive): > > isQualityScaffoldMergingEdge()-- Merge scaffolds 39672 (191435.2bp) and > 39878 (67088.0bp): gap -3026.4bp +- 96.4bp weight 143 BA_AB edge > isQualityScaffoldMergingEdge()-- scaffold 39672 instrumenter happy > 12692.0 gap 3.5 misorient close 0.0 correct 1.0 far 0.0 oriented close > 130.0 far 11.0 missing 816.8 external 209.8 > isQualityScaffoldMergingEdge()-- scaffold 39878 instrumenter happy > 4214.0 gap 0.0 misorient close 0.0 correct 1.0 far 0.0 oriented close 48.0 > far 1.0 missing 291.1 external 263.8 > isQualityScaffoldMergingEdge()-- scaffold (new) instrumenter happy > 16969.0 gap 35.8 misorient close 0.0 correct 2.0 far 0.0 oriented close > 178.0 far 109.0 missing 1000.0 external 229.2 > isQualityScaffoldMergingEdge()-- before: 0.914 satisfied (16905/1299 > good/bad mates) after: 0.918 satisfied (16968/1288 good/bad mates) > isQualityScaffoldMergingEdge()-- ARE happy enough to merge 100 (0.918 >= > 0.985) || (0.918 >= 0.914) || ((16968 > 16905) && (-0.175 <= 0.300)) > isQualityScaffoldMergingEdge()-- NEW pass (1800/0) > ExamineSEdgeForUsability_Interleaved()-- Expected end contigs to overlap, > didn't find it, will not merge. > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 21 (527789bp) into > scaffold 49855 (0bp) at offset 0.000 +/- 0.000 orient F > InsertScaffoldContentsIntoScaffold()-- Insert CI 487 19429bp fwd > 0 +- 0 19429 +- 505 was 0 +- 0 > 19429 +- 505 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9600072 74365bp fwd > 19409 +- 11175 93774 +- 13109 was 19409 +- 11175 > > 93774 +- 13109 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9571242 133049bp fwd > 93754 +- 46754 226803 +- 50213 was 93754 +- 46754 > > 226803 +- 50213 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9571243 150819bp fwd > 227078 +- 209488 377897 +- 213409 was 227078 +- 209488 > > 377897 +- 213409 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9571244 149912bp fwd > 377877 +- 228539 527789 +- 232437 was 377877 +- 228539 > > 527789 +- 232437 > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 43749 (445501bp) > into scaffold 49855 (527789bp) at offset 527768.962 +/- 489.705 orient R > InsertScaffoldContentsIntoScaffold()-- Insert CI 9599993 88959bp rev > 527769 +- 239811 616728 +- 242124 was 445501 +- 569549 > > 356542 +- 567236 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9599992 14970bp rev > 616708 +- 249455 631678 +- 249845 was 356562 +- 559904 > > 341592 +- 559515 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9599991 69788bp rev > 635107 +- 764942 704895 +- 766757 was 338163 +- 44417 > > 268375 +- 42603 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9469786 476bp fwd > 704875 +- 796682 705351 +- 796694 was 268395 +- 12678 > > 267919 +- 12666 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9599990 267939bp rev > 705331 +- 802393 973270 +- 809360 was 267939 +- 6966 > > 0 +- 0 > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 28 (34647bp) into > scaffold 49856 (0bp) at offset 0.000 +/- 0.000 orient R > InsertScaffoldContentsIntoScaffold()-- Insert CI 635 34647bp rev > 0 +- 0 34647 +- 901 was 34647 +- 901 > 0 +- 0 > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 47219 (43488bp) > into scaffold 49856 (34647bp) at offset 33447.000 +/- 29.489 orient R > InsertScaffoldContentsIntoScaffold()-- Insert CI 9623655 43488bp rev > 33447 +- 870 76935 +- 2000 was 43488 +- 1131 > > 0 +- 0 > CreateAContigInScaffold()-- new contig 9631137 in scaffold 49856 > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 183 (306704bp) into > scaffold 49857 (0bp) at offset 0.000 +/- 0.000 orient R > InsertScaffoldContentsIntoScaffold()-- Insert CI 9600994 101398bp rev > 0 +- 0 101398 +- 2636 was 306704 +- 102182 > 205306 +- 99545 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9600993 107087bp rev > 101378 +- 14121 208465 +- 16905 was 205326 +- 88061 > > 98239 +- 85277 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9600992 79255bp rev > 208445 +- 91912 287700 +- 93973 was 98259 +- 10270 > > 19004 +- 8209 > InsertScaffoldContentsIntoScaffold()-- Insert CI 3847 19024bp fwd > 287680 +- 101687 306704 +- 102182 was 19024 +- 495 > > 0 +- 0 > SetCIScaffoldTLength()-- adjusted scaffold 49857 from (306704 +- 102182) > to (306704 +- 102182) > InsertScaffoldContentsIntoScaffold()-- Insert scaffold 39383 (214584bp) > into scaffold 49857 (306704bp) at offset 306684.000 +/- 320.690 orient F > InsertScaffoldContentsIntoScaffold()-- Insert CI 9588022 214584bp fwd > 306684 +- 102842 521268 +- 108421 was 0 +- 0 > > 214584 +- 5579 > InsertScaffoldContentsIntoScaffold()-- Insert CI 9579474 180065bp fwd > 340832 +- 104702 520897 +- 109384 was 34148 +- 1860 > > 214213 +- 6541 > --> here again waiting for at least 2 hours > > --------------------------------------------------------------- > Dr. Heiner Kuhl > MPI Molecular Genetics Tel: + 49 + 30 / 8413 1776 > Next Generation Sequencing > Ihnestrasse 73 email: ku...@mo... > D-14195 Berlin http://www.molgen.mpg.de/SeqCore > --------------------------------------------------------------- > > > ------------------------------------------------------------------------------ > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE > Instantly run your Selenium tests across 300+ browser/OS combos. Get > unparalleled scalability from the best Selenium testing platform available. > Simple to use. Nothing to install. Get started now for free." > http://p.sf.net/sfu/SauceLabs > _______________________________________________ > wgs-assembler-users mailing list > wgs...@li... > https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users |
|
From: kuhl <ku...@mo...> - 2014-04-29 11:44:24
|
Dear CA developers,
I am currently running an assembly of a mammalian genome (2.1Gbp) using
illumina paired ends and mate pairs and 454 reads. I recently switched
from
using BOG to BOGART, which doubled the N50 of the unitigs. Thus, I was
quite
euphoric about the results, but now there is a problem in cgw. At some
scaffolds the assembler runs for hours and hours without any output to the
logfile. Is this a known problem?
Find below three examples from the cgw logfile.
Regards, Heiner
example 1(during Fill_Gaps iteration):
CreateAContigInScaffold()-- new contig 9629698 in scaffold 4689
CreateAContigInScaffold()-- new contig 9629699 in scaffold 4689
WARNING: Negative variance (379508.35 +- -46149.23) from edgeOrient I CI
9571434 F (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI
68466 R (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING:
Negative variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
(1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
(1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING: Negative
variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
(1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
(1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING: Negative
variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
(1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
(1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)trf failed with
info=11; no solution found, giving up.
RecomputeOffsetsInScaffold() returned RECOMPUTE_LAPACK on scaffold 4701;
no more work will help.
Here it took several hours until cgw proceeded
MergeMultiAlignsFast_new()-- failed to find overlap between contigs
9612721 and 9571434.
CreateAContigInScaffold()-- MergeMultiAlignsFast_new() failed.
CreateAContigInScaffold()-- contig 9612721 (original 9612721)
847501-614216
CreateAContigInScaffold()-- contig 9571434 (original 9571434)
1179469-799558
CreateAContigInScaffold()-- contig 9612720 (original 9612720)
1047392-847481
CreateAContigInScaffold()-- contig 9612719 (original 9612719)
1054076-1047372
CreateAContigInScaffold()-- contig 9578740 (original 9578740)
1059537-1054056
CreateAContigInScaffold()-- contig 9612718 (original 9612718)
1084352-1059517
CreateAContigInScaffold()-- contig 9591744 (original 9591744)
1103685-1084332
CreateAContigInScaffold()-- contig 9578739 (original 9578739)
1132833-1103665
CreateAContigInScaffold()-- contig 212091 (original 212091)
1132813-1140087
CreateAContigInScaffold()-- contig 68466 (original 68466) 1140067-1179067
* CIScaffold 4701 numCI:19 (a:75856 b:68466) length: 1179469
> Includes CIs
0: CI 75856 sid 4701 len 1256, 32.7 ends
0, 1256 var 0.0, 32.7 orient F
1: CI 169716 sid 4701 len 11486, 298.6 ends
1236, 12722 var 11663292.9,11663591.6 orient F
2: CI 244409 sid 4701 len 2610, 67.9 ends
12702, 15312 var 12466586.4,12466654.3 orient F
3: CI 78245 sid 4701 len 21682, 563.7 ends
36974, 15292 var 14686531.7,14685968.0 orient R
4: CI 9578742 sid 4701 len 239019, 6214.5 ends
277524, 38505 var 15137021.4,15130806.9 orient R
5: CI 9612724 sid 4701 len 76477, 1988.4 ends
354054, 277577 var 15143369.0,15141380.6 orient R
6: CI 9612723 sid 4701 len 96573, 2510.9 ends
450889, 354316 var 15148845.4,15146334.6 orient R
7: CI 9612722 sid 4701 len 129181, 3358.7 ends
580050, 450869 var 15166458.9,15163100.2 orient R
8: CI 9591745 sid 4701 len 31856, 828.3 ends
611886, 580030 var 15826952.4,15826124.1 orient R
9: CI 9612721 sid 4701 len 233285, 6065.4 ends
847501, 614216 var 16123024.1,16116958.7 orient R
10: CI 9571434 sid 4701 len 379911, 9877.7 ends
1179469, 799558 var 17520656.7,17510779.0 orient R
11: CI 9612720 sid 4701 len 199911, 5197.7 ends
1047392, 847481 var 16161345.7,16156148.0 orient R
12: CI 9612719 sid 4701 len 6704, 174.3 ends
1054076,1047372 var 16696287.7,16696113.4 orient R
13: CI 9578740 sid 4701 len 5481, 142.5 ends
1059537,1054056 var 16931871.5,16931729.0 orient R
14: CI 9612718 sid 4701 len 24835, 645.7 ends
1084352,1059517 var 17365259.4,17364613.7 orient R
15: CI 9591744 sid 4701 len 19353, 503.2 ends
1103685,1084332 var 17370622.7,17370119.5 orient R
16: CI 9578739 sid 4701 len 29168, 758.4 ends
1132833,1103665 var 17391605.6,17390847.2 orient R
17: CI 212091 sid 4701 len 7274, 189.1 ends
1132813,1140087 var 17414697.6,17414886.7 orient F
______________________________________________________________________________________________________________________
example 2(during Fill_Gaps iteration):
* CIScaffold 4670 numCI:2 (a:75476 b:9629697) length: 3296
> Includes CIs
0: CI 75476 sid 4670 len 1480, 38.5 ends
1480, 0 var 38.5, 0.0 orient R
1: CI 9629697 sid 4670 len 2692, 70.0 ends
604, 3296 var 9114.7, 9184.7 orient F
> M Edges A
cidA:4670 cidB:5344 weight:2 ori:O con:0 distance:-1668
stddev:150.706 (144551240,144551241)
cidA:4670 cidB:7729 weight:2 ori:A con:0 distance:43
stddev:134.686 (119529581,119529580)
cidA:4670 cidB:8383 weight:1 ori:O con:0 distance:-177
stddev:190.49 (143054256,143054255)
cidA:4670 cidB:8383 weight:7 ori:O con:1 distance:-1387
stddev:80.549 (148871104,148871105)
cidA:4670 cidB:14282 weight:2 ori:A con:0 distance:-184
stddev:134.697 (159606342,159606343)
cidA:4670 cidB:39401 weight:1 ori:O con:0 distance:-1032
stddev:190.549 (10056130,10056129)
cidA:4670 cidB:47012 weight:1 ori:A con:0 distance:-270
stddev:190.497 (163142964,163142963)
cidA:4670 cidB:47012 weight:1 ori:A con:0 distance:-1831
stddev:213.14 (40610662,40610663)
cidA:4670 cidB:47012 weight:4 ori:A con:0 distance:-971
stddev:100.436 (150398128,150398127)
cidA:4670 cidB:47525 weight:2 ori:A con:0 distance:-470
stddev:134.711 (64096137,64096138)
cidA:4670 cidB:48709 weight:3 ori:A con:0 distance:-1828
stddev:109.954 (3540936,3540935)
> M Edges B
cidA:4670 cidB:30766 weight:1 ori:N con:0 distance:279
stddev:830.551 (200817167,200817166)
cidA:4670 cidB:40711 weight:5 ori:N con:0 distance:-754
stddev:85.2075 (68316430,68316429)
cidA:4670 cidB:43356 weight:2 ori:N con:0 distance:-3106
stddev:150.768 (152889486,152889487)
<---- Here cgw stays for several hours without any message and but using
100% CPU
I stopped Fill gaps and restarted cgw at merge scaffolds aggressive
________________________________________________________________________
example 3(during merge scaffolds aggressive):
isQualityScaffoldMergingEdge()-- Merge scaffolds 39672 (191435.2bp) and
39878 (67088.0bp): gap -3026.4bp +- 96.4bp weight 143 BA_AB edge
isQualityScaffoldMergingEdge()-- scaffold 39672 instrumenter happy
12692.0 gap 3.5 misorient close 0.0 correct 1.0 far 0.0 oriented close
130.0 far 11.0 missing 816.8 external 209.8
isQualityScaffoldMergingEdge()-- scaffold 39878 instrumenter happy
4214.0 gap 0.0 misorient close 0.0 correct 1.0 far 0.0 oriented close 48.0
far 1.0 missing 291.1 external 263.8
isQualityScaffoldMergingEdge()-- scaffold (new) instrumenter happy
16969.0 gap 35.8 misorient close 0.0 correct 2.0 far 0.0 oriented close
178.0 far 109.0 missing 1000.0 external 229.2
isQualityScaffoldMergingEdge()-- before: 0.914 satisfied (16905/1299
good/bad mates) after: 0.918 satisfied (16968/1288 good/bad mates)
isQualityScaffoldMergingEdge()-- ARE happy enough to merge 100 (0.918 >=
0.985) || (0.918 >= 0.914) || ((16968 > 16905) && (-0.175 <= 0.300))
isQualityScaffoldMergingEdge()-- NEW pass (1800/0)
ExamineSEdgeForUsability_Interleaved()-- Expected end contigs to overlap,
didn't find it, will not merge.
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 21 (527789bp) into
scaffold 49855 (0bp) at offset 0.000 +/- 0.000 orient F
InsertScaffoldContentsIntoScaffold()-- Insert CI 487 19429bp fwd
0 +- 0 19429 +- 505 was 0 +- 0
19429 +- 505
InsertScaffoldContentsIntoScaffold()-- Insert CI 9600072 74365bp fwd
19409 +- 11175 93774 +- 13109 was 19409 +- 11175
93774 +- 13109
InsertScaffoldContentsIntoScaffold()-- Insert CI 9571242 133049bp fwd
93754 +- 46754 226803 +- 50213 was 93754 +- 46754
226803 +- 50213
InsertScaffoldContentsIntoScaffold()-- Insert CI 9571243 150819bp fwd
227078 +- 209488 377897 +- 213409 was 227078 +- 209488
377897 +- 213409
InsertScaffoldContentsIntoScaffold()-- Insert CI 9571244 149912bp fwd
377877 +- 228539 527789 +- 232437 was 377877 +- 228539
527789 +- 232437
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 43749 (445501bp)
into scaffold 49855 (527789bp) at offset 527768.962 +/- 489.705 orient R
InsertScaffoldContentsIntoScaffold()-- Insert CI 9599993 88959bp rev
527769 +- 239811 616728 +- 242124 was 445501 +- 569549
356542 +- 567236
InsertScaffoldContentsIntoScaffold()-- Insert CI 9599992 14970bp rev
616708 +- 249455 631678 +- 249845 was 356562 +- 559904
341592 +- 559515
InsertScaffoldContentsIntoScaffold()-- Insert CI 9599991 69788bp rev
635107 +- 764942 704895 +- 766757 was 338163 +- 44417
268375 +- 42603
InsertScaffoldContentsIntoScaffold()-- Insert CI 9469786 476bp fwd
704875 +- 796682 705351 +- 796694 was 268395 +- 12678
267919 +- 12666
InsertScaffoldContentsIntoScaffold()-- Insert CI 9599990 267939bp rev
705331 +- 802393 973270 +- 809360 was 267939 +- 6966
0 +- 0
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 28 (34647bp) into
scaffold 49856 (0bp) at offset 0.000 +/- 0.000 orient R
InsertScaffoldContentsIntoScaffold()-- Insert CI 635 34647bp rev
0 +- 0 34647 +- 901 was 34647 +- 901
0 +- 0
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 47219 (43488bp)
into scaffold 49856 (34647bp) at offset 33447.000 +/- 29.489 orient R
InsertScaffoldContentsIntoScaffold()-- Insert CI 9623655 43488bp rev
33447 +- 870 76935 +- 2000 was 43488 +- 1131
0 +- 0
CreateAContigInScaffold()-- new contig 9631137 in scaffold 49856
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 183 (306704bp) into
scaffold 49857 (0bp) at offset 0.000 +/- 0.000 orient R
InsertScaffoldContentsIntoScaffold()-- Insert CI 9600994 101398bp rev
0 +- 0 101398 +- 2636 was 306704 +- 102182
205306 +- 99545
InsertScaffoldContentsIntoScaffold()-- Insert CI 9600993 107087bp rev
101378 +- 14121 208465 +- 16905 was 205326 +- 88061
98239 +- 85277
InsertScaffoldContentsIntoScaffold()-- Insert CI 9600992 79255bp rev
208445 +- 91912 287700 +- 93973 was 98259 +- 10270
19004 +- 8209
InsertScaffoldContentsIntoScaffold()-- Insert CI 3847 19024bp fwd
287680 +- 101687 306704 +- 102182 was 19024 +- 495
0 +- 0
SetCIScaffoldTLength()-- adjusted scaffold 49857 from (306704 +- 102182)
to (306704 +- 102182)
InsertScaffoldContentsIntoScaffold()-- Insert scaffold 39383 (214584bp)
into scaffold 49857 (306704bp) at offset 306684.000 +/- 320.690 orient F
InsertScaffoldContentsIntoScaffold()-- Insert CI 9588022 214584bp fwd
306684 +- 102842 521268 +- 108421 was 0 +- 0
214584 +- 5579
InsertScaffoldContentsIntoScaffold()-- Insert CI 9579474 180065bp fwd
340832 +- 104702 520897 +- 109384 was 34148 +- 1860
214213 +- 6541
--> here again waiting for at least 2 hours
---------------------------------------------------------------
Dr. Heiner Kuhl
MPI Molecular Genetics Tel: + 49 + 30 / 8413 1776
Next Generation Sequencing
Ihnestrasse 73 email: ku...@mo...
D-14195 Berlin http://www.molgen.mpg.de/SeqCore
---------------------------------------------------------------
|
|
From: Rajiv M. <rm...@st...> - 2014-03-19 21:46:58
|
Hi,
I am working on several Drosophila assemblies using Moleculo long reads,
and am encountering an issue where there are apparently large and 100%
identical regions of overlap between the ends of different output
contigs. We identified these cases using a "nucmer show-tiling"
alignment to the reference genome, and the situation appears to be
fairly common in our assemblies. Often, but not always, the contig IDs
are sequential (e.g. ctg100001146456, ctg100001146457,
ctg100001146458). There is no additional sequence on the ends of the
contigs that got added thus preventing the join, so we are having
trouble figuring out what is going on.
I was wondering whether we are missing some important step or set some
parameter incorrectly in the spec file:
frgCorrBatchSize = 400000
frgCorrThreads = 6
ovlCorrConcurrency = 6
ovlCorrBatchSize = 400000
merylMemory = -segments 6 -threads 6
merylThreads = 6
ovlThreads = 6
ovlHashBits = 25
ovlHashBlockLength = 180000000
merOverlapperThreads = 6
merOverlapperSeedBatchSize = 400000
merOverlapperExtendBatchSize = 400000
cnsConcurrency = 6
fakeUIDs = 1
I know that the .frg file can also override some options, so I am
pasting this file as well:
{VER
ver:2
}
{LIB
act:A
acc:dmel
ori:U
mea:0.000
std:0.000
src:
.
nft:20
fea:
forceBOGunitigger=1
isNotRandom=0
doNotTrustHomopolymerRuns=0
doTrim_initialNone=1
doTrim_initialMerBased=0
doTrim_initialFlowBased=0
doTrim_initialQualityBased=0
doRemoveDuplicateReads=0
doTrim_finalLargestCovered=1
doTrim_finalEvidenceBased=0
doTrim_finalBestEdge=0
doRemoveSpurReads=1
doRemoveChimericReads=1
doCheckForSubReads=0
doConsensusCorrection=0
forceShortReadFormat=0
constantInsertSize=0
fastqQualityValues=sanger
fastqOrientation=innie
fastqReads=/srv/data/combined_trimmed.fq
.
}
{VER
ver:1
}
Thanks very much for your help,
Rajiv
|
|
From: Walenz, B. <bw...@jc...> - 2014-03-14 15:35:37
|
Overlapper found no overlaps? You can check the log in ovlStore.err; it should report the number of overlaps it found. Also, you can check the various log files in 1-overlapper; at the end they report the number of kmer hits that seeded an overlap.
Kmer hits without olaps = 124433
Kmer hits with olaps = 290461
Multiple overlaps/pair = 0
Total overlaps produced = 290461
Contained overlaps = 72270
Dovetail overlaps = 218191
Rejected by short window = 0
Rejected by long window = 0
Did the read get trimmed out? Check ‘gatekeeper –dumpinfo –tabular *gkpStore’ to get a summary of the current state of the reads.
Is the minimum overlap length too large? Error rate too low? Send the parameters used.
This won’t resolve your lack of overlaps, but unitigger=bogart is suggested; you’re using the default unitigger=bog.
b
On 3/14/14 11:14 AM, "Price, Stephen" <s....@uc...> wrote:
Hi all,
Wondered if anyone could help. Running assembly using illumina and 454 reads, which seemed to be going well until it got to the constructing unitigs stage:
________________________________
more 4-unitigger/unitigger.err
/share/apps/genomics/MaSuRCA-2.2.1/CA/Linux-amd64/bin/buildUnitigs: AS_configure()-- AS_CGW_ERROR_RATE set to 0.25
Bubble popping = on
Intersection breaking = on
Bad mate threshold = -7
Error threshold = 0.015 (1.500%)
Error limit = 0.000 errors
Genome Size = 0
sizeof(DoveTailNode) = 32
AS_OVS_openBinaryOverlapFile()-- Failed to open '/home/ucbtpri/mascura2/CA/genome.ovlStore/0001' for reading: No such file or directory
________________________________
4-unitigger/ is empty besides the unitigger.err
genome.ovlStore/ is pretty bare – just empty ‘idx’ and ‘ovs'
No longer have any of the .ovb.gz either.
Haven’t found any error messages earlier in the process which have helped me to understand what’s gone wrong. Can anyone shed any light on it?
Any help appreciated. Cheers,
Stephen
|
|
From: Price, S. <s....@uc...> - 2014-03-14 15:15:02
|
Hi all, Wondered if anyone could help. Running assembly using illumina and 454 reads, which seemed to be going well until it got to the constructing unitigs stage: ________________________________ more 4-unitigger/unitigger.err /share/apps/genomics/MaSuRCA-2.2.1/CA/Linux-amd64/bin/buildUnitigs: AS_configure()-- AS_CGW_ERROR_RATE set to 0.25 Bubble popping = on Intersection breaking = on Bad mate threshold = -7 Error threshold = 0.015 (1.500%) Error limit = 0.000 errors Genome Size = 0 sizeof(DoveTailNode) = 32 AS_OVS_openBinaryOverlapFile()-- Failed to open '/home/ucbtpri/mascura2/CA/genome.ovlStore/0001' for reading: No such file or directory ________________________________ 4-unitigger/ is empty besides the unitigger.err genome.ovlStore/ is pretty bare - just empty 'idx' and 'ovs' No longer have any of the .ovb.gz either. Haven't found any error messages earlier in the process which have helped me to understand what's gone wrong. Can anyone shed any light on it? Any help appreciated. Cheers, Stephen |
|
From: Ole K. T. <o.k...@ib...> - 2014-03-11 08:44:08
|
Hi Brian, I think we've talked about this kind of problem before, but I couldn't find anything on how to solve it. Basically, cgw crash because it hit an assertion. I'm unsure of how to proceed. This is at ckp12-FCS, so I almost got my assembly. Thank you. Error message: RecomputeOffsetsInScaffold() returned RECOMPUTE_LAPACK on scaffold 246972; no more work will help. SetUngappedFragmentPositions()-- Encountered bgn==end==123 in ungapped coords within SetUngappedFragmentPositions for 35431063(gapped coords 129,129) cgw: MultiAlignment_CNS.C:1228: int SetUngappedFragmentPositions(FragType, int32, MultiAlignT*): Assertion `frag->position.bgn != frag->position.end' failed. Failed with 'Aborted' Backtrace (mangled): /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x41cbb7] /lib64/libpthread.so.0[0x3810a0f500] /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] /lib64/libc.so.6(abort+0x175)[0x380fe34085] /lib64/libc.so.6[0x380fe2ba1e] /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z22AppendFragToLocalStore8FragTypeiii10UnitigType+0xdcb)[0x4ee32b] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z24MergeMultiAlignsFast_newP12VarArrayTypeP11CNS_Options+0x3c7)[0x4f8db7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z23CreateAContigInScaffoldP9NodeCGW_TP12VarArrayType7LengthTS3_+0x131)[0x44ae71] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z16CleanupAScaffoldP14ScaffoldGraphTP9NodeCGW_Tiii+0xad4)[0x44cf54] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z24LeastSquaresGapEstimatesP14ScaffoldGraphTP9NodeCGW_Tjj+0x3a8)[0x497ad8] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x162b)[0x41a21b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] Backtrace (demangled): [0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, siginfo*, void*) + 0x27 [0x41cbb7] [1] /lib64/libpthread.so.0() [0x3810a0f500] [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] [4] /lib64/libc.so.6() [0x380fe2ba1e] [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AppendFragToLocalStore(FragType, int, int, int, UnitigType) + 0xdcb [0x4ee32b] [7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MergeMultiAlignsFast_new(VarArrayType*, CNS_Options*) + 0x3c7 [0x4f8db7] [8] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::CreateAContigInScaffold(NodeCGW_T*, VarArrayType*, LengthT, LengthT) + 0x131 [0x44ae71] [9] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::CleanupAScaffold(ScaffoldGraphT*, NodeCGW_T*, int, int, int) + 0xad4 [0x44cf54] [10] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LeastSquaresGapEstimates(ScaffoldGraphT*, NodeCGW_T*, unsigned int, unsigned int) + 0x3a8 [0x497ad8] [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + 0x162b [0x41a21b] [12] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] GDB: Ole |
|
From: Ole K. T. <o.k...@ib...> - 2014-03-06 19:50:32
|
Thank you Brian. I took a back up of the tigStore and 7-0-CGW, and restarted cgw with another filter level (5 instead of 2), to just see what kind of contigs I might get from this. I'll investigate using your method later on. Ole On Mar 5, 2014, at 4:00 PM, Walenz, Brian wrote: > Hi, Ole- > > The good news is that the number isn't actually negative, its just being > printed incorrectly. The bad news is that this implies it is a VERY big > number. This number is stored on disk, and could mean you've got a > corrupted file. > > You can try backing up a cgw checkpoint or two. Easiest to try, but not > sure how far back the corruption goes. > > Instead, try to discover which untig/contig (I'm guessing contig) is broken, > then dump that contig through tigStore versions to find the first good > version. > > Without code modification, you can dump every contig. Once you know the > last one that works, walk forwards until you find the one that crashes. > There could be some old (deleted) contigs after the last one that prints. > Once you get the failing contig, back up tigStore versions to find the > latest one that works. > > Command like (where V is the cgw checkpoint version): > > tigStore -g *gkpStore -t *tigStore V -C -d layout | grep ^...tig > > Or, you can add a line to AS_CNS/MultiAlingStore.C function loadMultiAlign() > (at about line 611) to report whatever is being loaded. There are two > example fprintf calls that you can use. With the modification, you can just > rerun cgw (or dump tigStore). > > Unfortunately, I don't see much of a place for a bug here. The process > really is 'read a number from disk, allocate that much memory'. If it is a > corrupt file, then the contig data is also probably garbage. > > b > > > > On 3/4/14 1:39 PM, "Ole Kristian Tørresen" <o.k...@ib...> wrote: > >> Hi, >> cgw crashes trying to allocate a negative amount of memory. This is >> with CA 8.1. I've tried -recomputegaps, but that didn't help (it ended >> in the second crash below). >> >> First crash: >> Could not malloc memory (-47244640206 bytes) >> cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != >> __null' failed. >> >> Failed with 'Aborted' >> >> Backtrace (mangled): >> >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7 >> siginfoPv+0x27)[0x41cbb7] >> /lib64/libpthread.so.0[0x3810a0f500] >> /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] >> /lib64/libc.so.6(abort+0x175)[0x380fe34085] >> /lib64/libc.so.6[0x380fe2ba1e] >> /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFro >> mStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromS >> treamP8_IO_FILE+0x24)[0x4e8ae4] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14lo >> adMultiAlignEib+0x20b)[0x43f09b] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10Graph >> CGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10Grap >> hCGW_Tii10PairOrientiidi+0x145)[0x46a695] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4e696c] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17check_consistencyP15 >> Scaffold_Fill_tii+0x8b2)[0x4e7322] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z9Fill_GapsPcii+0x498)[ >> 0x435f78] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x1bfd)[0x41a7ed] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] >> >> Backtrace (demangled): >> >> [0] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, >> siginfo*, void*) + 0x27 [0x41cbb7] >> [1] /lib64/libpthread.so.0() [0x3810a0f500] >> [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] >> [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] >> [4] /lib64/libc.so.6() [0x380fe2ba1e] >> [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] >> [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] >> [7] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromSt >> ream(_IO_FILE*, >> MultiAlignT*) + 0x75 [0x4e8425] >> [8] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStre >> am(_IO_FILE*) >> + 0x24 [0x4e8ae4] >> [9] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMu >> ltiAlign(int, >> bool) + 0x20b [0x43f09b] >> [10] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T >> *, >> int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] >> [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] >> [12] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_ >> T*, >> int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] >> [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4e696c] >> [14] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::check_consistency(Scaff >> old_Fill_t*, >> int, int) + 0x8b2 [0x4e7322] >> [15] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::Fill_Gaps(char*, >> int, int) + 0x498 [0x435f78] >> [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + >> 0x1bfd [0x41a7ed] >> [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] >> [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] >> >> GDB: >> >> And with -recompute gaps. >> >> Could not malloc memory (-4928055532461031424 bytes) >> cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != >> __null' failed. >> >> Failed with 'Aborted' >> >> Backtrace (mangled): >> >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7 >> siginfoPv+0x27)[0x41cbb7] >> /lib64/libpthread.so.0[0x3810a0f500] >> /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] >> /lib64/libc.so.6(abort+0x175)[0x380fe34085] >> /lib64/libc.so.6[0x380fe2ba1e] >> /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFro >> mStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromS >> treamP8_IO_FILE+0x24)[0x4e8ae4] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14lo >> adMultiAlignEib+0x20b)[0x43f09b] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10Graph >> CGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10Grap >> hCGW_Tii10PairOrientiidi+0x145)[0x46a695] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x492286] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x495be6] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z24LeastSquaresGapEstim >> atesP14ScaffoldGraphTP9NodeCGW_Tjj+0x135)[0x497865] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x10f7)[0x419ce7] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] >> >> >> Backtrace (demangled): >> >> [0] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, >> siginfo*, void*) + 0x27 [0x41cbb7] >> [1] /lib64/libpthread.so.0() [0x3810a0f500] >> [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] >> [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] >> [4] /lib64/libc.so.6() [0x380fe2ba1e] >> [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] >> [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] >> [7] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromSt >> ream(_IO_FILE*, >> MultiAlignT*) + 0x75 [0x4e8425] >> [8] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStre >> am(_IO_FILE*) >> + 0x24 [0x4e8ae4] >> [9] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMu >> ltiAlign(int, >> bool) + 0x20b [0x43f09b] >> [10] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T >> *, >> int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] >> [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] >> [12] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_ >> T*, >> int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] >> [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x492286] >> [14] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x495be6] >> [15] >> /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LeastSquaresGapEstimate >> s(ScaffoldGraphT*, >> NodeCGW_T*, unsigned int, unsigned int) + 0x135 [0x497865] >> [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + >> 0x10f7 [0x419ce7] >> [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] >> [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] >> >> GDB: >> >> Brian, do you know what might be wrong here, and how I can fix it? >> >> Thank you, >> >> >> ------------------------------------------------------------------------------ >> Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. >> With Perforce, you get hassle-free workflows. Merge that actually works. >> Faster operations. Version large binaries. Built-in WAN optimization and the >> freedom to use Git, Perforce or both. Make the move to Perforce. >> http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk >> _______________________________________________ >> wgs-assembler-users mailing list >> wgs...@li... >> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users > |
|
From: Walenz, B. <bw...@jc...> - 2014-03-05 15:00:37
|
Hi, Ole- The good news is that the number isn't actually negative, its just being printed incorrectly. The bad news is that this implies it is a VERY big number. This number is stored on disk, and could mean you've got a corrupted file. You can try backing up a cgw checkpoint or two. Easiest to try, but not sure how far back the corruption goes. Instead, try to discover which untig/contig (I'm guessing contig) is broken, then dump that contig through tigStore versions to find the first good version. Without code modification, you can dump every contig. Once you know the last one that works, walk forwards until you find the one that crashes. There could be some old (deleted) contigs after the last one that prints. Once you get the failing contig, back up tigStore versions to find the latest one that works. Command like (where V is the cgw checkpoint version): tigStore -g *gkpStore -t *tigStore V -C -d layout | grep ^...tig Or, you can add a line to AS_CNS/MultiAlingStore.C function loadMultiAlign() (at about line 611) to report whatever is being loaded. There are two example fprintf calls that you can use. With the modification, you can just rerun cgw (or dump tigStore). Unfortunately, I don't see much of a place for a bug here. The process really is 'read a number from disk, allocate that much memory'. If it is a corrupt file, then the contig data is also probably garbage. b On 3/4/14 1:39 PM, "Ole Kristian Tørresen" <o.k...@ib...> wrote: > Hi, > cgw crashes trying to allocate a negative amount of memory. This is > with CA 8.1. I've tried -recomputegaps, but that didn't help (it ended > in the second crash below). > > First crash: > Could not malloc memory (-47244640206 bytes) > cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != > __null' failed. > > Failed with 'Aborted' > > Backtrace (mangled): > > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7 > siginfoPv+0x27)[0x41cbb7] > /lib64/libpthread.so.0[0x3810a0f500] > /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] > /lib64/libc.so.6(abort+0x175)[0x380fe34085] > /lib64/libc.so.6[0x380fe2ba1e] > /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFro > mStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromS > treamP8_IO_FILE+0x24)[0x4e8ae4] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14lo > adMultiAlignEib+0x20b)[0x43f09b] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10Graph > CGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10Grap > hCGW_Tii10PairOrientiidi+0x145)[0x46a695] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4e696c] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17check_consistencyP15 > Scaffold_Fill_tii+0x8b2)[0x4e7322] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z9Fill_GapsPcii+0x498)[ > 0x435f78] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x1bfd)[0x41a7ed] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] > > Backtrace (demangled): > > [0] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, > siginfo*, void*) + 0x27 [0x41cbb7] > [1] /lib64/libpthread.so.0() [0x3810a0f500] > [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] > [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] > [4] /lib64/libc.so.6() [0x380fe2ba1e] > [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] > [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] > [7] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromSt > ream(_IO_FILE*, > MultiAlignT*) + 0x75 [0x4e8425] > [8] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStre > am(_IO_FILE*) > + 0x24 [0x4e8ae4] > [9] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMu > ltiAlign(int, > bool) + 0x20b [0x43f09b] > [10] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T > *, > int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] > [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] > [12] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_ > T*, > int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] > [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4e696c] > [14] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::check_consistency(Scaff > old_Fill_t*, > int, int) + 0x8b2 [0x4e7322] > [15] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::Fill_Gaps(char*, > int, int) + 0x498 [0x435f78] > [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + > 0x1bfd [0x41a7ed] > [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] > [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] > > GDB: > > And with -recompute gaps. > > Could not malloc memory (-4928055532461031424 bytes) > cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != > __null' failed. > > Failed with 'Aborted' > > Backtrace (mangled): > > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7 > siginfoPv+0x27)[0x41cbb7] > /lib64/libpthread.so.0[0x3810a0f500] > /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] > /lib64/libc.so.6(abort+0x175)[0x380fe34085] > /lib64/libc.so.6[0x380fe2ba1e] > /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFro > mStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromS > treamP8_IO_FILE+0x24)[0x4e8ae4] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14lo > adMultiAlignEib+0x20b)[0x43f09b] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10Graph > CGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10Grap > hCGW_Tii10PairOrientiidi+0x145)[0x46a695] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x492286] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x495be6] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z24LeastSquaresGapEstim > atesP14ScaffoldGraphTP9NodeCGW_Tjj+0x135)[0x497865] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x10f7)[0x419ce7] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] > > > Backtrace (demangled): > > [0] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, > siginfo*, void*) + 0x27 [0x41cbb7] > [1] /lib64/libpthread.so.0() [0x3810a0f500] > [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] > [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] > [4] /lib64/libc.so.6() [0x380fe2ba1e] > [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] > [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] > [7] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromSt > ream(_IO_FILE*, > MultiAlignT*) + 0x75 [0x4e8425] > [8] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStre > am(_IO_FILE*) > + 0x24 [0x4e8ae4] > [9] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMu > ltiAlign(int, > bool) + 0x20b [0x43f09b] > [10] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T > *, > int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] > [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] > [12] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_ > T*, > int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] > [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x492286] > [14] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x495be6] > [15] > /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LeastSquaresGapEstimate > s(ScaffoldGraphT*, > NodeCGW_T*, unsigned int, unsigned int) + 0x135 [0x497865] > [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + > 0x10f7 [0x419ce7] > [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] > [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] > > GDB: > > Brian, do you know what might be wrong here, and how I can fix it? > > Thank you, > > > ------------------------------------------------------------------------------ > Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. > With Perforce, you get hassle-free workflows. Merge that actually works. > Faster operations. Version large binaries. Built-in WAN optimization and the > freedom to use Git, Perforce or both. Make the move to Perforce. > http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk > _______________________________________________ > wgs-assembler-users mailing list > wgs...@li... > https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users |
|
From: Ole K. T. <o.k...@ib...> - 2014-03-04 19:02:48
|
Hi, cgw crashes trying to allocate a negative amount of memory. This is with CA 8.1. I've tried -recomputegaps, but that didn't help (it ended in the second crash below). First crash: Could not malloc memory (-47244640206 bytes) cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != __null' failed. Failed with 'Aborted' Backtrace (mangled): /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x41cbb7] /lib64/libpthread.so.0[0x3810a0f500] /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] /lib64/libc.so.6(abort+0x175)[0x380fe34085] /lib64/libc.so.6[0x380fe2ba1e] /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFromStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromStreamP8_IO_FILE+0x24)[0x4e8ae4] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14loadMultiAlignEib+0x20b)[0x43f09b] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10GraphCGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10GraphCGW_Tii10PairOrientiidi+0x145)[0x46a695] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4e696c] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17check_consistencyP15Scaffold_Fill_tii+0x8b2)[0x4e7322] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z9Fill_GapsPcii+0x498)[0x435f78] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x1bfd)[0x41a7ed] /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] Backtrace (demangled): [0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, siginfo*, void*) + 0x27 [0x41cbb7] [1] /lib64/libpthread.so.0() [0x3810a0f500] [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] [4] /lib64/libc.so.6() [0x380fe2ba1e] [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] [7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromStream(_IO_FILE*, MultiAlignT*) + 0x75 [0x4e8425] [8] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStream(_IO_FILE*) + 0x24 [0x4e8ae4] [9] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMultiAlign(int, bool) + 0x20b [0x43f09b] [10] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T*, int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] [12] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_T*, int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4e696c] [14] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::check_consistency(Scaffold_Fill_t*, int, int) + 0x8b2 [0x4e7322] [15] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::Fill_Gaps(char*, int, int) + 0x498 [0x435f78] [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + 0x1bfd [0x41a7ed] [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] GDB: And with -recompute gaps. Could not malloc memory (-4928055532461031424 bytes) cgw: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != __null' failed. Failed with 'Aborted' Backtrace (mangled): /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x41cbb7] /lib64/libpthread.so.0[0x3810a0f500] /lib64/libc.so.6(gsignal+0x35)[0x380fe328a5] /lib64/libc.so.6(abort+0x175)[0x380fe34085] /lib64/libc.so.6[0x380fe2ba1e] /lib64/libc.so.6(__assert_perror_fail+0x0)[0x380fe2bae0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x4cdaf7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z27ReLoadMultiAlignTFromStreamP8_IO_FILEP11MultiAlignT+0x75)[0x4e8425] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z25LoadMultiAlignTFromStreamP8_IO_FILE+0x24)[0x4e8ae4] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_ZN15MultiAlignStore14loadMultiAlignEib+0x20b)[0x43f09b] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z12GetConsensusP10GraphCGW_TiP12VarArrayTypeS2_+0x8b)[0x46b68b] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x469a53] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z13OverlapChunksP10GraphCGW_Tii10PairOrientiidi+0x145)[0x46a695] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x492286] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x495be6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(_Z24LeastSquaresGapEstimatesP14ScaffoldGraphTP9NodeCGW_Tjj+0x135)[0x497865] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw(main+0x10f7)[0x419ce7] /lib64/libc.so.6(__libc_start_main+0xfd)[0x380fe1ecdd] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw[0x418899] Backtrace (demangled): [0] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::AS_UTL_catchCrash(int, siginfo*, void*) + 0x27 [0x41cbb7] [1] /lib64/libpthread.so.0() [0x3810a0f500] [2] /lib64/libc.so.6::(null) + 0x35 [0x380fe328a5] [3] /lib64/libc.so.6::(null) + 0x175 [0x380fe34085] [4] /lib64/libc.so.6() [0x380fe2ba1e] [5] /lib64/libc.so.6::(null) + 0 [0x380fe2bae0] [6] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x4cdaf7] [7] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::ReLoadMultiAlignTFromStream(_IO_FILE*, MultiAlignT*) + 0x75 [0x4e8425] [8] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LoadMultiAlignTFromStream(_IO_FILE*) + 0x24 [0x4e8ae4] [9] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::MultiAlignStore::loadMultiAlign(int, bool) + 0x20b [0x43f09b] [10] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::GetConsensus(GraphCGW_T*, int, VarArrayType*, VarArrayType*) + 0x8b [0x46b68b] [11] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x469a53] [12] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::OverlapChunks(GraphCGW_T*, int, int, PairOrient, int, int, double, int) + 0x145 [0x46a695] [13] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x492286] [14] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x495be6] [15] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::LeastSquaresGapEstimates(ScaffoldGraphT*, NodeCGW_T*, unsigned int, unsigned int) + 0x135 [0x497865] [16] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw::(null) + 0x10f7 [0x419ce7] [17] /lib64/libc.so.6::(null) + 0xfd [0x380fe1ecdd] [18] /projects/cees/bin/celera/wgs-8.1/Linux-amd64/bin/cgw() [0x418899] GDB: Brian, do you know what might be wrong here, and how I can fix it? Thank you, |
|
From: Yeo Z. X. <zhe...@ya...> - 2014-01-07 02:22:28
|
Thanks, Brian. Really appreciate you advice! Zhenxuan ________________________________________ From: Walenz, Brian [bw...@jc...] Sent: Tuesday, January 07, 2014 1:13 AM To: Yeo Zhen Xuan; wgs...@li... Cc: Bai Chang Subject: RE: BOGART error due to large data Hi- The 'bad_alloc' error indicates 'out of memory'. Bogart is assuming it has the entire machine to itself. In your case, this would be all 12 cores and all 1 tb of memory. If there are other processes on the machine, it can easily run out of memory. You can limit bogart with runCA options 'batThreads' and 'batMemory'. The default ovlMinLen is probably too small. It was set to 40 back when 10x coverage of Sanger reads was usual. With higher coverage, you can increase the minimum. 50 seems reasonable. This will cut down on the number of overlaps, most of which will be spurious or redundant. Fewer overlaps means less disk and memory usage. In your run, bogart could fit 77% of all overlaps into 1tb of memory, but ran out of space after 69%. 'batMemory=800' (gb) should limit bogart to loading around 65% of the overlaps. b ________________________________________ From: Yeo Zhen Xuan [zhe...@ya...] Sent: Monday, January 06, 2014 12:26 AM To: wgs...@li... Cc: Bai Chang Subject: [wgs-assembler-users] BOGART error due to large data Dear wgs-assembler admins, I am new to runCA. The IT supports from my institute had installed CA8.0 a few months ago. I managed to generate an assembly using my Illumina Miseq data with both the BOG and BOGART Unitiggers. However, only BOGART failed to complete when I included a bigger Hiseq dataset on top of the Miseq dataset (see attached unittiger.err: "terminate called after throwing an instance of 'std::bad_alloc'"). Since BOGART is the recommended Unitigger for Illumina data, we hope to apply it to generate our assembly using all the data we have. For your information, the BOGART is working when I used a subset of the Miseq + Hiseq dataset for assembly by setting the frgMinLen=100 and ovlMinLen=50, suggesting the error is likely due to the size of input data. Our Hiseq data is ~80M of paired-reads (~160M read fragments in total) with most of them are 100bp read after merTrim trimming ( ~50GB of data). The Miseq data is ~10 times smaller than the Hiseq data. Thank you for your supports. Best regards, Zhen Xuan YEO ------------------------------------------------------------ Senior Bioinformatics Specialist @ Yale-NUS College Centre for BioImaging Sciences Department of Biological Sciences Blk S1A, 14 Science Drive 4 Lee Wee Kheng Building 117557 Singapore (O) +65 65162723 (F) +65 67767882 Email: zhe...@ya... |
|
From: Walenz, B. <bw...@jc...> - 2014-01-06 17:14:47
|
Hi- The 'bad_alloc' error indicates 'out of memory'. Bogart is assuming it has the entire machine to itself. In your case, this would be all 12 cores and all 1 tb of memory. If there are other processes on the machine, it can easily run out of memory. You can limit bogart with runCA options 'batThreads' and 'batMemory'. The default ovlMinLen is probably too small. It was set to 40 back when 10x coverage of Sanger reads was usual. With higher coverage, you can increase the minimum. 50 seems reasonable. This will cut down on the number of overlaps, most of which will be spurious or redundant. Fewer overlaps means less disk and memory usage. In your run, bogart could fit 77% of all overlaps into 1tb of memory, but ran out of space after 69%. 'batMemory=800' (gb) should limit bogart to loading around 65% of the overlaps. b ________________________________________ From: Yeo Zhen Xuan [zhe...@ya...] Sent: Monday, January 06, 2014 12:26 AM To: wgs...@li... Cc: Bai Chang Subject: [wgs-assembler-users] BOGART error due to large data Dear wgs-assembler admins, I am new to runCA. The IT supports from my institute had installed CA8.0 a few months ago. I managed to generate an assembly using my Illumina Miseq data with both the BOG and BOGART Unitiggers. However, only BOGART failed to complete when I included a bigger Hiseq dataset on top of the Miseq dataset (see attached unittiger.err: "terminate called after throwing an instance of 'std::bad_alloc'"). Since BOGART is the recommended Unitigger for Illumina data, we hope to apply it to generate our assembly using all the data we have. For your information, the BOGART is working when I used a subset of the Miseq + Hiseq dataset for assembly by setting the frgMinLen=100 and ovlMinLen=50, suggesting the error is likely due to the size of input data. Our Hiseq data is ~80M of paired-reads (~160M read fragments in total) with most of them are 100bp read after merTrim trimming ( ~50GB of data). The Miseq data is ~10 times smaller than the Hiseq data. Thank you for your supports. Best regards, Zhen Xuan YEO ------------------------------------------------------------ Senior Bioinformatics Specialist @ Yale-NUS College Centre for BioImaging Sciences Department of Biological Sciences Blk S1A, 14 Science Drive 4 Lee Wee Kheng Building 117557 Singapore (O) +65 65162723 (F) +65 67767882 Email: zhe...@ya... |
9 messages has been excluded from this view by a project administrator.
