Re: [wgs-assembler-users] runCA stopped while updating overlapStore - how to resume???

[Walenz, B. <bw...@jc...>]

Hi, Christoph-

Good to hear!  You're the third person (I know of) to run the parallel
version.  Instead of fixing the older store build, I'd rather spend time to
integrate the new one with runCA, either as a set of jobs for SGE, or a
series of sequential jobs.  It's just scripting, but there might be some
performance issues to optimize.

If the store is complete, the bucket directories can be deleted.  The third
step should have done this for you.  Maybe not if you didn't enable
deleteearly or deletelate.  The store is complete if you have just the ####
files, an 'idx' and an 'ovs' file.  You should not have any ####.idx or
####.ovs files.  Is the extra space in the bucket??? directories?   The
difference (546 - 320 = 226) seems to be a reasonable size for the buckets.

Memory for buildUnitgs (aka bog) cannot be specified.  There isn't any data
we can keep on disk, or not load, or compute differently in a smaller memory
size.  Memory is used to store fragment meta data (clear lengths, mate
pairs) and best overlaps, and constructed unitigs.  The first two are of a
known size.  The number of unitigs depends on the assembly.  We've seen an
assembly that exhausted memory in bog, caused by junk fragments creating an
enormous number of single-fragment unitigs.

b



On 7/13/12 1:53 PM, "Christoph Hahn" <chr...@gm...> wrote:

> Hi Brian,
>
> It s done! I have by now also updated the overlapStore with the frg- and
> ovlcorr and I am in the process of building unitigs now.
>
> I like this parallel version for building the ovlStore. You were right
> the last jobs needed double the memory. When distributing the jobs to
> several CPUs it is very time efficient and also used fewer overall
> CPUhours in comparison to the regular overlapStore command. One thing
> though is that I think it needs substantially more disk space. I am not
> 100% sure (because its gone now..), but I believe the *.ovlStore build
> by the regular command used some 320G of disk space, while the one I
> have now is using 546G. Are all the bucket???? directories in *.ovlStore
> still needed?
>
> Overall I think I learned a lot about CA by running the latest steps
> again with the parallel version of ovlStore build and your help. Are
> there plans to include a failsafe for the overlapStore update function,
> until the process is finished? So that it can be resumed in case it
> stops for whatever reason.
>
> One more thing: Is there a way to specify the memory buildUnitigs is
> using?
>
> Thanks again for your help!!
>
> cheers,
> Christoph
>
>
> On 12.07.2012 18:52, Walenz, Brian wrote:
>> You've captured the process nicely.
>>
>> After #1 finishes, check that you have one 'sliceSizes' file per bucket
>> directory.  If any are missing, run that bucket again.  I think (hope) that
>> #2 will complain if any are missing, but this has been a problem in the past.
>>
>> Hopefully memory won't be an issue during sorting.  I estimate memory size as
>> 3 * (sizeof gz files) / #jobs.  But, if you have Illumina + long reads (454+,
>> Sanger), the balancing is screwed up and the early jobs (overlaps of Illumina
>> to Illumina) have fewer overlaps than the later jobs (Illumina to long
>> reads).  Every time I've run this, I could do 90-95% of the sort jobs on our
>> grid, but had to use large memory machines for the rest.
>>
>> More jobs creates more files, but I don't think it is necessarily slower.  I
>> haven't benchmarked it though.
>>
>> No jobID for #3, it is tiny, does little compute, and not too much I/O.  I
>> usually run this interactively off grid.
>>
>> b
>>
>> ________________________________________
>> From: Christoph Hahn [chr...@gm...]
>> Sent: Thursday, July 12, 2012 9:31 AM
>> To: Walenz, Brian
>> Cc: wgs...@li...
>> Subject: Re: [wgs-assembler-users] runCA stopped while updating overlapStore
>> - how to resume???
>>
>> Hi Brian,
>>
>> I ran the runCA-overlapStoreBuild.pl script now. It created the three
>> scripts:
>> 1-bucketize.sh
>> 2-sort.sh
>> 3-index.sh
>>
>> right now I am running 1-bucketize.sh for every job index from 1 to
>> 2135. I have distributed the jobs on several CPUs and that works nicely.
>>
>> when this is finished I need to run 2-sort.sh. I specified -jobs 100 in
>> the runCA-overlapStoreBuild.pl, so as far as I understand it should have
>> created 100 jobs, right? So, I run 2-sort.sh for jobIDs 1 to 100, then?
>> the jobID in this case is actually the slicenumber, right? so, for e.g.
>> 2-sort.sh 2 it will look through all bucket directories and pull out
>> slice002.gz, read them into memory and write the overlaps into the store.
>>
>> When this is done I just need to run 3-index.sh once. No jobIDs
>> required, right?
>>
>> Am I missing anything?
>>
>> cheers,
>> Christoph
>>
>>
>> On 07/11/2012 05:54 AM, Walenz, Brian wrote:
>>> The first step will create 1 job for each overlapper job.  These should be
>>> small memory, but there is some internal buffering done and I usually
>>> request 2gb for them anyway.
>>>
>>> The second step will create '-jobs j' jobs.  Memory size here is a giant
>>> unknown.  The '-memory m' option will cause the job to not run if it needs
>>> more than that much memory.  Currently, you'll have to increase -memory for
>>> these jobs and find a bigger machine.
>>>
>>> All jobs in both steps are single-threaded and run independently of each
>>> other.
>>>
>>> b
>>>
>>>
>>>
>>>
>>> On 7/10/12 6:46 PM, "Christoph Hahn"<chr...@gm...>  wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> Thanks! overlaps are being computed now and CVS version of CA has been
>>>> successfully compiled. Will try the runCA-overlapStoreBuild.pl once the
>>>> overlapper is finished. One question there: I understand that the memory
>>>> usage is regulated by the -jobs j parameter. higher value for j means
>>>> less memory for every job. How can I specify the number of CPUs to be
>>>> used in the parallel steps?
>>>>
>>>> Thanks for your help! I appreciate it!
>>>>
>>>> cheers,
>>>> Christoph
>>>>
>>>> On 07/10/2012 10:18 PM, Walenz, Brian wrote:
>>>>> Quick guess is that runCA is finding the old ovlStore and assuming it is
>>>>> complete, then continuing on to frgcorr.  runCA tests for the existence of
>>>>> name.ovlStore to determine if overlaps are finished; it doesn't check that
>>>>> the store is valid.  So, delete *ovlStore* too.
>>>>>
>>>>> Your latest build (from scratch) is suffering from a long standing
>>>>> dependency issue.  It needs kmer checked out and 'make install'ed.
>>>>>
>>>>> make[1]: *** No rule to make target `sweatShop.H', needed by
>>>>> `classifyMates.o'.  Stop.
>>>>> make[1]: *** Waiting for unfinished jobs....
>>>>> make: *** [objs] Error 1
>>>>>
>>>>> Once kmer is installed, wipe (again) the Linux-amd64 and rebuild.
>>>>>
>>>>> The kmer included in CA7 is too old for the CVS version of CA, so you'll
>>>>> need to grab it from subversion.
>>>>>
>>>>> http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=Check_
>>>>> ou
>>>>> t_and_Compile
>>>>>
>>>>> b
>>>>>
>>>>>
>>>>> On 7/10/12 4:00 PM, "Christoph Hahn"<chr...@gm...>  wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I actually tried to just rerun the overlapper. I moved the 1-overlapper
>>>>>> and the 3-overlapcorrection directories and just ran runCA and it
>>>>>> immediately starts with doing frgcorr. Do you mean recompute from the
>>>>>> very start? Is there a way to avoid recomputing the initial overlaps at
>>>>>> least(it took some 10000 CPUhours)??
>>>>>>
>>>>>> Tried to compile it again - not successful. Ran make in the src
>>>>>> directory (output in makelog) and also in the AS_RUN directory (output
>>>>>> AS_RUN-makelog).
>>>>>>
>>>>>> Thanks,
>>>>>> Christoph
>>>>>>
>>>>>>
>>>>>> On 07/10/2012 09:04 PM, Walenz, Brian wrote:
>>>>>>> Odd, the *gz should only be deleted after the store is successfully
>>>>>>> built.
>>>>>>> runCA might have been confused by the attempt to rerun.  The easiest
>>>>>>> will
>>>>>>> be
>>>>>>> to recompute.  :-(
>>>>>>>
>>>>>>> I've never seen the 'libCA.a' error before.  That particular program is
>>>>>>> the
>>>>>>> first to get built.  Looks like libCA.a wasn't created.  My fix for most
>>>>>>> strange compile errors is to remove the entire Linux-amd64 directory and
>>>>>>> recompile.  If that fails, send along the complete output of make and
>>>>>>> I'll
>>>>>>> take a look.
>>>>>>>
>>>>>>> b
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 7/10/12 2:15 PM, "Christoph Hahn"<chr...@gm...>  wrote:
>>>>>>>
>>>>>>>> Hi Brian,
>>>>>>>>
>>>>>>>> Thanks for your reply!
>>>>>>>>
>>>>>>>> I would be happy to try the new parallel overlap store build, but I
>>>>>>>> think I need the *.ovb.gz outputs for that and unfortunately I dont
>>>>>>>> have
>>>>>>>> them any more. Looks like they were deleted after the ovlStore was
>>>>>>>> build. So I guess I ll need to run the overlapper again, first. Am I
>>>>>>>> understanding that correctly?
>>>>>>>>
>>>>>>>> I have downloaded the cvs and tried to make, but I get:
>>>>>>>> *** No rule to make target `libCA.a', needed by `fragmentDepth'. Stop.
>>>>>>>>
>>>>>>>> I really appreciate your help!
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>> Christoph
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/10/2012 05:09 PM, Walenz, Brian wrote:
>>>>>>>>> Hi, Christoph-
>>>>>>>>>
>>>>>>>>> The original overlap store build is difficult to resume.  I think it
>>>>>>>>> can
>>>>>>>>> be
>>>>>>>>> done, but it will take code changes that are probably specific to the
>>>>>>>>> case
>>>>>>>>> you have.  Only if you do not have the *ovb.gz outputs from overlapper
>>>>>>>>> will
>>>>>>>>> I suggest this.
>>>>>>>>>
>>>>>>>>> Option 1 is then to restart.
>>>>>>>>>
>>>>>>>>> Option 2 is to use a new 'data-parallel' overlap store build
>>>>>>>>> (AS_RUN/runCA-overlapStoreBuild.pl).  It runs as a series of three
>>>>>>>>> grid
>>>>>>>>> jobs.  The first job is parallel, and transfers the overlapper output
>>>>>>>>> into
>>>>>>>>> buckets for sorting.  The second job, also parallel, sorts each
>>>>>>>>> bucket.
>>>>>>>>> The
>>>>>>>>> final job, sequential, builds an index for the store.  Since this
>>>>>>>>> compute
>>>>>>>>> is
>>>>>>>>> just a collection of jobs, it can be restarted/resumed/fixed easily.
>>>>>>>>>
>>>>>>>>> Its performance can be great -- at JCVI we've seen builds that we
>>>>>>>>> estimated
>>>>>>>>> would take 2 days using the original sequential build, finish in a few
>>>>>>>>> (4?)
>>>>>>>>> hours with the data parallel version.  But on our development cluster,
>>>>>>>>> it
>>>>>>>>> is
>>>>>>>>> slower than the sequential version.  It depends on the disk
>>>>>>>>> throughput.
>>>>>>>>> Our
>>>>>>>>> dev cluster is powered off of a 6-disk ZFS, while the production side
>>>>>>>>> has
>>>>>>>>> a
>>>>>>>>> big Isilon.
>>>>>>>>>
>>>>>>>>> It is only in CVS.  I just added command line help and a bit of
>>>>>>>>> documentation, so do an update first.
>>>>>>>>>
>>>>>>>>> Happy to provide help if you want to try it out.  More than happy to
>>>>>>>>> accept
>>>>>>>>> better documentation.
>>>>>>>>>
>>>>>>>>> b
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 7/10/12 6:47 AM, "Christoph Hahn"<chr...@gm...>  wrote:
>>>>>>>>>
>>>>>>>>>> Hei Ole,
>>>>>>>>>>
>>>>>>>>>> Thanks for your reply. I had looked on the preprocessing page you are
>>>>>>>>>> referring to just recently. Sounds like a good approach you are
>>>>>>>>>> using!
>>>>>>>>>> Will definitely consider that to make the assembly more effective in
>>>>>>>>>> a
>>>>>>>>>> next try. Thanks for that!
>>>>>>>>>> For now, I think I am pretty much over all the trimming and
>>>>>>>>>> correction
>>>>>>>>>> steps (once I get this last thing sorted out..). As far as I can see
>>>>>>>>>> the
>>>>>>>>>> next step is already building the unitigs, so I ll try to finish this
>>>>>>>>>> assembly as it is now. Will try to improve it afterwards. I am really
>>>>>>>>>> curious how a first attempt of a hybrid approach (454+illumina) will
>>>>>>>>>> perform in comparison to the pure illumina assemblies which I have
>>>>>>>>>> pretty much optimized now (and with which I am pretty happy, btw), I
>>>>>>>>>> think.
>>>>>>>>>>
>>>>>>>>>> I am afraid, your suggestion to do doFragmentCorrection=0 directly
>>>>>>>>>> now
>>>>>>>>>> will not work. For the next step (the unitigger) I ll need an intact
>>>>>>>>>> overlap store. As it is now, I think it is useless, being only
>>>>>>>>>> half-updated.. I also discovered that just rerunning the previous
>>>>>>>>>> overlapStore command (the one before the frg- and ovlcorrection) is
>>>>>>>>>> not
>>>>>>>>>> working as I thought it would.
>>>>>>>>>> Seems to be a very unfortunate situation - really dont know how to
>>>>>>>>>> proceed.. It would be fantastic if anyone could give me a tip what to
>>>>>>>>>> do!!
>>>>>>>>>>
>>>>>>>>>> Thanks for your help!
>>>>>>>>>>
>>>>>>>>>> much obliged,
>>>>>>>>>> Christoph
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 09.07.2012 13:20, Ole Kristian Tørresen wrote:
>>>>>>>>>>> Hi Christoph.
>>>>>>>>>>>
>>>>>>>>>>> This is not an answer to your question, but a suggestion for a
>>>>>>>>>>> work-around. If I remember correctly, you have both Illumina and 454
>>>>>>>>>>> reads. Celera runs, as you see below, frgcorrection and overlap
>>>>>>>>>>> based
>>>>>>>>>>> trimming to correct 454 reads, and merTrim to correct Illumina reads
>>>>>>>>>>> (can also be used on 454 reads). What I've been doing lately, is to
>>>>>>>>>>> run meryl on a trusted set of Illumina reads, pair end for example,
>>>>>>>>>>> I
>>>>>>>>>>> ran it on some overlapping reads which I had merged with FLASH. Then
>>>>>>>>>>> you can use the set of trusted k-mers to correct different datasets.
>>>>>>>>>>> For example, I first ran CA to the end of OBT (overlap based
>>>>>>>>>>> trimming)
>>>>>>>>>>> for my 454 reads, and then output the result as fastq-files. I used
>>>>>>>>>>> the trusted k-mer set to correct these 454 reads too. If you do this
>>>>>>>>>>> for all your reads, used either merTim or merTrim/OBT, and do
>>>>>>>>>>> deduplication on all the datasets too, then you'll end up with reads
>>>>>>>>>>> that you can use in assemblies where you skip relatively expensive
>>>>>>>>>>> steps as frgcorrection.
>>>>>>>>>>>
>>>>>>>>>>> I don't think frgcorrection is that useful for the type of data
>>>>>>>>>>> you're
>>>>>>>>>>> using anyway.
>>>>>>>>>>>
>>>>>>>>>>> If you have a set of corrected reads, you can use these settings for
>>>>>>>>>>> CA:
>>>>>>>>>>> doOBT=0
>>>>>>>>>>> doFragmentCorrection=0
>>>>>>>>>>>
>>>>>>>>>>> When I think of it, you might use doFragmentCorrection=0 on this
>>>>>>>>>>> assembly now. You might have to clean up your directory tree, like
>>>>>>>>>>> removing the 3-overlapcorrection directory and maybe some other
>>>>>>>>>>> steps
>>>>>>>>>>> too. Apply with caution.
>>>>>>>>>>>
>>>>>>>>>>> Most of the stuff I've mentioned I've taken from here:
>>>>>>>>>>> http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=
>>>>>>>>>>> Pre
>>>>>>>>>>> pr
>>>>>>>>>>> oc
>>>>>>>>>>> es
>>>>>>>>>>> sing
>>>>>>>>>>> and discussion with Brian.
>>>>>>>>>>>
>>>>>>>>>>> Ole
>>>>>>>>>>>
>>>>>>>>>>> On 9 July 2012 12:47, Christoph Hahn<chr...@gm...>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Dear users and developers,
>>>>>>>>>>>>
>>>>>>>>>>>> I have the following problem: In my assembly process I have just
>>>>>>>>>>>> completed
>>>>>>>>>>>> the fragment- and overlap error correction. Unfortunately runCA
>>>>>>>>>>>> stopped
>>>>>>>>>>>> in
>>>>>>>>>>>> the subsequent updating of the overlapStore, because of an
>>>>>>>>>>>> incorrectly
>>>>>>>>>>>> set
>>>>>>>>>>>> time limit..
>>>>>>>>>>>> If I am trying to resume the assembly now, I get the following
>>>>>>>>>>>> error:
>>>>>>>>>>>> ----------------------------------------START Mon Jul  9 11:05:53
>>>>>>>>>>>> 2012
>>>>>>>>>>>> /xanadu/home/chrishah/programmes/wgs-7.0/Linux-amd64/bin/overlapSto
>>>>>>>>>>>> re
>>>>>>>>>>>> -u
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/salaris.ovlStore
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/3-overlapco
>>>>>>>>>>>> rrection/salaris.erates>
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/3-overlapcorrection/overlap
>>>>>>>>>>>> Sto
>>>>>>>>>>>> re
>>>>>>>>>>>> -u
>>>>>>>>>>>> pd
>>>>>>>>>>>> ate-erates.err
>>>>>>>>>>>> 2>&1
>>>>>>>>>>>> ----------------------------------------END Mon Jul  9 11:05:54
>>>>>>>>>>>> 2012
>>>>>>>>>>>> (1
>>>>>>>>>>>> seconds)
>>>>>>>>>>>> ERROR: Failed with signal HUP (1)
>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>> ===
>>>>>>>>>>>> ==
>>>>>>>>>>>> ==
>>>>>>>>>>>> ==
>>>>>>>>>>>> ====
>>>>>>>>>>>>
>>>>>>>>>>>> runCA failed.
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------
>>>>>>>>>>>> Stack trace:
>>>>>>>>>>>>
>>>>>>>>>>>>        at
>>>>>>>>>>>> /usit/titan/u1/chrishah/programmes/wgs-7.0/Linux-amd64/bin/./runCA
>>>>>>>>>>>> line
>>>>>>>>>>>> 1237
>>>>>>>>>>>>               main::caFailure('failed to apply the overlap
>>>>>>>>>>>> corrections',
>>>>>>>>>>>> '/projects/nn9201k/Celera/work2/salaris1/3-overlapcorrection/o...')
>>>>>>>>>>>> called
>>>>>>>>>>>> at /usit/titan/u1/chrishah/programmes/wgs
>>>>>>>>>>>> -7.0/Linux-amd64/bin/./runCA line 4077
>>>>>>>>>>>>               main::overlapCorrection() called at
>>>>>>>>>>>> /usit/titan/u1/chrishah/programmes/wgs-7.0/Linux-amd64/bin/./runCA
>>>>>>>>>>>> line
>>>>>>>>>>>> 5880
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------
>>>>>>>>>>>> Last few lines of the relevant log file
>>>>>>>>>>>> (/projects/nn9201k/Celera/work2/salaris1/3-overlapcorrection/overla
>>>>>>>>>>>> pSt
>>>>>>>>>>>> or
>>>>>>>>>>>> e-
>>>>>>>>>>>> up
>>>>>>>>>>>> date-erates.err):
>>>>>>>>>>>>
>>>>>>>>>>>> AS_OVS_openBinaryOverlapFile()-- Failed to open
>>>>>>>>>>>> '/projects/nn9201k/Celera/work2/salaris1/salaris.ovlStore/0001~'
>>>>>>>>>>>> for
>>>>>>>>>>>> reading: No such file or directory
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------
>>>>>>>>>>>> Failure message:
>>>>>>>>>>>>
>>>>>>>>>>>> failed to apply the overlap corrections
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So it can obviously not find the file /salaris.ovlStore/0001~. The
>>>>>>>>>>>> reason
>>>>>>>>>>>> is, from what I can see, that the /salaris.ovlStore/0001~ file has
>>>>>>>>>>>> already
>>>>>>>>>>>> been updated to /salaris.ovlStore/0001 before it stopped. In fact
>>>>>>>>>>>> it
>>>>>>>>>>>> seems
>>>>>>>>>>>> to have stopped after updating /salaris.ovlStore/0249 (of 430). Is
>>>>>>>>>>>> there
>>>>>>>>>>>> a
>>>>>>>>>>>> way to tell runCA to continue from  /salaris.ovlStore/0250~,
>>>>>>>>>>>> instead
>>>>>>>>>>>> of
>>>>>>>>>>>> from
>>>>>>>>>>>> 0001~, which is obviously not there any more??
>>>>>>>>>>>> Another solution I was thinking of is to run the previous
>>>>>>>>>>>> overlapStore
>>>>>>>>>>>> command again manually (the one that was done before starting the
>>>>>>>>>>>> frgcorr
>>>>>>>>>>>> and ovlcorr:
>>>>>>>>>>>> /xanadu/home/chrishah/programmes/wgs-7.0/Linux-amd64/bin/overlapSto
>>>>>>>>>>>> re
>>>>>>>>>>>> -c
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/salaris.ovlStore.BUILDING
>>>>>>>>>>>> -g
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/salaris.gkpStore  -i 0 -M
>>>>>>>>>>>> 14000
>>>>>>>>>>>> -L
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/salaris.ovlStore.list>
>>>>>>>>>>>> /projects/nn9201k/Celera/work2/salaris1/salaris.ovlStore.err 2>&1)
>>>>>>>>>>>> to
>>>>>>>>>>>> restore the status from before the frgcorr and ovlcorr steps,
>>>>>>>>>>>> before
>>>>>>>>>>>> resuming runCA. This should restore the 0001~ file, right? The most
>>>>>>>>>>>> important thing is that I want to avoid rerunning the frgcorr and
>>>>>>>>>>>> ovlcorr
>>>>>>>>>>>> steps, because these steps were really resource intensive.
>>>>>>>>>>>>
>>>>>>>>>>>> I would really appreciate any comments or suggestions to my
>>>>>>>>>>>> problem!
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> in advance for your help!
>>>>>>>>>>>>
>>>>>>>>>>>> much obliged,
>>>>>>>>>>>> Christoph
>>>>>>>>>>>>
>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -------------------------------------------------------------------
>>>>>>>>>>>> ---
>>>>>>>>>>>> --
>>>>>>>>>>>> --
>>>>>>>>>>>> --
>>>>>>>>>>>> --
>>>>>>>>>>>> Live Security Virtual Conference
>>>>>>>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>>>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>>>>>>>> Discussions
>>>>>>>>>>>> will include endpoint security, mobile security and the latest in
>>>>>>>>>>>> malware
>>>>>>>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> wgs-assembler-users mailing list
>>>>>>>>>>>> wgs...@li...
>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
>>>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> ---
>>>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>> Live Security Virtual Conference
>>>>>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>>>>>> Discussions
>>>>>>>>>> will include endpoint security, mobile security and the latest in
>>>>>>>>>> malware
>>>>>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>>>>>> _______________________________________________
>>>>>>>>>> wgs-assembler-users mailing list
>>>>>>>>>> wgs...@li...
>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
>>>>
>>
>>
>