You can subscribe to this list here.
2009 |
Jan
|
Feb
(26) |
Mar
(37) |
Apr
(20) |
May
(20) |
Jun
(81) |
Jul
(65) |
Aug
(77) |
Sep
(85) |
Oct
(86) |
Nov
(233) |
Dec
(131) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2010 |
Jan
(201) |
Feb
(274) |
Mar
(257) |
Apr
(208) |
May
(99) |
Jun
(223) |
Jul
(131) |
Aug
(139) |
Sep
(195) |
Oct
(133) |
Nov
(196) |
Dec
(184) |
2011 |
Jan
(156) |
Feb
(181) |
Mar
(246) |
Apr
(131) |
May
(172) |
Jun
(157) |
Jul
(121) |
Aug
(200) |
Sep
(199) |
Oct
(114) |
Nov
(189) |
Dec
(68) |
2012 |
Jan
(130) |
Feb
(180) |
Mar
(232) |
Apr
(173) |
May
(149) |
Jun
(105) |
Jul
(107) |
Aug
(69) |
Sep
(59) |
Oct
(49) |
Nov
(43) |
Dec
(56) |
2013 |
Jan
(69) |
Feb
(58) |
Mar
(79) |
Apr
(90) |
May
(80) |
Jun
(62) |
Jul
(76) |
Aug
(63) |
Sep
(49) |
Oct
(66) |
Nov
(66) |
Dec
(32) |
2014 |
Jan
(57) |
Feb
(29) |
Mar
(84) |
Apr
(28) |
May
(73) |
Jun
(57) |
Jul
(65) |
Aug
(91) |
Sep
(118) |
Oct
(133) |
Nov
(99) |
Dec
(69) |
2015 |
Jan
(39) |
Feb
(52) |
Mar
(35) |
Apr
(19) |
May
(56) |
Jun
(29) |
Jul
(37) |
Aug
(44) |
Sep
(21) |
Oct
(17) |
Nov
(43) |
Dec
(35) |
2016 |
Jan
(24) |
Feb
(35) |
Mar
(32) |
Apr
(27) |
May
(33) |
Jun
(26) |
Jul
(20) |
Aug
(35) |
Sep
(10) |
Oct
(26) |
Nov
(22) |
Dec
(12) |
2017 |
Jan
(31) |
Feb
(20) |
Mar
(24) |
Apr
(19) |
May
(17) |
Jun
(8) |
Jul
(9) |
Aug
(15) |
Sep
(8) |
Oct
(4) |
Nov
(21) |
Dec
(8) |
2018 |
Jan
(6) |
Feb
(14) |
Mar
(10) |
Apr
(14) |
May
(10) |
Jun
(5) |
Jul
(8) |
Aug
(12) |
Sep
(7) |
Oct
(8) |
Nov
(19) |
Dec
(2) |
2019 |
Jan
(2) |
Feb
(8) |
Mar
(7) |
Apr
|
May
(5) |
Jun
(8) |
Jul
(2) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
2020 |
Jan
|
Feb
(2) |
Mar
(2) |
Apr
(7) |
May
|
Jun
(14) |
Jul
(7) |
Aug
(2) |
Sep
(14) |
Oct
(4) |
Nov
(1) |
Dec
|
2021 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
(3) |
Jul
(3) |
Aug
(3) |
Sep
(8) |
Oct
(1) |
Nov
(4) |
Dec
(1) |
2022 |
Jan
(11) |
Feb
(1) |
Mar
|
Apr
(3) |
May
|
Jun
|
Jul
|
Aug
(5) |
Sep
(5) |
Oct
(2) |
Nov
(3) |
Dec
|
2023 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(3) |
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
(1) |
2024 |
Jan
(2) |
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
From: Nowoshilow,Sergej <ser...@im...> - 2020-06-13 22:06:13
|
Hi Keiran, I don’t think so. As far as I know, samtools view is only used to convert (or view) SAM<->BAM<->CRAM. However, the file must not be sorted, while indexing implies that the file is sorted. I’d use ‘samtools view’ to convert the file if necessary, sort it using ‘samtools sort’ and then ‘samtools index -c’ to index the resulting file. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010<https://orcid.org/0000-0001-8360-5010> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:16 An: "sam...@li..." <sam...@li...> Betreff: [Samtools-help] --write-index - csi? Hi, Is it possible to write a csi index when using samtools view? I can't see anything in the --output-fmt* section relating to index format. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-13 21:52:01
|
Hi, Yes, I can confirm it does csi by default, bad assumptions... In the end I didn't need it as to avoid read write and decompression more tee/pee fun was needed: samtools markdup -u... | \ pee "bam_stats ...> out.bam.bas "\ "samtools view... | pee 'cat > out.bam' 'samtools index - out.bam.bai' \ 'md5sum -b > out.bam.md5'\ " This allows 2 items to digest the uncompressed data, while another 2 digest the compressed stream, outputting a total of 4 different files. Although this is a bit more complex to read, it's much easier to swap in bam+bai/csi or cram+crai by placeholders, while retaining the efficiency. Regards, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. From: "Nowoshilow,Sergej" <ser...@im...> Date: Saturday, 13 June 2020 at 20:31 To: Keiran Raine <kr...@sa...>, "sam...@li..." <sam...@li...> Subject: Re: [Samtools-help] --write-index - csi? [EXT] Hi Keiran, I see.. Sorry, I wasn’t aware of that option as I still use 1.9. However, this is what I found in the source code on Github. If @p fn contains the string "##idx##" (HTS_IDX_DELIM), the part before the delimiter will be used as the name of the data file and the part after it will be used as the name of the index. https://github.com/samtools/htslib/blob/34c15969fa49d78b07c649374fb008fb3ed1e98a/htslib/hts.h#L816 The following two lines suggests that it creates CSI by default https://github.com/samtools/samtools/blob/72d140b590cbacc975e96bf40f3db6e6370a5cbe/sam_utils.c#L102 [github.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_samtools_samtools_blob_72d140b590cbacc975e96bf40f3db6e6370a5cbe_sam-5Futils.c-23L102&d=DwMGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=Fx84Cy_lzbMQXG5n1vjTOWr1jHvlJFSu2d4HFxwTwKg&s=O3am9FNEC2B4EwIwfZQQtyyV-jginOkCWHJqbtkoVvI&e=> https://github.com/samtools/samtools/blob/72d140b590cbacc975e96bf40f3db6e6370a5cbe/sam_utils.c#L75 I cannot try it out since I don’t have samtools-1.10 right now, but maybe it helps. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010 [orcid.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__orcid.org_0000-2D0001-2D8360-2D5010&d=DwMGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=Fx84Cy_lzbMQXG5n1vjTOWr1jHvlJFSu2d4HFxwTwKg&s=fDRWss51AM3b7YqlwKdw-idlnb4Gu0h56u9rDPg5bF0&e=> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:33 An: "Nowoshilow,Sergej" <ser...@im...>, "sam...@li..." <sam...@li...> Betreff: Re: [Samtools-help] --write-index - csi? [EXT] Hi Sergej, "--write-index" is a valid option for samtools view in 1.10 (provided your data is sorted). My question, more clearly, would be "can you specify csi index for bam output instead of bai"? Regards, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. From: "Nowoshilow,Sergej" <ser...@im...> Date: Saturday, 13 June 2020 at 19:30 To: Keiran Raine <kr...@sa...>, "sam...@li..." <sam...@li...> Subject: Re: [Samtools-help] --write-index - csi? [EXT] Hi Keiran, I don’t think so. As far as I know, samtools view is only used to convert (or view) SAM<->BAM<->CRAM. However, the file must not be sorted, while indexing implies that the file is sorted. I’d use ‘samtools view’ to convert the file if necessary, sort it using ‘samtools sort’ and then ‘samtools index -c’ to index the resulting file. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010 [orcid.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__orcid.org_0000-2D0001-2D8360-2D5010&d=DwMGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=OUcnRnlcd1-X4t14yOV3uXvJMtfY4-xOCh6FrcpRBec&s=UPRZn_OWn3OGDmbO7f4nFs-1GyD4WTRHQnh5eKylxyI&e=> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:16 An: "sam...@li..." <sam...@li...> Betreff: [Samtools-help] --write-index - csi? Hi, Is it possible to write a csi index when using samtools view? I can't see anything in the --output-fmt* section relating to index format. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Nowoshilow,Sergej <ser...@im...> - 2020-06-13 20:04:08
|
Hi Keiran, I see.. Sorry, I wasn’t aware of that option as I still use 1.9. However, this is what I found in the source code on Github. If @p fn contains the string "##idx##" (HTS_IDX_DELIM), the part before the delimiter will be used as the name of the data file and the part after it will be used as the name of the index. https://github.com/samtools/htslib/blob/34c15969fa49d78b07c649374fb008fb3ed1e98a/htslib/hts.h#L816 The following two lines suggests that it creates CSI by default https://github.com/samtools/samtools/blob/72d140b590cbacc975e96bf40f3db6e6370a5cbe/sam_utils.c#L102 https://github.com/samtools/samtools/blob/72d140b590cbacc975e96bf40f3db6e6370a5cbe/sam_utils.c#L75 I cannot try it out since I don’t have samtools-1.10 right now, but maybe it helps. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010<https://orcid.org/0000-0001-8360-5010> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:33 An: "Nowoshilow,Sergej" <ser...@im...>, "sam...@li..." <sam...@li...> Betreff: Re: [Samtools-help] --write-index - csi? [EXT] Hi Sergej, "--write-index" is a valid option for samtools view in 1.10 (provided your data is sorted). My question, more clearly, would be "can you specify csi index for bam output instead of bai"? Regards, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. From: "Nowoshilow,Sergej" <ser...@im...> Date: Saturday, 13 June 2020 at 19:30 To: Keiran Raine <kr...@sa...>, "sam...@li..." <sam...@li...> Subject: Re: [Samtools-help] --write-index - csi? [EXT] Hi Keiran, I don’t think so. As far as I know, samtools view is only used to convert (or view) SAM<->BAM<->CRAM. However, the file must not be sorted, while indexing implies that the file is sorted. I’d use ‘samtools view’ to convert the file if necessary, sort it using ‘samtools sort’ and then ‘samtools index -c’ to index the resulting file. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010 [orcid.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__orcid.org_0000-2D0001-2D8360-2D5010&d=DwMGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=OUcnRnlcd1-X4t14yOV3uXvJMtfY4-xOCh6FrcpRBec&s=UPRZn_OWn3OGDmbO7f4nFs-1GyD4WTRHQnh5eKylxyI&e=> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:16 An: "sam...@li..." <sam...@li...> Betreff: [Samtools-help] --write-index - csi? Hi, Is it possible to write a csi index when using samtools view? I can't see anything in the --output-fmt* section relating to index format. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-13 18:33:28
|
Hi Sergej, "--write-index" is a valid option for samtools view in 1.10 (provided your data is sorted). My question, more clearly, would be "can you specify csi index for bam output instead of bai"? Regards, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. From: "Nowoshilow,Sergej" <ser...@im...> Date: Saturday, 13 June 2020 at 19:30 To: Keiran Raine <kr...@sa...>, "sam...@li..." <sam...@li...> Subject: Re: [Samtools-help] --write-index - csi? [EXT] Hi Keiran, I don’t think so. As far as I know, samtools view is only used to convert (or view) SAM<->BAM<->CRAM. However, the file must not be sorted, while indexing implies that the file is sorted. I’d use ‘samtools view’ to convert the file if necessary, sort it using ‘samtools sort’ and then ‘samtools index -c’ to index the resulting file. Best, Sergej Dr. Sergej Nowoshilow Post-doc in Tanaka Lab Elly Tanaka group, IMP Animal models of regeneration email: Ser...@im... phone: +43 (0) 1 79730 3203 orcid: 0000-0001-8360-5010 [orcid.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__orcid.org_0000-2D0001-2D8360-2D5010&d=DwMGaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=OUcnRnlcd1-X4t14yOV3uXvJMtfY4-xOCh6FrcpRBec&s=UPRZn_OWn3OGDmbO7f4nFs-1GyD4WTRHQnh5eKylxyI&e=> IMP Research Institute of Molecular Pathology Campus-Vienna-BioCenter 1 1030 Vienna, Austria Part of Vienna BioCenter www.viennabiocenter.org This message is confidential and may contain privileges information. It is intended for the named recipients only. If you receive it in error please notify me and permanently delete the original message and any copies. Von: Keiran Raine <kr...@sa...> Datum: Samstag, 13. Juni 2020 um 20:16 An: "sam...@li..." <sam...@li...> Betreff: [Samtools-help] --write-index - csi? Hi, Is it possible to write a csi index when using samtools view? I can't see anything in the --output-fmt* section relating to index format. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-13 18:15:34
|
Hi, Is it possible to write a csi index when using samtools view? I can't see anything in the --output-fmt* section relating to index format. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-12 13:42:56
|
Hi Andrew, Great, yes thank you, I can safely ignore this for my workflow. Thanks again, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. From: aw7 <aw...@sa...> Date: Friday, 12 June 2020 at 14:40 To: Keiran Raine <kr...@sa...> Cc: "sam...@li..." <sam...@li...> Subject: Re: [Samtools-help] markdup "-c" option [EXT] Hi Keiran, The -c option clears the duplicate flag and removes any 'dt' or 'do' tags on a read prior to duplicate marking. I added it by request so that a mix of duplicate marking programs did not give inconsistent results. There is no performance advantage, markdup does not take into account any previous information. Is this what you wanted to know? Andrew On 2020-06-12 13:29, Keiran Raine wrote: Hi, Can someone clarify the behaviour of "Clear previous duplicate settings and tags" please? If data has been duplicate marked at the "lane" level and then merged and passed into markdup again, should this be in use? Does it give performance advantage by reusing previous information? Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ Samtools-help mailing list Sam...@li...<mailto:Sam...@li...> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_samtools-2Dhelp&d=DwICAg&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=iKRz7N6xL3uJ6tuuPlViiQ&m=M1uS2vu4yCY2n1GopIIHRS3btcMUzjwvBQAfID-qilM&s=RhOZbZVZJCVybTLaD7RHPvlJxrjvqKiIMemJ_C0gDns&e= -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: aw7 <aw...@sa...> - 2020-06-12 13:40:27
|
Hi Keiran, The -c option clears the duplicate flag and removes any 'dt' or 'do' tags on a read prior to duplicate marking. I added it by request so that a mix of duplicate marking programs did not give inconsistent results. There is no performance advantage, markdup does not take into account any previous information. Is this what you wanted to know? Andrew On 2020-06-12 13:29, Keiran Raine wrote: > Hi, > > Can someone clarify the behaviour of "Clear previous duplicate settings and tags" please? > > If data has been duplicate marked at the "lane" level and then merged and passed into markdup again, should this be in use? Does it give performance advantage by reusing previous information? > > Thanks, > > Keiran Raine, M.Sc., MBCS > > Principal Software Developer > > Cancer, Ageing and Somatic Mutation > > Wellcome Sanger Institute > > kr...@sa... > > Office: D3-19 > > Ext: 4983 > > Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. > _______________________________________________ > Samtools-help mailing list > Sam...@li... > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_samtools-2Dhelp&d=DwICAg&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=iKRz7N6xL3uJ6tuuPlViiQ&m=M1uS2vu4yCY2n1GopIIHRS3btcMUzjwvBQAfID-qilM&s=RhOZbZVZJCVybTLaD7RHPvlJxrjvqKiIMemJ_C0gDns&e= -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-12 12:29:20
|
Hi, Can someone clarify the behaviour of "Clear previous duplicate settings and tags" please? If data has been duplicate marked at the "lane" level and then merged and passed into markdup again, should this be in use? Does it give performance advantage by reusing previous information? Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Keiran R. <kr...@sa...> - 2020-06-08 12:37:47
|
Thanks for the quick response John, I wasn't clear if it was used when using "-O"? It's not completely obvious: <prefix> is required unless the -o or -O options are used. Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa... Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. On 08/06/2020, 13:31, "John Marshall" <Joh...@gl...> wrote: On 8 Jun 2020, at 13:16, Keiran Raine <kr...@sa...> wrote: > > Just a quick query. Where does samtools write the temporary files for the collate step? Can I assume /tmp or the location defined by $TMPDIR? At the moment if you want them to appear somewhere in particular other than /tmp (which is probably your smallest temp space!), you get to specify it each time using collate's optional <prefix> command line argument. See <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_samtools_samtools_issues_1172&d=DwIFAg&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=0l44_2npVb7te0dQGjE0Xw&m=0LZ1KBMv5gMfrPdcCHLviqAXd3gXVGtI-sKIPpGOZEE&s=DUZQaK7Ufoq1fTAtjRDnRUA1zZUspiwBdE48rFyV5oA&e= >. John -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: John M. <Joh...@gl...> - 2020-06-08 12:31:54
|
On 8 Jun 2020, at 13:16, Keiran Raine <kr...@sa...> wrote: > > Just a quick query. Where does samtools write the temporary files for the collate step? Can I assume /tmp or the location defined by $TMPDIR? At the moment if you want them to appear somewhere in particular other than /tmp (which is probably your smallest temp space!), you get to specify it each time using collate's optional <prefix> command line argument. See <https://github.com/samtools/samtools/issues/1172>. John |
From: Keiran R. <kr...@sa...> - 2020-06-08 12:16:56
|
Hi, Just a quick query. Where does samtools write the temporary files for the collate step? Can I assume /tmp or the location defined by $TMPDIR? Thanks, Keiran Raine, M.Sc., MBCS Principal Software Developer Cancer, Ageing and Somatic Mutation Wellcome Sanger Institute kr...@sa...<mailto:kr...@sa...> Office: D3-19 Ext: 4983 Please note that due to childcare requirements during the Covid lockdown I am generally not available after 4pm. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: John M. <Joh...@gl...> - 2020-04-23 09:38:22
|
On 22 Apr 2020, at 17:10, Pedro Hollanda Carvalho <hol...@gm...<mailto:hol...@gm...>> wrote: I'm unable to perform samtools sort command. Result is a weird buch of symbols followed by letters and numbers as follows (just a part of it): yOR�����G(��< {`�m�wF�D/�q|lOS�}�H This weird bunch of symbols is your BAM file being printed to your terminal. The reason for this happening with samtools sort is that you are using a very outdated version of samtools that misunderstands the samtools options in the instructions you're following. This is also confirmed by the "SAM header is present" message from samtools view, which hasn't existed since before samtools 1.0 in 2014. You should install (or get your systems people to assist in installing) an up-to-date samtools, currently version 1.10. Once you've done that, the command lines you're using will work as expected. Also, the result of SAM is weird as it does not seem to fit the usual file format: $ head Kp02.sam @HD VN:1.0 SO:unsorted @SQ SN:OMLG01000001.1 LN:1427 @SQ SN:OMLG01000002.1 LN:12368 [snip] This is as expected for the headers in a SAM file. You'd have to head a few more lines to see the alignment records after all the headers. (It is a bit unusual to see VN:1.0 -- even the earliest deployed SAM files have VN:1.3 or so -- but it doesn't much matter and in this case is something that has been output by bowtie2.) The command used to perform samtools sort was samtools sort -O bam -o Kp03_sorted.bam When you've got the up-to-date samtools installed, you'll find that its sort command can read from SAM files too. So your samtools view -bS + samtools sort can be simplified to read the bowtie output SAM file directly: samtools sort -Obam -o Kp03_sorted.bam Kp03.sam John |
From: Pedro H. C. <hol...@gm...> - 2020-04-22 16:11:16
|
Hi I'm trying to perform snp calling of a fish species but itś being a hard task. My current problem is the following: After indexing a reference genome (bowtie2-build), align my sequences to this genome (bowtie2 -x) and converting SAM to BAM (samtools view -bS) I'm unable to perform samtools sort command. Result is a weird buch of symbols followed by letters and numbers as follows (just a part of it): yOR�����G(��< {`�m�wF�D/�q|lOS�}�HӁ 7P ����]��)G��x=LǗ�� #<\�v����[� ���\��n�O~'�mZ ��D3�pq�}�Ʀ����`�w���_�? ��K���%��1eEF[3Gy�)� �_w$�(��L+��� �� �� ��f�D���%iu4�H��Y�rfH���pe:�v W�/�2, '�I�� � <8o��� �su��Z������� {����r� �b ����)F}�N5 {4O��ւ���b���o������e W8�X9� �DO��� � d5�t(F C;4 E�p���XcαW^��(Nr ���g�������$�D�� � e��� � BC phcarvalho@mapinguary :/hiseq2019/katpel_refmap/t;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62;c62; I realized that my SAM files when inspected using 'head' command display the same symbols. This makes me wonder if the problem is not with the SAM file. Maybe the problem concern one of the previous steps that I have mentioned above. So, I briefly describe the results of these three previous steps below: 1.bowtie2-build $ sudo /opt/bowtie2/bowtie2-build /hiseq2019/katpel_refmap/thualb/GCA_900302625.1_ASM90030262v1_genomic.fna /hiseq2019/katpel_refmap/thualb/thualb (four pages of calculation) Total time for backward call to driver() for mirror index: 00:19:01 2.bowtie2 -x $ sudo /opt/bowtie2/bowtie2 -x /hiseq2019/katpel_refmap/thualb/thualb -U /hiseq2019/katpel_refmap/teste_trimming/Kp02.R1headcrop.fastq -S Kp02.sam --no-unal -p31 3006792 reads; of these: 3006792 (100.00%) were unpaired; of these: 1266671 (42.13%) aligned 0 times 1398982 (46.53%) aligned exactly 1 time 341139 (11.35%) aligned >1 times 57.87% overall alignment rate 3.samtools view $ samtools view -bS Kp02.sam > Kp02.bam [samopen] SAM header is present: 38995 sequences. Also, the result of SAM is weird as it does not seem to fit the usual file format: $ head Kp02.sam @HD VN:1.0 SO:unsorted @SQ SN:OMLG01000001.1 LN:1427 @SQ SN:OMLG01000002.1 LN:12368 @SQ SN:OMLG01000003.1 LN:56682 @SQ SN:OMLG01000004.1 LN:34652 @SQ SN:OMLG01000005.1 LN:30301 @SQ SN:OMLG01000006.1 LN:2594 @SQ SN:OMLG01000007.1 LN:14873 @SQ SN:OMLG01000008.1 LN:13482 @SQ SN:OMLG01000009.1 LN:10753 The command used to perform samtools sort was samtools sort -O bam -o Kp03_sorted.bam Could anyone bring me some light? I'm new to bioinformatics and have been struggling to learn it by myself. Thanks! Professor Adjunto ------------------ *Instituto de Biodiversidade e Sustentabilidade* *NUPEM - UFRJ* *Universidade Federal do Rio de Janeiro* Av. São José do Barreto, 764 São José do Barreto, Macaé, RJ CEP 27965-045 ------------------ «Be Yourself. Everyone Else is Already Taken.» |
From: Valeriu O. <vo...@sa...> - 2020-04-20 15:15:12
|
A solution would be to go through SAM, add the extra tag and convert back to BAM: samtools view -H old.bam > tmp.sam ; samtools view old.bam | awk '{printf "%s\tOQ:Z:%s\n", $0, $11}' >> tmp.sam && samtools view -bo new.bam tmp.sam From: Justin Jeyakani <gna...@gi...> Sent: 20 April 2020 14:09 To: sam...@li... Subject: [Samtools-help] read and write a bam: but duplicate the quality column (11) and write one with "OQ:Z:" flag [EXT] Hi All, I have over 1000 bam files and I need to read and write them as bam files. The output bam shd contain a header with the alignment, but the column 11 (quality) to be repeated once, the second should contain quality with "OQ:Z:" flag. The script/package/tools shd be robust. Appreciated providing the script or tools to do this! Thanks in advance. **eg:** **input bam::** E00579:50:HK2VJALXX:6:1220:15300:41040 2115 chr1 9999 0 90H60M chr5 18606598 0 GATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A SA:Z:chr5,18606834,-,51S99M,37,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:58 E00579:50:HK2VJALXX:6:1212:4066:24884 113 chr1 9999 0 34S60M56S chr5 18606897 0 CCTAGAACAGCTCTTCCTTTATTTTCTTTTTCTGGATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACAGAGATAACTATTGATACAACACCTTCATGACCCTAAGGTACTATCATAGAGTTCT<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA SA:Z:chr5,18606769,+,58M92S,0,0;chr5,18606834,+,107S43M,0,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:59 **output bam::** E00579:50:HK2VJALXX:6:1220:15300:41040 2115 chr1 9999 0 90H60M chr5 18606598 0 GATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A SA:Z:chr5,18606834,-,51S99M,37,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:58 **OQ:Z:JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A** E00579:50:HK2VJALXX:6:1212:4066:24884 113 chr1 9999 0 34S60M56S chr5 18606897 0 CCTAGAACAGCTCTTCCTTTATTTTCTTTTTCTGGATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACAGAGATAACTATTGATACAACACCTTCATGACCCTAAGGTACTATCATAGAGTTCT<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA SA:Z:chr5,18606769,+,58M92S,0,0;chr5,18606834,+,107S43M,0,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:59 **OQ:Z:T<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA** This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Robert D. <rm...@sa...> - 2020-04-20 14:45:19
|
On Mon, 20 Apr 2020, Pedro Hollanda Carvalho wrote: > Hi, could anyone help me with this? > > I'm trying to convert SAM to BAM using samtools view and I have the > following error message > > $ sudo samtools view -bS Kp01.sam > Kp01.bam > -bash: Kp01.bam: Permission denied Judging from the error message, you either don't have permission to create Kp01.bam in the current directory, or one already exists that you can't overwrite. Note that the '> Kp01.bam' part of the command above is being run as yourself, and not as root (via 'sudo'). Also, *** you should not be running samtools as root via sudo ***. It is not designed to be used that way. Instead, you should use sudo (if necessary) to create a directory where your normal account has write permissions, and then use that to store the results from samtools. For example (assuming directory 'output' does not exist: sudo mkdir output sudo chown `whoami`.`groups | awk '{ print $1 }'` output samtools view -bS Kp01.sam > output/Kp01.bam > I have already tried from different locations and used sudo command but > always got the same error You probably need to check permissions of the various files and directories that you have made. If you have been using 'sudo' you may now have a mixture of ownership, which is likely to be painful. In general, you shouldn't use 'sudo' unless you really have to. Rob Davies rm...@sa... The Sanger Institute http://www.sanger.ac.uk/ Hinxton, Cambs., Tel. +44 (1223) 834244 CB10 1SA, U.K. Fax. +44 (1223) 494919 -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Pedro H. C. <hol...@gm...> - 2020-04-20 13:57:54
|
Hi, could anyone help me with this? I'm trying to convert SAM to BAM using samtools view and I have the following error message $ sudo samtools view -bS Kp01.sam > Kp01.bam -bash: Kp01.bam: Permission denied I have already tried from different locations and used sudo command but always got the same error Thanks Dr. Pedro Hollanda Carvalho Professor Adjunto ------------------ *Instituto de Biodiversidade e Sustentabilidade* *NUPEM - UFRJ* *Universidade Federal do Rio de Janeiro* Av. São José do Barreto, 764 São José do Barreto, Macaé, RJ CEP 27965-045 ------------------ «Be Yourself. Everyone Else is Already Taken.» |
From: Justin J. <gna...@gi...> - 2020-04-20 13:41:37
|
Hi All, I have over 1000 bam files and I need to read and write them as bam files. The output bam shd contain a header with the alignment, but the column 11 (quality) to be repeated once, the second should contain quality with "OQ:Z:" flag. The script/package/tools shd be robust. Appreciated providing the script or tools to do this! Thanks in advance. **eg:** **input bam::** E00579:50:HK2VJALXX:6:1220:15300:41040 2115 chr1 9999 0 90H60M chr5 18606598 0 GATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A SA:Z:chr5,18606834,-,51S99M,37,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:58 E00579:50:HK2VJALXX:6:1212:4066:24884 113 chr1 9999 0 34S60M56S chr5 18606897 0 CCTAGAACAGCTCTTCCTTTATTTTCTTTTTCTGGATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACAGAGATAACTATTGATACAACACCTTCATGACCCTAAGGTACTATCATAGAGTTCT<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA SA:Z:chr5,18606769,+,58M92S,0,0;chr5,18606834,+,107S43M,0,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:59 **output bam::** E00579:50:HK2VJALXX:6:1220:15300:41040 2115 chr1 9999 0 90H60M chr5 18606598 0 GATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A SA:Z:chr5,18606834,-,51S99M,37,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:58 **OQ:Z:JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJFJJJJFJJJJJFFJJJJJJJJJFJ<A** E00579:50:HK2VJALXX:6:1212:4066:24884 113 chr1 9999 0 34S60M56S chr5 18606897 0 CCTAGAACAGCTCTTCCTTTATTTTCTTTTTCTGGATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACAGAGATAACTATTGATACAACACCTTCATGACCCTAAGGTACTATCATAGAGTTCT<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA SA:Z:chr5,18606769,+,58M92S,0,0;chr5,18606834,+,107S43M,0,0; MD:Z:60 PG:Z:MarkDuplicates RG:Z:HK2VJALXX.6 NM:i:0 AS:i:60 XS:i:59 **OQ:Z:T<<-FA-7<AFF<<<7<FJAF-JFJFF-J<JJAAJFFF-AAJJJJFJJJJJAJJJJJJFAJJJJJJJJFJJJJJJJJJJF<FJJJFJJJJFJJFJFJJJJJJJFJJJJJJJJJJFJJJJFJJJJJJJJFJJJJJJJJJJJJJJJJJFFFAA** This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person. |
From: Justin J. <gna...@gi...> - 2020-04-12 01:30:59
|
Hi all, I need to append the base quality of fastq file into my existing bam file which does not contain OQ. Other than making bam to sam and flattening the fastq as one liner to get the matching quality for the aligned reads, is there any parameters in samtools or any other tools readily available to do this? The bam is generated by parabricks germline pipeline. Ultimately I need to convert the bam to cram with OQ. Thanks in advance. This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person. |
From: Andrew W. <aw...@sa...> - 2020-03-04 10:17:21
|
Hi Jon, > This resource is very helpful: > https://www.htslib.org/algorithms/duplicate.html, One day I will write a full description of how duplicate marking works. I keep telling myself that. > but does samtools > markdup handle interchromosomal read-pairs like picard? It does. > What are the differences in the algorithms? samtools markdup run using mode s should give identical results to picard for duplicate marking. There are two modes and I will just quote the man page here: "Duplicate decision method for paired reads. Values are 't' or 's'. Mode t measures positions based on template start/end (default). Mode 's' measures positions based on sequence start. While the two methods identify mostly the same reads as duplicates, mode 's' tends to return more results. Unpaired reads are treated identically by both modes." The differences only appear when both reads are are mapped to the same direction (FF or RR). There are differences in optical duplicate marking for HiSeq X data but that is down to an integer overflow bug in picard. Regards, Andrew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Jon B. <jo...@ar...> - 2020-03-03 21:31:21
|
Hi, I'm interested in the difference between samtools markdup vs picard markduplicates. Most of the information I find is comparing Samtools rmdup to picard markduplicates. This resource is very helpful: https://www.htslib.org/algorithms/duplicate.html, but does samtools markdup handle interchromosomal read-pairs like picard? What are the differences in the algorithms? Jon -- Jon Belton, PhD Scientist II Arima Genomics, Inc. jo...@ar... |
From: James B. <jk...@sa...> - 2020-02-27 09:39:19
|
On Thu, Feb 27, 2020 at 04:36:39AM +0000, Richard Wilton wrote: > That's what the SAM spec says in its description of the @SQ header record. But given the wide variety of sort orders that can be represented in SAM format, this can't be right. For SAM, every record contains a copy of the SQ SN string. However it is indeed sorting by the order of SQ entries rather than the actual RNAME string itself. BAM does not store the SN string verbatim. Instead it has a binary refID ("tid" in samtools world) which is an integer referencing the Nth SQ line. It's a bit more logical as sorting by this refID integer is the same as sorting by SQ line order, so there is no confusion. Hence the sort order remains the same for both SAM and BAM. > In fact, the set of @SQ records functions as a lookup table, keyed by SN and referenced by RNAME in each alignment record. Is that what the spec is trying to say? I see the distinction - sorting by the key itself rather than sorting by the ordering of SQ lines. However I think the spec is saying exactly what you think it's trying to say, so if you find something ambiguous please quote the exact bit that is confusing. It currently states: "For coordinate sort, the major sort key is the {\sf RNAME} field, with order defined by the order of {\tt @SQ} lines in the header." This is correct. The key which is sorted on is RNAME and the collation order is the Nth line rather than the string itself. I'm not sure what you mean by "wide variety of sort orders". It's true users can define their own sort orderings and we have a sub-sort field too, but in those cases the above sentence doesn't apply (note it is prefaced with "For coordinate sort"). Thanks, James PS. Note while the spec says one thing, it is not necessarily what happens in the wild due to accidents and misunderstandings. I've seen BAMs where they have been split apart by chromosome with BAM filenames named after RNAME, then some form of parallel processing takes place, and finally a "samtools cat in.*.bam -o out.bam" is ran which ends up placing in lexicographic RNAME order rather than Nth SQ line order, despite the @HD header claiming to be coordinate sorted. Obviously this is incorrect, although many tools will still work. -- James Bonfield (jk...@sa...) The Sanger Institute, Hinxton, Cambs, CB10 1SA -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Richard W. <ric...@jh...> - 2020-02-27 04:52:07
|
That's what the SAM spec says in its description of the @SQ header record. But given the wide variety of sort orders that can be represented in SAM format, this can't be right. In fact, the set of @SQ records functions as a lookup table, keyed by SN and referenced by RNAME in each alignment record. Is that what the spec is trying to say? Thank you! |
From: Robert D. <rm...@sa...> - 2019-12-06 17:23:39
|
Samtools (and HTSlib and BCFtools) version 1.10 is now available from GitHub and SourceForge https://sourceforge.net/projects/samtools/ https://github.com/samtools/htslib/releases/tag/1.10 https://github.com/samtools/samtools/releases/tag/1.10 https://github.com/samtools/bcftools/releases/tag/1.10 The main changes are listed below: ------------------------------------------------------------------------------ htslib - changes v1.10 ------------------------------------------------------------------------------ Brief summary ------------- There are many changes in this release, so the executive summary is: * Addition of support for references longer than 2Gb (NB: SAM and VCF formats only, not their binary counterparts). This may need changes in code using HTSlib. See README.large_positions.md for more information. * Added a SAM header API. * Major speed up to SAM reading and writing. This also now supports multi-threading. * We can now auto-index on-the-fly while writing a file. This also includes to bgzipped SAM.gz. * Overhaul of the S3 interface, which now supports version 4 signatures. This also makes writing to S3 work. These also required some ABI changes. See below for full details. Features / updates ------------------ * A new SAM/BAM/CRAM header API has been added to HTSlib, allowing header data to be updated without having to parse or rewrite large parts of the header text. See htslib/sam.h for function definitions and documentation. (#812) The header typedef and several pre-existing functions have been renamed to have a sam_hdr_ prefix: sam_hdr_t, sam_hdr_init(), sam_hdr_destroy(), and sam_hdr_dup(). (The existing bam_hdr_-prefixed names are still provided for compatibility with existing code.) (#887, thanks to John Marshall) * Changes to hfile_s3, which provides support for the AWS S3 API. (#839) - hfile_s3 now uses version 4 signatures by default. Attempting to write to an S3 bucket will also now work correctly. It is possible to force version 2 signatures by creating environment variable HTS_S3_V2 (the exact value does not matter, it just has to exist). Note that writing depends on features that need version 4 signatures, so forcing version 2 will disable writes. - hfile_s3 will automatically retry requests where the region endpoint was not specified correctly, either by following the 301 redirect (when using path-style requests) or reading the 400 response (when using virtual-hosted style requests and version 4 signatures). The first region to try can be set by using the AWS_DEFAULT_REGION environment variable, by setting "region" in ".aws/credentials" or by setting "bucket_location" in ".s3cfg". - hfile_s3 now percent-escapes the path component of s3:// URLs. For backwards-compatibility it will ignore any paths that have already been escaped (detected by looking for '%' followed by two hexadecimal digits.) - New environment variables HTS_S3_V2, HTS_S3_HOST, HTS_S3_S3CFG and HTS_S3_PART_SIZE to force version-2 signatures, control the S3 server hostname, the configuration file and upload chunk sizes respectively. * Numerous SAM format improvements. - Bgzipped SAM files can now be indexed and queried. The library now recognises sam.gz as a format name to ease this usage. (#718, #916) - The SAM reader and writer now supports multi-threading via the thread-pool. (#916) Note that the multi-threaded SAM reader does not currently support seek operations. Trying to do this (for example with an iterator range request) will result in the SAM readers dropping back to single-threaded mode. - Major speed up of SAM decoding and encoding, by around 2x. (#722) - SAM format can now handle 64-bit coordinates and references. This has implications for the ABI too (see below). Note BAM and CRAM currently cannot handle references longer than 2Gb, however given the speed and threading improvements SAM.gz is a viable workaround. (#709) * We can now automatically build indices on-the-fly while writing SAM, BAM, CRAM, VCF and BCF files. (Note for SAM and VCF this only works when bgzipped.) (#718) * HTSlib now supports the @SQ-AN header field, which lists alternative names for reference sequences. This means given "@SQ SN:1 AN:chr1", tools like samtools can accept requests for "1" or "chr1" equivalently. (#931) * Zero-length files are no longer considered to be valid SAM files (with no header and no alignments). This has been changed so that pipelines such as `somecmd | samtools ...` with `somecmd` aborting before outputting anything will now propagate the error to the second command. (#721, thanks to John Marshall; #261 reported by Adrian Tan) * Added support for use of non-standard index names by pasting the data filename and index filename with ##idx##. For example "/path1/my_data.bam##idx##/path2/my_index.csi" will open bam file "/path1/my_data.bam" and index file "/path2/my_index.csi". (#884) This affects hts_idx_load() and hts_open() functions. * Improved the region parsing code to handle colons in reference names. Strings can be disambiguated by the use of braces, so for example when reference sequences called "chr1" and "chr1:100-200" are both present, the regions "{chr1}:100-200" and "{chr1:100-200}" unambiguously indicate which reference is being used. (#708) A new function hts_parse_region() has been added along with specialisations for sam_parse_region() and fai_parse_region(). * CRAM encoding now has additional checks for MD/NM validity. If they are incorrect, it stores the (incorrect copy) verbatim so round-trips "work". (#792) * Sped up decoding of CRAM by around 10% when the MD tag is being generated. (#874) * CRAM REF_PATH now supports %Ns (where N is a single digit) expansion in http URLs, similar to how it already supported this for directories. (#791) * BGZF now permits indexing and seeking using virtual offsets in completely uncompressed streams. (#904, thanks to Adam Novak) * bgzip now asks for extra confirmation before decompressing files that don't have a known compression extension (e.g. .gz). This avoids `bgzip -d foo.bam.bai` producing a foo.bam file that is very much not a BAM-formatted file. (#927, thanks to John Marshall) * The htsfile utility can now copy files (including to/from URLs using HTSlib's remote access facilities) with the --copy option, in addition to its existing uses of identifying file formats and displaying sequence or variant data. (#756, thanks to John Marshall) * Added tabix --min-shift option. (#752, thanks to Garrett Stevens) * Tabix now has an -D option to disable storing a local copy of a remote index. (#870) * Improved support for MSYS Windows compiler environment. (#966) * External htslib plugins are now supported on Windows. (#966) API additions and improvements ------------------------------ * New API functions bam_set_mempolicy() and bam_get_mempolicy() have been added. These allow more control over the ownership of bam1_t alignment record data; see documentation in htslib/sam.h for more information. (#922) * Added more HTS_RESULT_USED checks, this time for VCF I/O. (#805) * khash can now hash kstrings. This makes it easier to hash non-NUL-terminated strings. (#713) * New haddextension() filename extension API function. (#788, thanks to John Marshall) * New hts_resize() macro, designed to replace uses of hts_expand() and hts_expand0(). (#805) * Added way of cleaning up unused jobs in the thread pool via the new hts_tpool_dispatch3() function. (#830) * New API functions hts_reglist_create() and sam_itr_regarray() are added to create hts_reglist_t region lists from `chr:<from>-<to>` type region specifiers. (#836) * Ksort has been improved to facilitate library use. See KSORT_INIT2 (adds scope / namespace capabilities) and KSORT_INIT_STATIC interfaces. (#851, thanks to John Marshall) * New kstring functions (#879): KS_INITIALIZE - Initializer for structure assignment ks_initialize() - Initializer for pointed-to kstrings ks_expand() - Increase kstring capacity by a given amount ks_clear() - Set kstring length to zero ks_free() - Free the underlying buffer ks_c_str() - Returns the kstring buffer as a const char *, or an empty string if the length is zero. * New API functions hts_idx_load3(), sam_index_load3(), tbx_index_load3() and bcf_index_load3() have been added. These allow control of whether remote indexes should be cached locally, and allow the error message printed when the index does not exist to be suppressed. (#870) * Improved hts_detect_format() so it no longer assumes all text is SAM unless positively identified otherwise. It also makes a stab at detecting bzip2 format and identifying BED, FASTA and FASTQ files. (#721, thanks to John Marshall; #200, #719 both reported by Torsten Seemann) * File format errors now set errno to EFTYPE (BSD, MacOS) when available instead of ENOEXEC. (#721) * New API function bam_set_qname (#942) * In addition to the existing hts_version() function, which reflects the HTSlib version being used at runtime, <htslib/hts.h> now also provides HTS_VERSION, a preprocessor macro reflecting the HTSlib version that a program is being compiled against. (#951, thanks to John Marshall; #794) ABI changes ----------- This release contains a number of things which change the ApplicationBinary Interface (ABI). This means code compiled against an earlierlibrary will require recompiling. The shared library soversion hasbeen bumped. * On systems that support it, the default symbol visibility has been changed to hidden and the only exported symbols are ones that form part of the officially supported ABI. This is to make clear exactly which symbols are considered parts of the library interface. It also helps packagers who want to check compatibility between HTSlib versions. (#946; see for example issues #311, #616, and #695) * HTSlib now supports 64 bit reference positions. This means several structures, function parameters, and return values have been made bigger to allow larger values to be stored. While most code that uses HTSlib interfaces should still build after this change, some alterations may be needed - notably to printf() formats where the values of structure members are being printed. (#709) Due to file format limitations, large positions are only supported when reading and writing SAM and VCF files. See README.large_positions.md for more information. * An extra field has been added to the kbitset_t struct so bitsets can be made smaller (and later enlarged) without involving memory allocation. (#710, thanks to John Marshall) * A new field has been added to the bam_pileup1_t structure to keep track of which CIGAR operator is being processed. This is used by a new bam_plp_insertion() function which can be used to return the sequence of any inserted bases at a given pileup location. If the alignment includes CIGAR P operators, the returned sequence will include pads. (#699) * The hts_itr_t and hts_itr_multi_t structures have been merged and can be used interchangeably. Extra fields have been added to hts_itr_t to support this. hts_itr_multi_t is now a typedef for hts_itr_t; sam_itr_multi_next() is now an alias for sam_itr_next() and hts_itr_multi_destroy() is an alias for hts_itr_destroy(). (#836) * An improved regidx interface has been added. To allow this, struct reg_t has been removed, regitr_t has been modified and various new API functions have been added to htslib/regidx.h. While parts of the old regidx API have been retained for backwards compatibility, it is recommended that all code using regidx should be changed to use the new interface. (#761) * Elements in the hts_reglist_t structure have been reordered slightly so that they pack together better. (#761) * bgzf_utell() and bgzf_useek() now use type off_t instead of long for the offset. This allows them to work correctly on files longer than 2G bytes on Windows and 32-bit Linux. (#868) * A number of functions that used to return void now return int so that they can report problems like memory allocation failures. Callers should take care to check the return values from these functions. (#834) The affected functions are: ksort.h: ks_introsort(), ks_mergesort() sam.h: bam_mplp_init_overlaps() synced_bcf_reader.h: bcf_sr_regions_flush() vcf.h: bcf_format_gt(), bcf_fmt_array(), bcf_enc_int1(), bcf_enc_size(), bcf_enc_vchar(), bcf_enc_vfloat(), bcf_enc_vint(), bcf_hdr_set_version(), bcf_hrec_format() vcfutils.h: bcf_remove_alleles() * bcf_set_variant_type() now outputs VCF_OVERLAP for spanning deletions (ALT=*). (#726) * A new field (hrecs) has been added to the bam_hdr_t structure for use by the new header API. The old sdict field is now not used and marked as deprecated. The l_text field has been changed from uint32_t to size_t, to allow for very large headers in SAM files. The text and l_text fields have been left for backwards compatibility, but should not be accessed directly in code that uses the new header API. To access the header text, the new functions sam_hdr_length() and sam_hdr_str() should be used instead. (#812) * The old cigar_tab field is now marked as deprecated; use the new bam_cigar_table[] instead. (#891, thanks to John Marshall) * The bam1_core_t structure's l_qname and l_extranul fields have been rearranged and enlarged; l_qname still includes the extra NULs. (Almost all code should use bam_get_qname(), bam_get_cigar(), etc, and has no need to use these fields directly.) HTSlib now supports the SAM specification's full 254 QNAME length again. (#900, thanks to John Marshall; #520) * bcf_index_load() no longer tries the '.tbi' suffix when looking for BCF index files (.tbi indexes are for text files, not binary BCF). (#870) * htsFile has a new 'state' member to support SAM multi-threading. (#916) * A new field has been added to the bam1_t structure, and others have been rearranged to remove structure holes. (#709; #922) Bug fixes --------- * Several BGZF format fixes: - Support for multi-member gzip files. (#744, thanks to Adam Novak; #742) - Fixed error handling code for native gzip formatted files. (64c4927) - CRCs checked when threading too (previously only when non-threaded). (#745) - Made bgzf_useek function work with threads. (#818) - Fixed rare threading deadlocks. (#831) - Reading of very short files (<28 bytes) that do not contain an EOF block. (#910) * Fixed some thread pool deadlocks caused by race conditions. (#746, #906) * Many additional memory allocation checks in VCF, BCF, SAM and CRAM code. This also changes the return type of some functions. See ABI changes above. (#920 amongst others) * Replace some sam parsing abort() calls with proper errors. (#721, thanks to John Marshall; #576) * Fixed to permit SAM read names of length 252 to 254 (the maximum specified by the SAM specification). (#900, thanks to John Marshall) * Fixed mpileup overlap detection heuristic to work with BAMs having long CIGARs (more than 65536 operations). (#802) * Security fix: CIGAR strings starting with the "N" operation can no longer cause underflow on the bam CIGAR structure. Similarly CIGAR strings that are entirely "D" ops could leak the contents of uninitialised variables. (#699) * Fixed bug where alignments starting 0M could cause an invalid memory access in sam_prob_realn(). (#699) * Fixed out of bounds memory access in mpileup when given a reference with binary characters (top-bit set). (#808, thanks to John Marshall) * Fixed crash in mpileup overlap_push() function. (#882; #852 reported by Pierre Lindenbaum) * Fixed various potential CRAM memory leaks when recovering from error cases. * Fixed CRAM index queries for unmapped reads (#911; samtools/samtools#958 reported by @acorvelo) * Fixed the combination of CRAM embedded references and multiple slices per container. This was incorrectly setting the header MD5sum. (No impact on default CRAM behaviour.) (b2552fd) * Removed unwanted explicit data flushing in CRAM writing, which on some OSes caused major slowdowns. (#883) * Fixed inefficiencies in CRAM encoding when many small references occur within the middle of large chromosomes. Previously it switched into multi-ref mode, but not back out of it which caused the read POS field to be stored poorly. (#896) * Fixed CRAM handling of references when the order of sequences in a supplied fasta file differs to the order of the @SQ headers. (#935) * Fixed BAM and CRAM multi-threaded decoding when used in conjunction with the multi-region iterator. (#830; #577, #822, #926 all reported by Brent Pedersen) * Removed some unaligned memory accesses in CRAM encoder and undefined behaviour in BCF reading (#867, thanks to David Seifert) * Repeated calling of bcf_empty() no longer crashes. (#741) * Fixed bug where some 8 or 16-bit negative integers were stored using values reserved by the BCF specification. These numbers are now promoted to the next size up, so -121 to -128 are stored using at least 16 bits, and -32761 to -32768 are stored using 32 bits. Note that while BCF files affected by this bug are technically incorrect, it is still possible to read them. When converting to VCF format, HTSlib (and therefore bcftools) will interpret the values as intended and write out the correct negative numbers. (#766, thanks to John Marshall; samtools/bcftools#874) * Allow repeated invocations of bcf_update_info() and bcf_update_format_*() functions. (#856, thanks to John Marshall; #813 reported by Steffen Möller) * Memory leak removed in knetfile's kftp_parse_url() function. (#759, thanks to David Alexander) * Fixed various crashes found by libfuzzer (invalid data leading to errors), mostly but not exclusively in CRAM, VCF and BCF decoding. (#805) * Improved robustness of BAI and CSI index creation and loading. (#870; #967) * Prevent (invalid) creation of TBI indices for BCF files. (#837; samtools/bcftools#707) * Better parsing of handling of remote URLs with ?param=val components and their interaction with remote index URLs. (#790; #784 reported by Mark Ebbert) * hts_idx_load() now checks locally for all possible index names before attempting to download a remote index. It also checks that the remote file it downloads is actually an index before trying to save and use it. (#870; samtools/samtools#1045 reported by Albert Vilella) * hts_open_format() now honours the compression field, no longer also requiring an explicit "z" in the mode string. Also fixed a 1 byte buffer overrun. (#880) * Removed duplicate hts_tpool_process_flush prototype. (#816, reported by James S Blachly) * Deleted defunct cram_tell declaration. (66c41e2; #915 reported by Martin Morgan) * Fixed overly aggressive filename suffix checking in bgzip. (#927, thanks to John Marshall; #129, reported by @hguturu) * Tabix and bgzip --help output now goes to standard output. (#754, thanks to John Marshall) * Fixed bgzip index creation when using multiple threads. (#817) * Made bgzip -b option honour -I (index filename). (#817) * Bgzip -d no longer attempts to unlink(NULL) when decompressing stdin. (#718) Miscellaneous other changes --------------------------- * Integration with Google OSS fuzzing for automatic detection of more bugs. (Thanks to Google for their assistance and the bugs it has found.) (#796, thanks to Markus Kusano) * aclocal.m4 now has the pkg-config macros. (6ec3b94d; #733 reported by Thomas Hickman) * Improved C++ compatibility of some header files. (#772; #771 reported by @cwrussell) * Improved strict C99 compatibility. (#860, thanks to John Marshall) * Travis and AppVeyor improvements to aid testing. (#747; #773 thanks to Lennard Berger; #781; #809; #804; #860; #909) * Various minor compiler warnings fixed. (#708; #765; #846, #860, thanks to John Marshall; #865; #966; #973) * Various new and improved error messages. * Documentation updates (mostly in the header files). * Even more testing with "make check". * Corrected many copyright dates. (#979) * The default non-configure Makefile now uses libcurl instead of knet, so it can support https. (#895) ------------------------------------------------------------------------------ samtools - changes v1.10 ------------------------------------------------------------------------------ Changes affecting the whole of samtools, or multiple sub-commands: * Samtools now uses the new HTSlib header API. As this adds more checks for invalid headers, it is possible that some illegal files will now be rejected when they would have been allowed by earlier versions. (#998) Examples of problems that will now be rejected include @SQ lines with no SN: tag, and @RG or @PG lines with no ID: tag. * samtools sub-commands will now add '@PG' header lines to output sam/bam/cram files. To disable this, use the '--no-PG' option. (#1087; #1097) * samtools now supports alignment records with reference positions greater than 2 gigabases. This allows samtools to process alignments for species which have large chromosomes, like axolotl and lungfish. Note that due to file format limitations, data with large reference positions must use the SAM format. (#1107; #1117) * Improved the efficiency of reading and writing SAM format data by 2 fold (single thread). This is further improved by the ability to use multiple threads, as previously done with BAM and CRAM. * samtools can now write BGZF-compressed SAM format. To enable this, either save files with a '.sam.gz' suffix, or use '--output-fmt sam.gz'. * samtools can now index BGZF-compressed SAM files. * The region parsing code has been improved to handle colons in reference names. Strings can be disambiguated by the use of braces, so for example when reference sequences called "chr1" and "chr1:100-200" are both present, the regions "{chr1}:100-200" and "{chr1:100-200}" unambiguously indicate which reference is being used. (#864) * samtools flags, flagstats, idxstats and stats now have aliases flag, flagstat, idxstat and stat. (#934) * A new global '--write-index' option has been added. This allows output sam.gz/bam/cram files to be indexed while they are being written out. This should work with addreplacerg, depad, markdup, merge, sort, split, and view. (#1062) * A global '--verbosity' option has been added to enable/disable debugging output. (#1124, thanks to John Marshall) * It is now possible to have data and index files stored in different locations. There are two ways to tell samtools where to find the index: 1. Samtools bedcov, depth, merge, mpileup, stats, tview, and view accept a new option (-X). When this is used, each input sam/bam/cram listed on the command line should have a corresponding index file. Note that all the data files should be listed first, followed by all the index files. (#978, thanks to Mingfei Shao) 2. A delimiter '##idx##' can be appended to the data file name followed by the index file name. This can be used both for input files and outputs when indexing on-the-fly. * HTSlib (and therefore SAMtools) now uses version 4 signatures by default for its s3:// plug-in. It can also write to S3 buckets, as long as version 4 signatures are in use. See HTSlib's NEWS file and htslib-s3-plugin manual page for more information. * HTSlib (and therefore SAMtools) no longer considers a zero-length file to be a valid SAM file. This has been changed so that pipelines such as `somecmd | samtools ...` with `somecmd` aborting before outputting anything will now propagate the error to the second command. * The samtools manual page has been split up into one for each sub-command. The main samtools.1 manual page now lists the sub-commands and describes the common global options. (#894) * The meaning of decode_md, store_md and store_nm in the fmt-option section of the samtools.1 man page has been clarified. (#898, thanks to Evan Benn) * Fixed numerous memory leaks. (#892) * Fixed incorrect macro definition on Windows. (#950) * bedcov, phase, misc/ace2sam and misc/wgsim now check for failure to open files. (#1013, thanks to Julie Blommaert and John Marshall) Changes affecting specific sub-commands: * A new "coverage" sub-command has been added. This prints a tabular format of the average coverage and percent coverage for each reference sequence, as well as number of aligned reads, average mapping quality and base quality. It can also (with the '-m' option) plot a histogram of coverage across the genome. (#992, thanks to Florian Breitwieser) * samtools calmd: - Reference bases in MD: tags are now converted to upper case. (#981, #988) * samtools depth: - Add new options to write a header to the output (-H) and to direct the output to a file (-o). (#937, thanks to Pierre Lindenbaum) - New options '-g' and '-G' can be used to filter reads. (#953) - Fix memory leak when failing to set CRAM options. (#985, thanks to Florian Breitwieser) - Fix bug when using region filters where the '-a' option did not work for regions with no coverage. (#1113; #1112 reported by Paweł Sztromwasser) * samtools fasta and fastq: - '-1 FILE -2 FILE' with the same filename now works properly. (#1042) - '-o FILE' is added as a synonym for '-1 FILE -2 FILE'. (#1042) - The '-F' option now defaults to 0x900 (SECONDARY,SUPPLEMENTARY). Previously secondary and supplementary records were filtered internally in a way that could not be turned off. (#1042; #939 reported by @finswimmer) - Allow reading from a pipe without an explicit '-' on the command line. (#1042; #874 reported by John Marshall) - Turn on multi-threading for bgzf compressed output files. (#908) - Fixed bug where the samtools fastq -i would output incorrect information in the Casava tags for dual-index reads. It also now prints the tags for dual indices in the same way as bcl2fastq, using a '+' sign between the two parts of the index. (#1059; #1047 reported by Denis Loginov) * samtools flagstat: - Samtools flagstat can now optionally write its output in JSON format or as a tab-separated values file. (#1106, thanks to Vivek Rai). * samtools markdup: - It can optionally tag optical duplicates (reads following Illumina naming conventions only). The is enabled with the '-d' option, which sets the distance for duplicates to be considered as optical. (#1091; #1103; #1121; #1128; #1134) - The report stats (-s) option now outputs counts for optical and non-primary (supplementary / secondary) duplicates. It also reports the Picard "estimate library size" statistic. A new '-f' option can be used to save the statistics in a given file. (#1091) - The rules for calling duplicates can be changed using the new --mode option. This mainly changes the position associated with each read in a pair. '--mode t' (the default) is the existing behaviour where the position used is that of the outermost template base associated with the read. Alternatively '--mode s' always uses the first unclipped sequence base. In practice, this only makes a difference for read pairs where the two reads are aligned in the same direction. (#1091) - A new '-c' option can be used to clear any existing duplicate tags. (#1091) - A new '--include-fails' option makes markdup include QC-failed reads. (#1091) - Fixed buffer overflow in temporary file writer when writing a mixture of long and short alignment records. (#911; #909) * samtools mpileup: - mpileup can now process alignments including CIGAR P (pad) operators correctly. They will now also produce the correct output for alignments where insertions are immediately followed by deletions, or deletions by insertions. Note that due to limitations in HTSlib, they are still unable to output sequences that have been inserted before the first aligned base of a read. (#847; #842 reported by Tiffany Delhomme. See also htslib issue #59 and pull request #699). - In samtools mpileup, a deletion or pad on the reverse strand is now marked with a different character ('#') than the one used on a forward strand ('*'), if the '--reverse-del' option is used. (#1070) - New option '--output-extra' can be used to add columns for user selected alignment fields or aux tags. (#1073) - Fixed double-counting of overlapping bases in alignment records with deletions or reference skips longer than twice the insert size. (#989; #987 reported by @dariomel) - Improved manual page with documentation about what each output column means. (#1055, thanks to John Marshall) * samtools quickcheck: - Add unmapped (-u) option, which disables the check for @SQ lines in the header. (#920, thanks to Shane McCarthy) * samtools reheader: - A new option '-c' allows the input header to be passed to a given command. Samtools then takes the output of this command and uses it as the replacement header. (#1007) - Make it clear in help message that reheader --in-place only works on CRAM files. (#921, thanks to Julian Gehring) - Refuse to in-place reheader BAM files, instead of unexpectedly writing a BAM file to stdout. (#935) * samtools split: - In samtools split, the '-u' option no longer accepts an extra file name from which a replacement header was read. The two file names were separated using a colon, which caused problems on Windows and prevented the use of URLs. A new '-h' option has been added to allow the replacement header file to be specified in its own option. (#961) - Fixed bug where samtools split would crash if it read a SAM header that contained an @RG line with no ID tag. (#954, reported by @blue-bird1) * samtools stats: - stats will now compute base compositions for BC, CR, OX and RX tags, and quality histograms for QT, CY, BZ and QX tags. (#904) - New stats FTC and LTC showing total number of nucleotides for first and last fragments. (#946) - The rules for classifying reads as "first" or "last" fragment have been tightened up. (#949) - Fixed bug where stats could over-estimate coverage when using the target-regions option or when a region was specified on the command-line. (#1027; #1025, reported by Miguel Machado; #1029, reported by Jody Phelan). - Fixed error in stats GCD percentile depth calculation when the depth to be reported fell between two bins. It would report the depth entirely from the lower bin instead of taking a weighted average of the two. (#1048) - Better catching and reporting of out of memory conditions. (#984; #982, reported by Jukka Matilainen) - Improved manual page. (#927) * samtools tview: - tview can now display alignments including CIGAR P operators, D followed by I and I followed by D correctly. See mpileup above for more information. (#847; #734, reported by Ryan Lorig-Roach) - The "go to position" text entry box has been made wider. (#968, thanks to John Marshall) - Fixed samtools tview -s option which was not filtering reads correctly. It now only shows reads from the requested sample or read group. (#1089) * samtools view: - New options '-d' and '-D' to only output alignments which have a tag with a given type and value. (#1001, thanks to Gert Hulselmans) * misc/plot-bamstats script: - Fixed merge (-m) option. (#923, #924 both thanks to Marcus D Sherman) - Made the quality heatmap work with gnuplot version 5.2.7 and later. (#1068; #1065 reported by Martin Mokrejš) - Fixed --do-ref-stats bug where fasta header lines would be counted as part of the sequence when the --targets option was used. (#1120, thanks to Neil Goodgame) * Removed the misc/varfilter.py Python script, as it takes consensus-pileup as input, which was removed from samtools in release 0.1.17 in 2011. (#1125) ------------------------------------------------------------------------------ bcftools - changes v1.10 ------------------------------------------------------------------------------ * Numerous bug fixes, usability improvements and sanity checks were added to prevent common user errors. * The -r, --regions (and -R, --regions-file) option should never create unsorted VCFs or duplicates records again. This also fixes rare cases where a spanning deletion makes a subsequent record invisible to `bcftools isec` and other commands. * Additions to filtering and formatting expressions - support for the spanning deletion alternate allele (ALT=*) - new ILEN filtering expression to be able to filter by indel length - new MEAN, MEDIAN, MODE, STDEV, phred filtering functions - new formatting expression %PBINOM (phred-scaled binomial probability), %INFO (the whole INFO column), %FORMAT (the whole FORMAT column), %END (end position of the REF allele), %END0 (0-based end position of the REF allele), %MASK (with multiple files indicates the presence of the site in other files) * New plugins - `+gvcfz`: compress gVCF file by resizing gVCF blocks according to specified criteria - `+indel-stats`: collect various indel-specific statistics - `+parental-origin`: determine parental origin of a CNV region - `+remove-overlaps`: remove overlapping variants. - `+split-vep`: query structured annotations such INFO/CSQ created by bcftools/csq or VEP - `+trio-dnm`: screen variants for possible de-novo mutations in trios * `annotate` - new -l, --merge-logic option for combining multiple overlapping regions * `call` - new `bcftools call -G, --group-samples` option which allows grouping samples into populations and applying the HWE assumption within but not across the groups. * `csq` - significant reduction of memory usage in the local -l mode for VCFs with thousands of samples and 20% reduction in the non-local haplotype-aware mode. - fixes a small memory leak and formatting issue in FORMAT/BCSQ at sites with many consequences - do not print protein sequence of start_lost events - support for "start_retained" consequence - support for symbolic insertions (ALT="<INS...>"), "feature_elongation" consequence - new -b, --brief-predictions option to output abbreviated protein predictions. * `concat` - the `--naive` command now checks header compatibility when concatenating multiple files. * `consensus` - add a new `-H, --haplotype 1pIu/2pIu` feature to output first/second allele for phased genotypes and the IUPAC code for unphased genotypes - new -p, --prefix option to add a prefix to sequence names on output * `+contrast` - added support for Fisher's test probability and other annotations * `+fill-from-fasta` - new -N, --replace-non-ACGTN option * `+dosage` - fix some serious bugs in dosage calculation * `+fill-tags` - extended to perform simple on-the-fly calculations such as calculating INFO/DP from FORMAT/DP. * `merge` - add support for merging FORMAT strings - bug fixed in gVCF merging * `mpileup` - a new optional SCR annotation for the number of soft-clipped reads * `reheader` - new -f, --fai option for updating contig lines in the VCF header * `+trio-stats` - extend output to include DNM homs and recurrent DNMs * VariantKey support -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |
From: Carrasco V. T. <car...@iz...> - 2019-07-25 15:47:15
|
Hi to all I have a question about the SGB value in mpileup, I read the equations in "Segregation based metric for variant call QC", but maybe because of my poor math skills jejeje, but I don't get how is possible to get negatives values. Nevertheless as far as I understand SGB value look to discriminate if a nonRef variant is real or not according to the deep using Poisson and HWE, is this right? what's the difference between a value of -0.5 and -0.6? Thanks very much and I'm very sorry to bother you with this kind of questions, Best |
From: Darin L. <dar...@gm...> - 2019-07-01 18:15:01
|
Team, I was looking through all the manual pages for SAMtools, etc and I didn’t find anything on compiling SAMtools or htslib with MPI. Is it possible, supported, and documented? -Darin |