From: Daofeng Li <li...@gm...> - 2013-11-26 23:58:06
|
Hi Heng, I still have the minor issue that the different behavior of kurl-tabix between 32bit and 64bit system. I am querying a big .gz file (>4G) in an HTTP URL, 64bit tabix gives me correct result but 32bit one gives nothing. I open the verbose output of tabix, the 64bit gives message: < HTTP/1.1 206 Partial Content < Date: Tue, 26 Nov 2013 22:38:55 GMT < Server: Apache/2.2.22 (Ubuntu) < Last-Modified: Sat, 19 Oct 2013 14:23:26 GMT < ETag: "23dc7-11d67e47a-4e918cb52f57e" < Accept-Ranges: bytes < Content-Length: 2580773085 < Content-Range: bytes 2207542173-4788315257/4788315258 < Content-Type: application/x-gzip The 32bit: < HTTP/1.1 200 OK < Date: Tue, 26 Nov 2013 22:19:26 GMT < Server: Apache/2.2.22 (Ubuntu) < Last-Modified: Sat, 19 Oct 2013 14:23:26 GMT < ETag: "23dc7-11d67e47a-4e918cb52f57e" < Accept-Ranges: bytes < Content-Length: 4788315258 < Content-Type: application/x-gzip < X-Pad: avoid browser bug < * HTTP server doesn't seem to support byte ranges. Cannot resume. * Closing connection #0 I am wondering that is there a range limit of 32 bit system or something else caused this difference? Any response would be appreciated. Thanks for your time. Best, Daofeng On Tue, Nov 26, 2013 at 3:18 PM, Daofeng Li <li...@gm...> wrote: > Hi Heng, > > Yes, it works very well :) > > Thank you very much!!! > > Best, > > Daofeng > > > On Tue, Nov 26, 2013 at 3:09 PM, Heng Li <lh...@sa...> wrote: > >> Please update to the latest kurl.{c,h} from >> github.com/attractivechaos/klib. The older version was indeed buggy on >> multiple seeking. >> >> Heng >> >> On Nov 26, 2013, at 3:10 PM, Daofeng Li <li...@gm...> wrote: >> >> Hi Heng, >> >> Seems this version of samtools have a problem when querying multiple >> regions (supply more than 1 region from command line) >> >> For example: >> The official samtools: >> >> samtools view http://url/bam1.bam chr10:96799628-96829654 > y >> >> [knet_seek] SEEK_END is not supported for HTTP. Offset is unchanged. >> >> [bam_header_read] EOF marker is absent. The input is probably truncated. >> >> >> this samtools: >> >> ./samtools view http://url/bam1.bam chr10:96799628-96829654 >> chr1:47285869-47285883 > x >> >> [bam_header_read] EOF marker is absent. The input is probably truncated. >> >> Segmentation fault (core dumped) >> >> >> So the file y contains correct output, but file x was an empty file. >> >> >> I ran gdb over this samtools binary, here are some output from gdb: >> >> >> Program received signal SIGSEGV, Segmentation fault. >> >> 0x00007ffff6d952bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> >> (gdb) bt >> >> #0 0x00007ffff6d952bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> >> #1 0x00000000004400e4 in kurl_read (ku=0x663060, buf=0x7fffffffdb00, >> nbytes=18) at /usr/include/x86_64-linux-gnu/bits/string3.h:52 >> >> #2 0x00000000004267cf in bgzf_read_block (fp=0x69b2a0) at bgzf.c:322 >> >> #3 0x0000000000427425 in bgzf_read (fp=0x69b2a0, data=<optimized out>, >> length=4) at bgzf.c:360 >> >> #4 0x000000000042d94b in bam_read1 (fp=0x69b2a0, b=0xf9b510) at bam.c:198 >> >> #5 0x0000000000435fa1 in bam_iter_read (fp=0x69b2a0, iter=0xf9b970, >> b=0xf9b510) at bam_index.c:731 >> >> #6 0x000000000043615f in bam_fetch (fp=0x69b2a0, idx=0x69a180, tid=0, >> beg=<optimized out>, end=<optimized out>, data=0x6734f0, func=0x408a10 >> <view_func>) at bam_index.c:751 >> >> #7 0x0000000000409bac in main_samview (argc=4, argv=0x7fffffffe2c0) at >> sam_view.c:266 >> >> #8 0x00007ffff6c6c76d in __libc_start_main () from >> /lib/x86_64-linux-gnu/libc.so.6 >> >> #9 0x0000000000403735 in _start () >> >> >> Seems the problem was caused by kurl.c? Do you happen to have some >> suggestions? >> >> >> Thanks :) >> >> Daofeng >> >> >> On Mon, Nov 18, 2013 at 10:31 PM, Heng Li <lh...@sa...> wrote: >> >>> So, I have implemented a knetfile replacement on top of libcurl. A crude >>> proof of implementation version is here: >>> >>> https://github.com/lh3/samtools-legacy/tree/kurl >>> >>> If your libcurl is compiled --with-ssl --with-libssh2, you can use this >>> version of samtools to open BAM files from https:// or sftp://. As many >>> ssh servers are configured to allow sftp, you can run "samtools tview" on a >>> BAM from your remote servers via ssh. >>> >>> kurl.{c,h} is the key component. It is largely a drop-in replacement of >>> knetfile with very similar APIs. Perhaps it is worth evaluating whether we >>> should add kurl.{c,h} to htslib as a compile time option. The obvious >>> downside is that using libcurl adds dependencies, including the >>> dependencies (e.g. openssl and libssh2) that libcurl relies on. >>> However, libcurl supports more protocols as well as the majority of >>> authentication methods. I believe it is also more robust than knetfile. >>> >>> Heng >>> >>> PS: you may compile kurl.{c,h} with: >>> >>> gcc -g -Wall -O2 -DKURL_MAIN kurl.c -L/usr/local/lib -lcurl -o kurl >>> >>> The binary "kurl" works like "cat" on local files and "curl" on remote >>> files. >>> >>> On Jul 8, 2013, at 10:43 AM, Heng Li <lh...@sa...> wrote: >>> >>> libcurl has an example fopen.c <http://curl.haxx.se/libcurl/c/fopen.html>. >>> It supports reading a stream but does not support random access for now. In >>> principle, you can start from there to implement a full knetfile >>> replacement on top of libcurl. >>> >>> Heng >>> >>> On Jul 8, 2013, at 9:54 AM, Daofeng Li <li...@gm...> wrote: >>> >>> H >>> ello John, >>> >>> Thank you very much for your response. >>> Understand that less dependency is good for compiling at >>> different environments. >>> I tried to search some related webpages by Google about how >>> to incorporate libcurl to samtools, seems there's no such content. I was >>> not able to understand the API documentation listed in libcurl's website. >>> >>> Do you happen to have some guide to how to build samtools with libcurl? >>> I just need to access files stored at Dropbox, which own https:// links. >>> >>> Thanks again. >>> >>> Best, >>> >>> Daofeng >>> >>> >>> On Mon, Jul 8, 2013 at 4:24 AM, John Marshall <jm...@sa...> wrote: >>> >>>> On 5 Jul 2013, at 19:38, Daofeng Li wrote: >>>> > Is it possible that to make kentfile support https:// link? >>>> >>>> knetfile contains its own very simple implementations of FTP and HTTP >>>> clients. It won't ever contain its own SSL implementation and at the >>>> moment there is no code in it to use e.g. openssl; so at the moment https: >>>> URLs are not supported. >>>> >>>> We've always been reluctant to acquire dependencies on external >>>> libraries such as libcurl or openssl, as this makes it harder for people to >>>> build samtools in all their various environments. Probably the sensible >>>> way to support https: URLs would be to have the option at build-time to use >>>> libcurl instead of knetfile.c/.h, so you could choose to build it with >>>> extra dependencies to get this functionality. We've talked about doing >>>> this, but it's not high on our priority list at the moment. >>>> >>>> John >>>> >>>> -- >>>> The Wellcome Trust Sanger Institute is operated by Genome Research >>>> Limited, a charity registered in England with number 1021457 and a >>>> company registered in England with number 2742969, whose registered >>>> office is 215 Euston Road, London, NW1 2BE. >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> This SF.net <http://sf.net/> email is sponsored by Windows: >>> >>> Build for Windows Store. >>> >>> >>> http://p.sf.net/sfu/windows-dev2dev_______________________________________________ >>> Samtools-help mailing list >>> Sam...@li... >>> https://lists.sourceforge.net/lists/listinfo/samtools-help >>> >>> >>> >>> -- The Wellcome Trust Sanger Institute is operated by Genome Rese arch >>> Limited, a charity registered in England with number 1021457 and a company >>> registered in England with number 2742969, whose registered office is 215 >>> Euston Road, London, NW1 2BE. >>> >>> ------------------------------------------------------------------------------ >>> This SF.net <http://sf.net/> email is sponsored by Windows: >>> >>> Build for Windows Store. >>> >>> >>> http://p.sf.net/sfu/windows-dev2dev_______________________________________________ >>> Samtools-help mailing list >>> Sam...@li... >>> https://lists.sourceforge.net/lists/listinfo/samtools-help >>> >>> >>> >>> -- The Wellcome Trust Sanger Institute is operated by Genome Research >>> Limited, a charity registered in England with number 1021457 and a company >>> registered in England with number 2742969, whose registered office is 215 >>> Euston Road, London, NW1 2BE. >>> >> >> >> >> -- The Wellcome Trust Sanger Institute is operated by Genome Research >> Limited, a charity registered in England with number 1021457 and a company >> registered in England with number 2742969, whose registered office is 215 >> Euston Road, London, NW1 2BE. >> > > |