From: Daofeng Li <li...@gm...> - 2013-11-23 22:58:26
|
Hi Heng, Sorry for bothering again. I have compiled tabix to use kurl successfully like I mentioned before (on an X86_64 Ubuntu). Today I am trying to build same kurl tabix on an i686 Ubuntu, I could compile the binary, but this time the tabix binary could not fetch data of a bgzip file in my URL. (The one compiled on X86_64 works well.) In the i686 Ubuntu, I also tried in install officail tabix by 'apt-get install tabix', when I am trying to query my HTTP file, I got error messages like: tabix http://my.url.gz chr7:118763739-118864386 -f [get_local_version] downloading the index file... *** glibc detected *** tabix: free(): invalid pointer: 0x09ccc8e0 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x73e42)[0xb75a5e42] tabix[0x8050acf] tabix[0x8053955] tabix[0x8049ae5] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb754b4d3] tabix[0x8049e0d] ======= Memory map: ======== 08048000-08059000 r-xp 00000000 08:05 2105508 /usr/bin/tabix 08059000-0805a000 r--p 00010000 08:05 2105508 /usr/bin/tabix 0805a000-0805b000 rw-p 00011000 08:05 2105508 /usr/bin/tabix 09c8b000-0a605000 rw-p 00000000 00:00 0 [heap] b70ba000-b70d6000 r-xp 00000000 08:05 6816692 /lib/i386-linux-gnu/libgcc_s.so.1 b70d6000-b70d7000 r--p 0001b000 08:05 6816692 /lib/i386-linux-gnu/libgcc_s.so.1 b70d7000-b70d8000 rw-p 0001c000 08:05 6816692 /lib/i386-linux-gnu/libgcc_s.so.1 b70d8000-b7121000 rw-p 00000000 00:00 0 b7145000-b7293000 rw-p 00000000 00:00 0 b7293000-b72dc000 rw-p 00000000 00:00 0 b72dc000-b7325000 rw-p 00000000 00:00 0 b7325000-b736e000 rw-p 00000000 00:00 0 b736e000-b73b7000 rw-p 00000000 00:00 0 b73b7000-b7400000 rw-p 00000000 00:00 0 b7400000-b7449000 rw-p 00000000 00:00 0 b7449000-b7492000 rw-p 00000000 00:00 0 b7492000-b74db000 rw-p 00000000 00:00 0 b74db000-b7500000 rw-p 00000000 00:00 0 b7500000-b7513000 r-xp 00000000 08:05 6816755 /lib/i386-linux-gnu/ libresolv-2.15.so b7513000-b7514000 ---p 00013000 08:05 6816755 /lib/i386-linux-gnu/ libresolv-2.15.so b7514000-b7515000 r--p 00013000 08:05 6816755 /lib/i386-linux-gnu/ libresolv-2.15.so b7515000-b7516000 rw-p 00014000 08:05 6816755 /lib/i386-linux-gnu/ libresolv-2.15.so b7516000-b7518000 rw-p 00000000 00:00 0 b7518000-b751d000 r-xp 00000000 08:05 6816720 /lib/i386-linux-gnu/ libnss_dns-2.15.so b751d000-b751e000 r--p 00004000 08:05 6816720 /lib/i386-linux-gnu/ libnss_dns-2.15.so b751e000-b751f000 rw-p 00005000 08:05 6816720 /lib/i386-linux-gnu/ libnss_dns-2.15.so b751f000-b7521000 r-xp 00000000 08:05 6815799 /lib/libnss_mdns4_minimal.so.2 b7521000-b7522000 r--p 00001000 08:05 6815799 /lib/libnss_mdns4_minimal.so.2 b7522000-b7523000 rw-p 00002000 08:05 6815799 /lib/libnss_mdns4_minimal.so.2 b7523000-b752e000 r-xp 00000000 08:05 6816722 /lib/i386-linux-gnu/ libnss_files-2.15.so b752e000-b752f000 r--p 0000a000 08:05 6816722 /lib/i386-linux-gnu/ libnss_files-2.15.so b752f000-b7530000 rw-p 0000b000 08:05 6816722 /lib/i386-linux-gnu/ libnss_files-2.15.so b7530000-b7532000 rw-p 00000000 00:00 0 b7532000-b76d1000 r-xp 00000000 08:05 6816671 /lib/i386-linux-gnu/ libc-2.15.so b76d1000-b76d3000 r--p 0019f000 08:05 6816671 /lib/i386-linux-gnu/ libc-2.15.so b76d3000-b76d4000 rw-p 001a1000 08:05 6816671 /lib/i386-linux-gnu/ libc-2.15.so b76d4000-b76d7000 rw-p 00000000 00:00 0 b76d7000-b76eb000 r-xp 00000000 08:05 6816782 /lib/i386-linux-gnu/libz.so.1.2.3.4 b76eb000-b76ec000 r--p 00013000 08:05 6816782 /lib/i386-linux-gnu/libz.so.1.2.3.4 b76ec000-b76ed000 rw-p 00014000 08:05 6816782 /lib/i386-linux-gnu/libz.so.1.2.3.4 b7701000-b7703000 rw-p 00000000 00:00 0 b7703000-b7704000 r-xp 00000000 00:00 0 [vdso] b7704000-b7724000 r-xp 00000000 08:05 6816651 /lib/i386-linux-gnu/ ld-2.15.so b7724000-b7725000 r--p 0001f000 08:05 6816651 /lib/i386-linux-gnu/ ld-2.15.so b7725000-b7726000 rw-p 00020000 08:05 6816651 /lib/i386-linux-gnu/ ld-2.15.so bff7c000-bff9d000 rw-p 00000000 00:00 0 [stack] Aborted (core dumped) And 2nd time when I try this same command, I got another error message: tabix http://my.url.gz chr7:118763739-118864386 -f *** glibc detected *** tabix: free(): invalid pointer: 0x08de48d0 *** *** glibc detected *** tabix: corrupted double-linked list: 0x08de4730 *** Inconsistency detected by ld.so: dl-open.c: 221: dl_open_worker: Assertion `_dl_debug_initialize (0, args->nsid)->r_state == RT_CONSISTENT' failed! I have omited the true URL, but if you want to give it a test, please let me know. Do you happen to know why tabix would work differently between x86_64 and i686 platform? Do I need to make some changes to make it work also on i686? Thanks for your time. Best, Daofeng On Tue, Nov 19, 2013 at 2:31 PM, Daofeng Li <li...@gm...> wrote: > Hi Heng, > > I managed to figure it out :) > I still need to edit index.c bgzf.c and main.c (according to the files in > samtools folder). > > Thank you again! I got both samtools and tabix works on https:// now. > > Best, > > Daofeng > > > On Tue, Nov 19, 2013 at 12:44 PM, Daofeng Li <li...@gm...> wrote: > >> Hi Heng, >> >> sorry for disturb again. >> I compiled the samtools in the kurl branch, it works very well when I am >> trying to access the bam file in my dropbox folder (https:// link). >> I also want to apply this kurl module to tabix, so I copied the >> kurl.{c,h} to a fresh downloaded tabix folder. Edit the Makefile as >> following: >> >> (replace all knentfile with kurl) >> >> *DFLAGS*= -D_FILE_OFFSET_BITS=64 -D_USE_KURL -DBGZF_CACHE >> >> *tabix:*lib *$(AOBJS)* >> >> *$(CC)* *$(CFLAGS)* -o *$@* *$(AOBJS)* -L. -ltabix -lm >> *$(LIBPATH)* -lz -L/usr/local/lib -lcurl >> >> >> *bgzip:*bgzip.o bgzf.o kurl.o >> >> *$(CC)* *$(CFLAGS)* -o *$@* bgzip.o bgzf.o kurl.o -lz >> -L/usr/local/lib -lcurl >> >> >> I could be able to compile bgzip and tabix successfully, but I am still >> could not access the bgzip/tabix processed bed files in my dropbox folder. >> >> The error message was "[main] fail to load the index file." >> >> >> Am I doing something wrong here? >> >> Thanks for any suggestions in advance :) >> >> >> Best, >> >> Daofeng >> >> >> On Mon, Nov 18, 2013 at 10:31 PM, Heng Li <lh...@sa...> wrote: >> >>> So, I have implemented a knetfile replacement on top of libcurl. A crude >>> proof of implementation version is here: >>> >>> https://github.com/lh3/samtools-legacy/tree/kurl >>> >>> If your libcurl is compiled --with-ssl --with-libssh2, you can use this >>> version of samtools to open BAM files from https:// or sftp://. As many >>> ssh servers are configured to allow sftp, you can run "samtools tview" on a >>> BAM from your remote servers via ssh. >>> >>> kurl.{c,h} is the key component. It is largely a drop-in replacement of >>> knetfile with very similar APIs. Perhaps it is worth evaluating whether we >>> should add kurl.{c,h} to htslib as a compile time option. The obvious >>> downside is that using libcurl adds dependencies, including the >>> dependencies (e.g. openssl and libssh2) that libcurl relies on. >>> However, libcurl supports more protocols as well as the majority of >>> authentication methods. I believe it is also more robust than knetfile. >>> >>> Heng >>> >>> PS: you may compile kurl.{c,h} with: >>> >>> gcc -g -Wall -O2 -DKURL_MAIN kurl.c -L/usr/local/lib -lcurl -o kurl >>> >>> The binary "kurl" works like "cat" on local files and "curl" on remote >>> files. >>> >>> On Jul 8, 2013, at 10:43 AM, Heng Li <lh...@sa...> wrote: >>> >>> libcurl has an example fopen.c <http://curl.haxx.se/libcurl/c/fopen.html>. >>> It supports reading a stream but does not support random access for now. In >>> principle, you can start from there to implement a full knetfile >>> replacement on top of libcurl. >>> >>> Heng >>> >>> On Jul 8, 2013, at 9:54 AM, Daofeng Li <li...@gm...> wrote: >>> >>> H >>> ello John, >>> >>> Thank you very much for your response. >>> Understand that less dependency is good for compiling at >>> different environments. >>> I tried to search some related webpages by Google about how >>> to incorporate libcurl to samtools, seems there's no such content. I was >>> not able to understand the API documentation listed in libcurl's website. >>> >>> Do you happen to have some guide to how to build samtools with libcurl? >>> I just need to access files stored at Dropbox, which own https:// links. >>> >>> Thanks again. >>> >>> Best, >>> >>> Daofeng >>> >>> >>> On Mon, Jul 8, 2013 at 4:24 AM, John Marshall <jm...@sa...> wrote: >>> >>>> On 5 Jul 2013, at 19:38, Daofeng Li wrote: >>>> > Is it possible that to make kentfile support https:// link? >>>> >>>> knetfile contains its own very simple implementations of FTP and HTTP >>>> clients. It won't ever contain its own SSL implementation and at the >>>> moment there is no code in it to use e.g. openssl; so at the moment https: >>>> URLs are not supported. >>>> >>>> We've always been reluctant to acquire dependencies on external >>>> libraries such as libcurl or openssl, as this makes it harder for people to >>>> build samtools in all their various environments. Probably the sensible >>>> way to support https: URLs would be to have the option at build-time to use >>>> libcurl instead of knetfile.c/.h, so you could choose to build it with >>>> extra dependencies to get this functionality. We've talked about doing >>>> this, but it's not high on our priority list at the moment. >>>> >>>> John >>>> >>>> -- >>>> The Wellcome Trust Sanger Institute is operated by Genome Research >>>> Limited, a charity registered in England with number 1021457 and a >>>> company registered in England with number 2742969, whose registered >>>> office is 215 Euston Road, London, NW1 2BE. >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> This SF.net <http://sf.net/> email is sponsored by Windows: >>> >>> Build for Windows Store. >>> >>> >>> http://p.sf.net/sfu/windows-dev2dev_______________________________________________ >>> Samtools-help mailing list >>> Sam...@li... >>> https://lists.sourceforge.net/lists/listinfo/samtools-help >>> >>> >>> >>> -- The Wellcome Trust Sanger Institute is operated by Genome Rese arch >>> Limited, a charity registered in England with number 1021457 and a company >>> registered in England with number 2742969, whose registered office is 215 >>> Euston Road, London, NW1 2BE. >>> >>> ------------------------------------------------------------------------------ >>> This SF.net email is sponsored by Windows: >>> >>> Build for Windows Store. >>> >>> >>> http://p.sf.net/sfu/windows-dev2dev_______________________________________________ >>> Samtools-help mailing list >>> Sam...@li... >>> https://lists.sourceforge.net/lists/listinfo/samtools-help >>> >>> >>> >>> -- The Wellcome Trust Sanger Institute is operated by Genome Research >>> Limited, a charity registered in England with number 1021457 and a company >>> registered in England with number 2742969, whose registered office is 215 >>> Euston Road, London, NW1 2BE. >>> >> >> > |