Re: [Samtools-devel] java sam too slow over http

[Jim R. <jro...@br...>]

Hi,

As Alec indicates we haven't spent anytime optimizing the buffer size  
because performance has been good in general.  I suspect with a 1kb  
buffer you are going to get poor performance on the queries as this is  
going to cause many more http requests to be made.   The theory is  
that it is better to make a few large http requests than many small  
ones, but this isn't necessarily universally true.   However it  
doesn't make sense to me that it would take 14 seconds to load 512kb  
of data over http,  there could be an issue with your web server.

Keep us informed of your findings!

Thanks

Jim


On Sep 28, 2010, at 8:08 AM, Alec Wysoker wrote:

> Hi Wolf,
>
> I must confess that I don't ever use BAM over HTTP, except when  
> running
> unit tests.  Jim Robinson has been our primary client (and author) for
> that functionality, and as he seems satisfied with it I haven't looked
> at it very hard.
>
> I just added a ctor for SeekableBufferedStream in which you can pass  
> in
> whatever buffer size you like.  If you want to use a different buffer
> size, you can replace code like this:
>
>     new SAMFileReader(url, indexFile, eagerDecode)
>
> with this:
>
>     new SAMFileReader(new SeekableBufferedStream(new
> SeekableHTTPStream(url), yourDesiredBufferSize), indexFile,  
> eagerDecode)
>
> Let me know how it goes.
>
> -Alec
>
>
> On 9/28/10 7:00 AM, Asraniel wrote:
>> i searched a little more and i found the source of my problem to be  
>> the buffer
>> size:
>>
>> public class SeekableBufferedStream extends SeekableStream {
>>
>>     private int bufferSize = 512000;
>>
>> this means that the constructor will at any case download 512kb.  
>> there is
>> probably no reason for that. i was able to get the constructor time  
>> to 2
>> seconds by changing this value to 1000. There are probably better  
>> values.
>>
>> So, what was the exact reasoning behind this value?
>>
>> thank you
>>
>> Beat Wolf
>>
>> Am Dienstag 28 September 2010, um 11.29:11 schrieb Asraniel:
>>
>>> hi,
>>>
>>> i'm playing around with loading bam files over http. What i  
>>> noticed is that
>>> it is quite slow. slower than it should be.
>>> To give some numbers:
>>> Constructor of SAMFileReader for a local bamfile is nearly 0
>>> Same but for a http bam file>  14 seconds.
>>>
>>> the bam file is quite small, so i would expect it to be much faster.
>>>
>>> I was wondering if there are any plans to optimize this?
>>>
>>> thank you
>>>
>>> Beat Wolf
>>>
>>
>> ------------------------------------------------------------------------------
>> Start uncovering the many advantages of virtual appliances
>> and start using them to simplify application deployment and
>> accelerate your shift to cloud computing.
>> http://p.sf.net/sfu/novell-sfdev2dev
>> _______________________________________________
>> Samtools-devel mailing list
>> Sam...@li...
>> https://lists.sourceforge.net/lists/listinfo/samtools-devel
>>
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Samtools-devel mailing list
> Sam...@li...
> https://lists.sourceforge.net/lists/listinfo/samtools-devel