Re: [micro-manager-general] Saving images at high speed

[Nico S. <nic...@uc...>]

Hi Pavak,

> This is great news! I'm hoping to take advantage of this once the 
> diSPIM plugin in 2.0g is a bit further along.

Good point, and you may want to bring this up with Jon Daniels.  I 
ported a then current version of the diSPIM plugin to 2.0 a while ago, 
but developments on both sides (diSPIM 1.4 plugin and MM 2.0) probably 
make that port useless, so this will need to be done again.  I'll be 
happy to help, but kind of waiting for Jon to take the lead on that.

> I've noticed some behavior in 1.4 that, I know is probably not worth 
> resolving with 2.0 as the future, but was wondering if you have 
> insight into and whether the underlying bottleneck in 2.0 may have 
> been the same (and potentially resolved with these new optimizations).
>
> When acquiring relatively small frames (say 400x400 px) at high 
> effective framerates (280 fps in my case, 4 volumes / s, 35 z 
> positions x 2 cameras per volume), 1.4 struggles to empty the sequence 
> buffer when acquiring to a fast NVMe drive, to a 5x SATA SSD RAID0, or 
> to memory.
>
> Initially the buffer is kept empty, but over time performance blips 
> (maybe background system interrupts causing hiccups?) result in the 
> accumulation of 100 or so frames in the buffer. These blips are 
> quickly cleared out, but after a few hundred volumes are acquired the 
> acquisition is not able to keep the buffer emptying fast enough and 
> frames start steadily accumulating in the sequence buffer. If we use a 
> large buffer (say 32 GB), we can finish the acquisition but it takes 
> tens of minutes for the buffer to then empty out. This seems to be 
> behavior tied to FPS rather than the average data throughput, as 
> increasing the per frame size by 4x as many pixels and decreasing the 
> effective FPS by the same ratio resolves this behavior. In terms of 
> system details, this is a new diSPIM running 2x Hamamatsu Fusions on 
> the 3/9 nightly build of 1.4. The workstation itself is fairly 
> powerful with 2x 8-core xeon's running at 3.4 GHz.

I am afraid that the behavior on 2.0 will be similar.  The blips you are 
seeing are most likely cause by the Java garbage collector, which has a 
lot of work to do, because lots of objects are created and destroyed.  
This is in large part due to the image metadata.  It will be a useful 
new project to assert that image metadata processing and saving is 
indeed a bottleneck for the type of datasets you describe, and study how 
to reduce that bottleneck.  My first inclination to discard per image 
metadata after the first image will not work for your situation, since 
you will at least need the metadata tag indicating the camera the image 
came from. I bought the O'Reailly book on "Java performance", so that is 
a start;)


Best,

Nico