Re: [micro-manager-general] Saving images at high speed

[PAVAK S. <pa...@uc...>]

Hi Nico,

This is great news! I'm hoping to take advantage of this once the diSPIM
plugin in 2.0g is a bit further along.

I've noticed some behavior in 1.4 that, I know is probably not worth
resolving with 2.0 as the future, but was wondering if you have insight
into and whether the underlying bottleneck in 2.0 may have been the same
(and potentially resolved with these new optimizations).

When acquiring relatively small frames (say 400x400 px) at high effective
framerates (280 fps in my case, 4 volumes / s, 35 z positions x 2 cameras
per volume), 1.4 struggles to empty the sequence buffer when acquiring to a
fast NVMe drive, to a 5x SATA SSD RAID0, or to memory.

Initially the buffer is kept empty, but over time performance blips (maybe
background system interrupts causing hiccups?) result in the accumulation
of 100 or so frames in the buffer. These blips are quickly cleared out, but
after a few hundred volumes are acquired the acquisition is not able to
keep the buffer emptying fast enough and frames start steadily accumulating
in the sequence buffer. If we use a large buffer (say 32 GB), we can finish
the acquisition but it takes tens of minutes for the buffer to then empty
out. This seems to be behavior tied to FPS rather than the average data
throughput, as increasing the per frame size by 4x as many pixels and
decreasing the effective FPS by the same ratio resolves this behavior.

In terms of system details, this is a new diSPIM running 2x Hamamatsu
Fusions on the 3/9 nightly build of 1.4. The workstation itself is fairly
powerful with 2x 8-core xeon's running at 3.4 GHz.

Do you have any suggestions or should I wait to test more with 2.0g?

Best,
Pavak

On Wed, Apr 8, 2020, 12:25 PM Nico Stuurman <nic...@uc...> wrote:

> Hi all,
>
> Saving data at high speed has been an issue for a while in
> Micro-Manager.  Specifically, saving data from now standard sCMOS camera
> at ~ 2k x 2k x 2 bytes at 100 fps used to work a few years ago in MM
> 1.4, but has not worked in MM 2.0 for quite some time. While sheltering
> in place, I finally had time to look into the bottle necks, remove them,
> and now can save such a data-stream on the very fast internal hard drive
> of my laptop.
>
> For those interested in the technical changes, look here:
> https://github.com/micro-manager/micro-manager/pull/735
>
> Updated code is in the latest MM 2.0-gamma nightly builds:
> https://micro-manager.org/wiki/Download_Micro-Manager_Latest_Release
>
>
> Best,
>
>
> Nico
>
>
> _______________________________________________
> micro-manager-general mailing list
> mic...@li...
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>