C-states of Intel CPU's cause firewire transfer corruptions
Brought to you by:
aeb,
bencollins
From: Nizamov S. <niz...@gm...> - 2012-08-08 12:46:45
|
Hello, recently I tried to connect my dc1394 compliant FW400 camera from my older laboratory PC (dual-core P4-3200 GHz/ 2Gb RAM), running 32-bit WinXP, to a newer much more powerful workstation running Linux (64-bit Ubuntu 12.04). On old PC this CCD camera was connected using VT6307 card. I used this setup for more than the year and it showed no corruptions at all, despite the stream was topping the FW-400 speed - 12(16)-bit grayscale 1 MP @ 16 fps, 4092 bit packet size. I save data stream for hours, later process and visualize them and pretty much sure that there were no corruptions. However, after connecting the same camera using the same VT6307 card to the quad-core p5@2.8GHz, 8 Gb RAM (Linux), I noticed that there are corruptions in the acquired frames. Corruptions are in form of lines, which are not overwritten by the newer frame data and contain the older buffer data (or zeros, if buffer was clean). The length of this line is nearly of the buffer size of VT6307 (circa 2 kbytes or 1k pixels). I wrote a simple test program using dc1394 library, which would always clean out the buffer after acquiring the frame. Because my camera has an DAC offset it never outputs the pixel values less than circa 190. So it was easy to distinguish real pixel data (in the range of 190-4096) from corruptions. With the same VT6307 card and CCD camera on my newer Linux workstation, I noticed that about 1 byte in 15000 bytes would be lost (actually in my case that means that approximately 1 line of pixels would be zeroed out in every 5-10 frames). I guess that this is very small error rate and in most applications it is practically unnoticeable (e.g. in case of video, if one frame contains a very small random portion of older frame and these frames do not differ much). But in my use-case it is not acceptable and I tried to fix the issue. I tried VT6306 and VT6307 cards. I changed PCI ports, made sure that there is no other PCI devices on the bus, adjusted all possible PCI bus and device settings including exotic ones (memory/address stepping), played with giving real-time priority to my test program, higher interrupt priority to the firewire module, tried to force VT6307 to run in OHCI1.1 mode, recompile dc1394 libraries, coriander etc. I tried a fresh Linux kernel 3.5. I got the smaller rate of corruptions but never could get rid of them completely. I could not find the LSI chips with 4kb buffer therefore I bought FW card with 2 Kb buffer, but with much more robust TI chip (43AB23). Things get actually worse - if with VT6306/VT6307 I would get zeros instead of actual pixel data, now alongside such pixels there would be also a few pixels with values higher than 4096 - also corruptions, but of another kind. At the same time I was evaluating a newly bought USB2 camera (not dcam/iidc compatible, with proprietary library). It worked flawlessly on my older laboratory PC and with my notebook (the same 64-bit ubuntu 12.04 as my workstation). But it mysteriously failed with my workstation too! All the controls would work perfectly but I could get data stream out of it only at USB1 speeds - something like 1 frame per second. Looking through the vendors site I found an application note which blamed that newer Intel CPU's introduced so known C-states for power consumption reduction. They claimed that even if a single core of CPU gets into such state, not only its frequency, but the bandwidth of buses will be reduced too! Being a little bit skeptic, I nevertheless disabled these C-states. It turns out that disabling them in BIOS is not enough, linux kernel loads module intel_idle which re-enables these C-states. The solution is loading the kernel with "intel_idle.max_cstate=0" option. Check in dmesg for the "intel_idle: disabled" line to see if it worked. And it really solved both my problems - both USB2 and FW400 cameras work on my workstation with practically saturation level transfer speeds (more than 30MB/s) and without any corruptions! Hopefully it will help someone to fix their problem with firewire corruptions. With best regards, S.Nizamov |