Thread: C-states of Intel CPU's cause firewire transfer corruptions
Brought to you by:
aeb,
bencollins
Menu
▾
▴
linux1394-user
|
From: Nizamov S. <niz...@gm...> - 2012-08-08 12:46:45
|
Hello, recently I tried to connect my dc1394 compliant FW400 camera from my older laboratory PC (dual-core P4-3200 GHz/ 2Gb RAM), running 32-bit WinXP, to a newer much more powerful workstation running Linux (64-bit Ubuntu 12.04). On old PC this CCD camera was connected using VT6307 card. I used this setup for more than the year and it showed no corruptions at all, despite the stream was topping the FW-400 speed - 12(16)-bit grayscale 1 MP @ 16 fps, 4092 bit packet size. I save data stream for hours, later process and visualize them and pretty much sure that there were no corruptions. However, after connecting the same camera using the same VT6307 card to the quad-core p5@2.8GHz, 8 Gb RAM (Linux), I noticed that there are corruptions in the acquired frames. Corruptions are in form of lines, which are not overwritten by the newer frame data and contain the older buffer data (or zeros, if buffer was clean). The length of this line is nearly of the buffer size of VT6307 (circa 2 kbytes or 1k pixels). I wrote a simple test program using dc1394 library, which would always clean out the buffer after acquiring the frame. Because my camera has an DAC offset it never outputs the pixel values less than circa 190. So it was easy to distinguish real pixel data (in the range of 190-4096) from corruptions. With the same VT6307 card and CCD camera on my newer Linux workstation, I noticed that about 1 byte in 15000 bytes would be lost (actually in my case that means that approximately 1 line of pixels would be zeroed out in every 5-10 frames). I guess that this is very small error rate and in most applications it is practically unnoticeable (e.g. in case of video, if one frame contains a very small random portion of older frame and these frames do not differ much). But in my use-case it is not acceptable and I tried to fix the issue. I tried VT6306 and VT6307 cards. I changed PCI ports, made sure that there is no other PCI devices on the bus, adjusted all possible PCI bus and device settings including exotic ones (memory/address stepping), played with giving real-time priority to my test program, higher interrupt priority to the firewire module, tried to force VT6307 to run in OHCI1.1 mode, recompile dc1394 libraries, coriander etc. I tried a fresh Linux kernel 3.5. I got the smaller rate of corruptions but never could get rid of them completely. I could not find the LSI chips with 4kb buffer therefore I bought FW card with 2 Kb buffer, but with much more robust TI chip (43AB23). Things get actually worse - if with VT6306/VT6307 I would get zeros instead of actual pixel data, now alongside such pixels there would be also a few pixels with values higher than 4096 - also corruptions, but of another kind. At the same time I was evaluating a newly bought USB2 camera (not dcam/iidc compatible, with proprietary library). It worked flawlessly on my older laboratory PC and with my notebook (the same 64-bit ubuntu 12.04 as my workstation). But it mysteriously failed with my workstation too! All the controls would work perfectly but I could get data stream out of it only at USB1 speeds - something like 1 frame per second. Looking through the vendors site I found an application note which blamed that newer Intel CPU's introduced so known C-states for power consumption reduction. They claimed that even if a single core of CPU gets into such state, not only its frequency, but the bandwidth of buses will be reduced too! Being a little bit skeptic, I nevertheless disabled these C-states. It turns out that disabling them in BIOS is not enough, linux kernel loads module intel_idle which re-enables these C-states. The solution is loading the kernel with "intel_idle.max_cstate=0" option. Check in dmesg for the "intel_idle: disabled" line to see if it worked. And it really solved both my problems - both USB2 and FW400 cameras work on my workstation with practically saturation level transfer speeds (more than 30MB/s) and without any corruptions! Hopefully it will help someone to fix their problem with firewire corruptions. With best regards, S.Nizamov |
|
From: Clemens L. <cl...@la...> - 2012-08-08 13:50:55
|
Nizamov Shawkat wrote: > On old PC this CCD camera was connected using VT6307 card. I used this > setup for more than the year and it showed no corruptions at all [...] > However, after connecting the same camera using the same VT6307 card > to the quad-core p5@2.8GHz The P5 was sold in 1993. Do you mean the i5-2300? > 8 Gb RAM (Linux), I noticed that there are corruptions in the acquired > frames. [...] > With the same VT6307 card and CCD camera on my newer Linux workstation, > I noticed that about 1 byte in 15000 bytes would be lost (actually in > my case that means that approximately 1 line of pixels would be zeroed > out in every 5-10 frames). Please note that modern Intel chipsets do not have PCI support, so many mainboards include some PCIe/PCI bridge chip -- and almost all vendors try to save a few cents by using cheap crap. Which bridge do you have (lspci)? PCI transfers data one 32-bit word at a time, while PCIe uses packets that can be much larger. How a PCI data burst is split into packets depends on the bridge implementation, especially on how much buffer space it has. PCIe cards with a PCI controller should use a suitable bridge; PCIe controllers do not have this problem. > I guess that this is very small error rate The expected error rate for a correctly designed and configured system is expected to be essentially zero. > I changed PCI ports, made sure that there is no other PCI devices on > the bus, adjusted all possible PCI bus and device settings including > exotic ones (memory/address stepping), played with giving real-time > priority to my test program, higher interrupt priority to the firewire > module, tried to force VT6307 to run in OHCI1.1 mode, recompile dc1394 > libraries, coriander etc. I tried a fresh Linux kernel 3.5. I got the > smaller rate of corruptions but never could get rid of them > completely. In theory, none of these should be able to affect your C-state problem. Were there methods that actually did have a significant effect? > I could not find the LSI chips with 4kb buffer therefore I > bought FW card with 2 Kb buffer, but with much more robust TI chip > (43AB23). PCI or PCIe? If the latter, which bridge? > I found an application note which blamed that newer Intel CPU's > introduced so known C-states for power consumption reduction. They > claimed that even if a single core of CPU gets into such state, not > only its frequency, but the bandwidth of buses will be reduced too! All modern CPUs include the memory controller, so they have to be woken up for the controller to be able to write data to memory. The deeper the sleep state, the longer it takes to wake up. (In theory, it should be possible to configure the MC to prevent it from sleeping, but I doubt that there is any BIOS option for this.) Regards, Clemens |
|
From: Nizamov S. <niz...@gm...> - 2012-08-08 15:40:21
|
2012/8/8 Clemens Ladisch <cl...@la...>:
> Nizamov Shawkat wrote:
>> On old PC this CCD camera was connected using VT6307 card. I used this
>> setup for more than the year and it showed no corruptions at all [...]
>> However, after connecting the same camera using the same VT6307 card
>> to the quad-core p5@2.8GHz
>
> The P5 was sold in 1993. Do you mean the i5-2300?
Sorry, i5-760 @ 2.80GHz. Gigabyte GA-P55A-UD3 motherboard.
>
>> 8 Gb RAM (Linux), I noticed that there are corruptions in the acquired
>> frames. [...]
>> With the same VT6307 card and CCD camera on my newer Linux workstation,
>> I noticed that about 1 byte in 15000 bytes would be lost (actually in
>> my case that means that approximately 1 line of pixels would be zeroed
>> out in every 5-10 frames).
>
> Please note that modern Intel chipsets do not have PCI support, so many
> mainboards include some PCIe/PCI bridge chip -- and almost all vendors
> try to save a few cents by using cheap crap.
>
> Which bridge do you have (lspci)?
>
It's all Intel :
00:00.0 Host bridge: Intel Corporation Core Processor DMI (rev 11)
00:03.0 PCI bridge: Intel Corporation Core Processor PCI Express Root
Port 1 (rev 11)
00:08.0 System peripheral: Intel Corporation Core Processor System
Management Registers (rev 11)
00:08.1 System peripheral: Intel Corporation Core Processor Semaphore
and Scratchpad Registers (rev 11)
00:08.2 System peripheral: Intel Corporation Core Processor System
Control and Status Registers (rev 11)
00:08.3 System peripheral: Intel Corporation Core Processor
Miscellaneous Registers (rev 11)
00:10.0 System peripheral: Intel Corporation Core Processor QPI Link (rev 11)
00:10.1 System peripheral: Intel Corporation Core Processor QPI
Routing and Protocol Registers (rev 11)
00:1a.0 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1a.1 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1a.2 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1a.7 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB2 Enhanced Host Controller (rev 06)
00:1b.0 Audio device: Intel Corporation 5 Series/3400 Series Chipset
High Definition Audio (rev 06)
00:1c.0 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI
Express Root Port 1 (rev 06)
00:1c.1 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI
Express Root Port 2 (rev 06)
00:1c.4 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI
Express Root Port 5 (rev 06)
00:1d.0 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1d.1 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1d.2 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1d.3 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB Universal Host Controller (rev 06)
00:1d.7 USB controller: Intel Corporation 5 Series/3400 Series Chipset
USB2 Enhanced Host Controller (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation 5 Series Chipset LPC Interface
Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation 5 Series/3400 Series
Chipset 6 port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation 5 Series/3400 Series Chipset SMBus
Controller (rev 06)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee
ATI Cedar PRO [Radeon HD 5450]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cedar HDMI
Audio [Radeon HD 5400/6300 Series]
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
04:00.0 Signal processing controller: National Instruments PXIe-4353
05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23
IEEE-1394a-2000 Controller (PHY/Link)
3f:00.0 Host bridge: Intel Corporation Core Processor QuickPath
Architecture Generic Non-Core Registers (rev 04)
3f:00.1 Host bridge: Intel Corporation Core Processor QuickPath
Architecture System Address Decoder (rev 04)
3f:02.0 Host bridge: Intel Corporation Core Processor QPI Link 0 (rev 04)
3f:02.1 Host bridge: Intel Corporation Core Processor QPI Physical 0 (rev 04)
3f:03.0 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller (rev 04)
3f:03.1 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Target Address Decoder (rev 04)
3f:03.4 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Test Registers (rev 04)
3f:04.0 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 0 Control Registers (rev 04)
3f:04.1 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 0 Address Registers (rev 04)
3f:04.2 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 0 Rank Registers (rev 04)
3f:04.3 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 0 Thermal Control Registers (rev 04)
3f:05.0 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 1 Control Registers (rev 04)
3f:05.1 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 1 Address Registers (rev 04)
3f:05.2 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 1 Rank Registers (rev 04)
3f:05.3 Host bridge: Intel Corporation Core Processor Integrated
Memory Controller Channel 1 Thermal Control Registers (rev 04)
>> I guess that this is very small error rate
>
> The expected error rate for a correctly designed and configured system
> is expected to be essentially zero.
>
I mean that in general such errors are not noticed by the user. Not
that it is acceptable by design.
I did not noticed it too in coriander when checking it as a generic
video camera.
But on absolutely static and generally homogenous image with
illumination conditions changing each frame, inter-frame buffer
corruptions were revealed immediately.
>> I changed PCI ports, made sure that there is no other PCI devices on
>> the bus, adjusted all possible PCI bus and device settings including
>> exotic ones (memory/address stepping), played with giving real-time
>> priority to my test program, higher interrupt priority to the firewire
>> module, tried to force VT6307 to run in OHCI1.1 mode, recompile dc1394
>> libraries, coriander etc. I tried a fresh Linux kernel 3.5. I got the
>> smaller rate of corruptions but never could get rid of them
>> completely.
>
> In theory, none of these should be able to affect your C-state problem.
> Were there methods that actually did have a significant effect?
>
There were no significant or obviously notable changes.
Slight changes that I observed most probably should be attributed to
the background processes, which wake up CPU cores more or less
frequently. I have a bunch of desktop and development applications
open, however top and iotop did not show any heavyweight processes
during acquisition.
>> I could not find the LSI chips with 4kb buffer therefore I
>> bought FW card with 2 Kb buffer, but with much more robust TI chip
>> (43AB23).
>
> PCI or PCIe? If the latter, which bridge?
>
It is PCI.
05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23
IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 [OHCI])
Flags: bus master, medium devsel, latency 32, IRQ 18
Memory at fbbff000 (32-bit, non-prefetchable) [size=2K]
Memory at fbbf8000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: firewire_ohci
Kernel modules: firewire-ohci
As far as I understand, it is connected via this bridge and it is the
only PCI device connected to this bridge:
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
(prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=32
>> I found an application note which blamed that newer Intel CPU's
>> introduced so known C-states for power consumption reduction. They
>> claimed that even if a single core of CPU gets into such state, not
>> only its frequency, but the bandwidth of buses will be reduced too!
>
> All modern CPUs include the memory controller, so they have to be woken
> up for the controller to be able to write data to memory. The deeper
> the sleep state, the longer it takes to wake up. (In theory, it should
> be possible to configure the MC to prevent it from sleeping, but I doubt
> that there is any BIOS option for this.)
>
>
It explains a lot, thank you.
With best regards,
S.Nizamov
|
|
From: Clemens L. <cl...@la...> - 2012-08-08 16:50:31
|
Nizamov Shawkat wrote: > i5-760 @ 2.80GHz. <http://download.intel.com/design/processor/datashts/322164.pdf> mentions in section 2.1.6 a "fast exit" memory controller power-down mode, which is to be used when there are two DIMMs per channel. Section 4.3 mentions an "active power down" mode, but I haven't found any register to configure this in <http://download.intel.com/design/processor/datashts/322165.pdf>. Regards, Clemens |
|
From: Damien D. <dd...@do...> - 2012-08-08 14:58:05
|
Shawkat, On Wed, 2012-08-08 at 14:46 +0200, Nizamov Shawkat wrote: [snip] > Looking through the vendors site I found an application note > which blamed that newer Intel CPU's introduced so known C-states for > power consumption reduction. They claimed that even if a single core > of CPU gets into such state, not only its frequency, but the bandwidth > of buses will be reduced too! Being a little bit skeptic, I > nevertheless disabled these C-states. It turns out that disabling them > in BIOS is not enough, linux kernel loads module intel_idle which > re-enables these C-states. The solution is loading the kernel with > "intel_idle.max_cstate=0" option. Check in dmesg for the "intel_idle: > disabled" line to see if it worked. > > And it really solved both my problems - both USB2 and FW400 cameras > work on my workstation with practically saturation level transfer > speeds (more than 30MB/s) and without any corruptions! > > Hopefully it will help someone to fix their problem with firewire corruptions. The corruption of 1394 data (or at least IIDC data) when entering C1E and possibly other sleep states is a known issue. For 5 or 6 years, the very first thing I do when I buy a computer is disable all the C-states in the BIOS! But I was not aware that Linux could re-enable them; this is probably a feature of newer kernels. I will definitely keep your kernel option in mind in the future... Thanks for sharing :) -- Damien 高原 Douxchamps http://damien.douxchamps.net/ |
Thanks for helping keep SourceForge clean.
X