Re: Problems with Texas Instruments TSB82AA2 chipset and cameras on both new and old stacks
Brought to you by:
aeb,
bencollins
From: Stefan R. <st...@s5...> - 2011-09-19 19:38:33
|
(Full quote because the message did not go through to the list.) On Sep 19 Joan Pau Beltran wrote: > Hi again, > > Here I can provide some new information about the mentioned problem. > Sorry for the long idle time but the hardware at my company was not > available for me again until now. > The PC104-Plus firewire card has been replaced by this other model: > CM17208 by RTD. > According to the output of lspci (attached) and the manual of the card, > it comes with the same chips : TSB82AA2 Link Layer and TSB81BA3 > Transceiver/Arbiter. > > We are using two Pointgrey Bumblebee2 stereo cameras connected to that > card (in Format7 mode and selecting the apropiate packet size to not > blow up the bus bandwith). > The problem is that the cameras work for a while (sometimes hours, > sometime minutes) but eventually they hang (no error messages from > libdc1394). > > To set an scenario, we are still using Ubuntu 10.04 x86-64, but with > backported kernel 2.6.38-10. > We remove the new stack modules and load it again with debug option: > > user@fugu-c:~$ sudo modprobe -r -v firewire_ohci > > rmmod > > /lib/modules/2.6.38-10-generic/kernel/drivers/firewire/firewire-ohci.ko > > rmmod > > /lib/modules/2.6.38-10-generic/kernel/drivers/firewire/firewire-core.ko > > rmmod /lib/modules/2.6.38-10-generic/kernel/lib/crc-itu-t.ko > > user@fugu-c:~$ sudo modprobe -v firewire_ohci debug=15 > > insmod /lib/modules/2.6.38-10-generic/kernel/lib/crc-itu-t.ko > > insmod > > /lib/modules/2.6.38-10-generic/kernel/drivers/firewire/firewire-core.ko > > insmod > > /lib/modules/2.6.38-10-generic/kernel/drivers/firewire/firewire-ohci.ko debug=15 > As shown in the attached file, the correct drivers seem to be loaded. For the list: ------------------------- lsmod.txt from Joan ------------------------- Module Size Used by firewire_ohci 40012 0 firewire_core 62534 9 firewire_ohci crc_itu_t 12707 1 firewire_core cp210x 21776 0 binfmt_misc 17498 1 ppdev 17180 0 i915 518574 2 drm_kms_helper 42054 1 i915 usbserial 42893 1 cp210x drm 227552 3 i915,drm_kms_helper r8712u 307506 0 psmouse 73514 0 i2c_i801 17449 1 serio_raw 13208 0 i2c_algo_bit 13368 1 i915 video 19280 1 i915 hwmon_vid 12746 0 coretemp 13519 0 i2c_dev 13355 2 lp 17789 0 parport 46360 2 ppdev,lp e1000e 158424 0 ----------------------------------------------------------------------- ------------------------- lspci.txt from Joan ------------------------- 00:00.0 Host bridge: Intel Corporation N10 Family DMI Bridge 00:02.0 VGA compatible controller: Intel Corporation N10 Family Integrated Graphics Controller 00:02.1 Display controller: Intel Corporation N10 Family Integrated Graphics Controller 00:19.0 Ethernet controller: Intel Corporation 82567V-3 Gigabit Network Connection (rev 04) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 04) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 04) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 04) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f4) 00:1f.0 ISA bridge: Intel Corporation 82801HEM (ICH8M) LPC Interface Controller (rev 04) 00:1f.2 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA IDE Controller (rev 04) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 04) 01:04.0 FireWire (IEEE 1394): Texas Instruments TSB82AA2 IEEE-1394b Link Layer Controller (rev 01) ----------------------------------------------------------------------- Module list and lspci output are fine. > The other file attached is the kernel log. Some parts seem to be > misaligned, I don't know why, but they are not important. For the list: I uploaded Joan's kern.log to http://paste.ubuntu.com/693238/. It is 5 MB big which is why Joan's mail was retained by the list server. > Everything is working fine, images are read from the cameras at the > correct frame rate, and the log only contain bus reset messages (what do > they mean?). But approximately at second 1421 the cameras get stacked, > and new messages appear in the log. I can not interpret their meaning, > but could they point out to a software problem? Or to a hardware one? > The hang is much more frequent when the cylinder that contains the > cameras is closed, which may be related with the heat problems mentioned > in the previous posts. > > What do you think? > > Thanks again for your interest and attention. In the first part of your log, everything looks perfectly fine. From timestamp 1418.424221 to 1431.015034, you get a storm of bus reset events. None of these bus resets is concluded with a so-called self-ID- complete event, which would be when the nodes on the bus identified themselves to each other after the bus reset. Normal FireWire traffic can only happen after a self-ID-complete event and before the next bus reset event. At timestamp 1431.015048, the first self-ID-complete event happens, i.e. a successful conclusion of a bus reset. After that, the kernel and your application are apparently able to use the cameras again. Bus resets _normally_ happen: 1. If you plug a FireWire device in to or out of the FireWire bus or power a FireWire device up or down, 2. if bus management software or node management firmware or special application software requests a bus reset as a means to announce a change of node configuration to all peers. Bus resets _abnormally_ happen: 3. If rogue software requests a bus reset for no good reason, 4. if a physical interface chip malfunctions due to hardware errata, 5. if a physical interface chip of a device or a cable or a combination of chips and cable works too far off the standardized electrical parameters of the FireWire physical layer. The storm of bus resets from timestamp 1418.424221 onwards is of course of one of the abnormal types. On 3.: firewire-core since kernel 2.6.36 contains safeguards which prevent any software that runs on the Linux node to cause a storm of bus resets. On 4.: Texas Instruments TSB81BA3 (all revisions up to revision C inclusive) is a seriously buggy PHY which tends to fall into bus reset storms at the drop of a hat. However, this is only known to happen in case of pure 1394b "beta mode" buses: http://www.ti.com/litv/pdf/sllz015c Your cameras are 1394a devices, therefore the bus operates in "alpha mode" a.k.a. data-strobe mode. The above mentioned TSB81BA3 errata are *not* known to happen in alpha mode; instead, as far as documented by TI and as far as I have observed myself, having one or more 1394a node on the bus keeps TSB81BA3 working correctly. On 5.: Various circumstances can contribute to such malfunctions, alone or in combination: - Plainly damaged hardware of course, - badly constructed or miswired or too long cables, - badly laid out circuit boards, - too weak or electrically too noisy power supplies, - badly selected or overheating clocking crystals, - overheating physical interface chips, - electromagnetic interference from within one of the FireWire nodes or from external sources. My suggestion to you, if you haven't done so already: Check whether you have high-quality cables, whether the environmental conditions in your lab (EMI, temperature, humidity, mechanical vibrations, radiation,...) are in line with what is specified for your embedded PC and for the Bumblebee2 cameras or what is generally assumed for office use, whether power supply is adequate and stable, whether your embedded PC is properly shielded against EMI (inbound and outbound). If you are more or less certain that your equipment and lab are OK, you should approach Point Grey or/and the vendors of the components of your embedded PC with the issue. If you have adverse conditions in your lab (or wherever this equipment is deployed) which you cannot improve, ask Point Grey and the other component vendors as well whether they have recommendations for counter measures. ----------------------------------------------------------------------- Oh wait, I now looked up the prior postings of this thread in my mail archive. On 07 Dec 2010 you wrote that this equipment is built into a vehicle. So that pretty much means that the last paragraph applies, unless if the issue also happens if you have cameras and PC taken out of the vehicle and are testing it under good conditions in your office. The 1394 Trade Association has a working group which deals with 1394 based in-vehicle networks. There tend to be special 1394b-2002/ 1394-2008 beta mode based cables and connectors involved which are quite different from the traditional 1394-1995/ 1394a-2000 cables and plugs. -- Stefan Richter -=====-==-== =--= =--== http://arcgraph.de/sr/ |