funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

Hello.

	I have built a home-brew receiver and have been using it for about a year now. I have never had any problems with it until now. I upgraded my aging P-II 266 to an Athlon 1700+ (err... 1466Mhz) with a KT266A board. Ever since this upgrade, it has behaved funny.
	Lirc works fine at first, but after some time (of an app using it) it will stop reporting button presses. irw will not show anything is being pressed. As soon as I close all programs that use lirc, I can start any lirc app and it will work again (for a limited time). It's kind of hard to believe that my hardware upgrade would cause this seamingly unrelated behavior. If I run an app, but not use lirc, it will still crap-out after some time. When I press buttons more frequently (playing a DVD), it craps-out sooner.
	I thought that upgrading my lirc version would help (it didn't). I then did a new Linux From Scratch install (was going to happen anyways, I didn't do it to solve this problem ;), and a new install of lirc, but same problem.

Any pointers? Thank-you for your time.

Dan Eriksen

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

Hello.

	I would appreciate if anyone could tell me why the following might hang? This line is in the waitfordata routine.

ret=select(maxfd+1,&fds,NULL,NULL,NULL);

	I am thinking this is where lircd talks to the kernel module? If so, this would indicate I am having problems in the kernel module and not lircd. As you can surely tell, I am not really a coder, but I have done some.

	Lircd will still remove clients and if all clients detach, it returns to return to functional operation again. As explained below, after a (seemingly) random period of time (either using or not using) lirc, it hangs on this line.

Thanks,
Dan Eriksen

On Wed, 6 Mar 2002 11:36:01 -0500
Dan Eriksen <eriksen@...> wrote:

> 	I have built a home-brew receiver and have been using it for about a year now. I have never had any problems with it until now. I upgraded my aging P-II 266 to an Athlon 1700+ (err... 1466Mhz) with a KT266A board. Ever since this upgrade, it has behaved funny.
> 	Lirc works fine at first, but after some time (of an app using it) it will stop reporting button presses. irw will not show anything is being pressed. As soon as I close all programs that use lirc, I can start any lirc app and it will work again (for a limited time). It's kind of hard to believe that my hardware upgrade would cause this seamingly unrelated behavior. If I run an app, but not use lirc, it will still crap-out after some time. When I press buttons more frequently (playing a DVD), it craps-out sooner.
> 	I thought that upgrading my lirc version would help (it didn't). I then did a new Linux From Scratch install (was going to happen anyways, I didn't do it to solve this problem ;), and a new install of lirc, but same problem.
> 
> Any pointers? Thank-you for your time.
> 

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

Hi Dan,

You have the right idea.  select() is supposed to wait until some new 
data or event
takes place.  If you look in the lirc_serial.c code you will find a 
related select()
routine.  In this routine, it just looks to see if new data is in the queue.
Data is collected in the queue when an interrupt occurs.

You didn't say if you are using the homebrew serial-port hardware.  Lets 
assume you are.
Notice that when lircd gets a client, it causes the lirc_serial.c driver 
to start
using the interrupts.  You can see this by:
cat /proc/interrupts
You should see a line with lirc_serial in it, probably IRQ 4 registered and
the count of how many interrupts have come in.  This count of interrupts
you can use to troubleshoot.

Notice when everthing is working OK, that this count increases.
Does it increase when things are not working?

Notice how the /proc/interrupts entry goes away when all the lircd
clients are removed.  So lircd is causing the driver to shut down its
hardware.  When a client returns, the driver re-initializes the interrupt
and serial hardware.  This may account for the behavior you are seeing.

You may want to look over your interrupt settings.  Is any other
device sharing this interrupt?  Any odd settings relating to the serial port
or IRQ's in the BIOS setup?  You might want to try a different serial
port if you have one.  Does /proc/interrupts for lirc_serial increment
during the error condition?

If that fails, there may be some verbose settings which could be turned
on to troubleshoot further.


Karl.

Dan Eriksen wrote:

>Hello.
>
>	I would appreciate if anyone could tell me why the following might hang? This line is in the waitfordata routine.
>
>ret=select(maxfd+1,&fds,NULL,NULL,NULL);
>
>	I am thinking this is where lircd talks to the kernel module? If so, this would indicate I am having problems in the kernel module and not lircd. As you can surely tell, I am not really a coder, but I have done some.
>
>	Lircd will still remove clients and if all clients detach, it returns to return to functional operation again. As explained below, after a (seemingly) random period of time (either using or not using) lirc, it hangs on this line.
>
>Thanks,
>Dan Eriksen
>
>On Wed, 6 Mar 2002 11:36:01 -0500
>Dan Eriksen <eriksen@...> wrote:
>
>>	I have built a home-brew receiver and have been using it for about a year now. I have never had any problems with it until now. I upgraded my aging P-II 266 to an Athlon 1700+ (err... 1466Mhz) with a KT266A board. Ever since this upgrade, it has behaved funny.
>>	Lirc works fine at first, but after some time (of an app using it) it will stop reporting button presses. irw will not show anything is being pressed. As soon as I close all programs that use lirc, I can start any lirc app and it will work again (for a limited time). It's kind of hard to believe that my hardware upgrade would cause this seamingly unrelated behavior. If I run an app, but not use lirc, it will still crap-out after some time. When I press buttons more frequently (playing a DVD), it craps-out sooner.
>>	I thought that upgrading my lirc version would help (it didn't). I then did a new Linux From Scratch install (was going to happen anyways, I didn't do it to solve this problem ;), and a new install of lirc, but same problem.
>>
>>Any pointers? Thank-you for your time.
>>
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

	Thank-you for all the helpful information Karl Bongers.


> You didn't say if you are using the homebrew serial-port hardware. 

	Yes. Used schematic found on lirc.org.

> 
> Notice when everthing is working OK, that this count increases.
> Does it increase when things are not working?

	No, it stops increasing.

> You may want to look over your interrupt settings.  Is any other
> device sharing this interrupt?  Any odd settings relating to the serial port
> or IRQ's in the BIOS setup?  You might want to try a different serial
> port if you have one.

	ttyS00 at 0x03f8 (irq = 4) is a 16550A
	ttyS01 at 0x02f8 (irq = 3) is a 16550A
I use ttyS0 after using "setserial /dev/ttyS0 uart none". But I will have to try ttyS1 eventually. I can't see anything else using irq 4, but maybe something uses it at the times when it hangs (I wouldn't think so).

	What would you advise I do next?

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

Hi Dan,

Try disconnecting and reconnecting the circuit when the problem
comes up.  Does it work afterwards?  The idea is that
maybe the circuit is just barely working and maybe the power is drooping
to the circuit or the circuit is latching up.  Try reconnecting the circuit
when it is working, does it still work(it should)?

Keep in mind that the homebrew circuit draws its power from 1 serial
port pin, basically the circuit is a cheap and dirty hack and is not 
guarenteed
to work based on RS232 specs.  You might try adding that small .1uf cap
that the circuit suggests if you did not do so.

Try the other serial port.
I'll bet it is hardware related.
Only a small chance it is a driver bug,
 the driver is very simple.

Karl.

Dan Eriksen wrote:

>	Thank-you for all the helpful information Karl Bongers.
>
>
>>You didn't say if you are using the homebrew serial-port hardware. 
>>
>
>	Yes. Used schematic found on lirc.org.
>
>>Notice when everthing is working OK, that this count increases.
>>Does it increase when things are not working?
>>
>
>	No, it stops increasing.
>
>>You may want to look over your interrupt settings.  Is any other
>>device sharing this interrupt?  Any odd settings relating to the serial port
>>or IRQ's in the BIOS setup?  You might want to try a different serial
>>port if you have one.
>>
>
>	ttyS00 at 0x03f8 (irq = 4) is a 16550A
>	ttyS01 at 0x02f8 (irq = 3) is a 16550A
>I use ttyS0 after using "setserial /dev/ttyS0 uart none". But I will have to try ttyS1 eventually. I can't see anything else using irq 4, but maybe something uses it at the times when it hangs (I wouldn't think so).
>
>	What would you advise I do next?
>
>Dan
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Thu, 21 Mar 2002 23:38:19 -0600
Karl Bongers <karl@...> wrote:

> Try the other serial port.
> I'll bet it is hardware related.
> Only a small chance it is a driver bug,
>  the driver is very simple.
> 
	Yes. And the fact that this only started when I upgraded to a new motherboard might indicate that the serial port on this board isn't able to supply the extra power. Anyway, I'll try the things you suggested. It takes several hours before it stops working (sometimes will go a whole day). The kicker is that this board has IrDA connectors, but I cannot seem to get hold of a module for it.

Thanks
	Dan

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Thu, 21 Mar 2002 23:38:19 -0600
Karl Bongers <karl@...> wrote:

> Try disconnecting and reconnecting the circuit when the problem
> comes up.  Does it work afterwards?  The idea is that
> maybe the circuit is just barely working and maybe the power is drooping
> to the circuit or the circuit is latching up.  Try reconnecting the circuit
> when it is working, does it still work(it should)?

	When the driver is working, I can disconnect and reconnect the circuit and it continues to work. When it is having problems, disconnecting and reconnecting has no effect. It continues to not work.

>  You might try adding that small .1uf cap
> that the circuit suggests if you did not do so.
> 
	Where does the circuit suggest that? It only mentions that C1 (4.7uF) is optional, but recommended. I included it.

	I guess the other (ugly) solution would be to give it an external power supply. That would (likely) clear up whether it is hardware or software (but as you said, likely hardware).

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

Dan,

I did mean the 4.7uf cap in the circuit(not .1uf as I said), I just
got sloppy and took a guess from memory, sorry.

If you had a voltmeter or scope, you might want to take
a few readings when it is broken.

Also as a test, you could try forcing some interrupts
to occur by pulling off the circuit, and touching some wire
or 1k resistor across pin 1(dcd)
and pin 7(rts).  Or try across pin 1 and pin 4(dtr).
Just try *not* to rub your shoes on the carpet before you
do this, static is the one thing I've known to blow
out serial port hardware.

When I do this across 1 and 7, I see the /proc/interrupts count go
up by a few 100 counts.  Connecting 1 and 4 does not
do anything.  This is expected, as DTR is the output
turned on(positive), while RTS is off(negative).  And
they bias the RS232 inputs off slightly.

This is with lircd and irw running.  My remote continues
to work before and after I remove my home brew serial
circuit for this test.

If we get interrupts with the wire, then it's probably a hardware
problem.  If we can't get interrupts, then this might indicate
a  software problem.  We are pretty sure that re-initializing the
serial port in the driver corrects the problem, so if you wanted
to pursue it we could add some code to the driver to selectively
re-initialize things to try and narrow it down further...

Karl.

Dan Eriksen wrote:

>On Thu, 21 Mar 2002 23:38:19 -0600
>Karl Bongers <karl@...> wrote:
>
>>Try disconnecting and reconnecting the circuit when the problem
>>comes up.  Does it work afterwards?  The idea is that
>>maybe the circuit is just barely working and maybe the power is drooping
>>to the circuit or the circuit is latching up.  Try reconnecting the circuit
>>when it is working, does it still work(it should)?
>>
>
>	When the driver is working, I can disconnect and reconnect the circuit and it continues to work. When it is having problems, disconnecting and reconnecting has no effect. It continues to not work.
>
>> You might try adding that small .1uf cap
>>that the circuit suggests if you did not do so.
>>
>	Where does the circuit suggest that? It only mentions that C1 (4.7uF) is optional, but recommended. I included it.
>
>	I guess the other (ugly) solution would be to give it an external power supply. That would (likely) clear up whether it is hardware or software (but as you said, likely hardware).
>
>Dan
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Sat, 23 Mar 2002 15:33:16 -0600
Karl Bongers <karl@...> wrote:

> Just try *not* to rub your shoes on the carpet before you
> do this, static is the one thing I've known to blow
> out serial port hardware.

	Thanks for the reminder, lots of static in this place.

> This is with lircd and irw running.  My remote continues
> to work before and after I remove my home brew serial
> circuit for this test.

	I get interrupts as well when I touch 1 and 7. None for 1 and 4. I can then reconnect my receiver and it functions still (same as you).
	Interestingly, when it has begun acting up, terminating 1 and 7 with a 1k resister does nothing. Same with 1 and 4. I then disconnect all lirc clients, connect irw and it generates interrupts again. Hooking-up my receiver will show it working properly.

	Ouch, would have been easier if it was definately a hardware problem, eh?
	Still could be hardware, but this problem started right about when I upgraded my motherboard/CPU.

So, are you able to help me hack kernel drivers :)

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

Dan,

Heres some code which will selectively reset
serial port hardware.  Put this bit of code at the start of
lirc_write() function.   You may need to reconfigure
the lirc source package if it does not have sending turned
on(not sure about this.)

So when it hangs, send out a write with:
rc SEND_ONCE <remote> <key>
(this is from the xrc download for sending).
This should log in /var/log/messages so you
can keep track of what it is doing.   Hopefully
one of these steps will start it working and tell
us more..  (My remote continues to work after
each step.)


    static int reset_level=0;
    int result;
    switch(reset_level) {
        case 0:
            printk(KERN_ERR  LIRC_DRIVER_NAME  ": Set power-pin(%d)\n", 
reset_level);
            /* Set line for power source */
            soutp(UART_MCR, hardware[type].off);
        break;
        case 1:
            /* Clear registers. */
            sinp(UART_IIR);
            printk(KERN_ERR  LIRC_DRIVER_NAME  ": Clear Ints(%d)\n", 
reset_level);
        break;
        case 2:
            /* Clear registers. */
            sinp(UART_LSR);
            sinp(UART_RX);
            sinp(UART_IIR);
            sinp(UART_MSR);
            /* Set line for power source */
            soutp(UART_MCR, hardware[type].off);
            printk(KERN_ERR  LIRC_DRIVER_NAME  ": Clear Regs(%d)\n", 
reset_level);
        break;
        case 3:
            printk(KERN_ERR  LIRC_DRIVER_NAME  ": Reset All(%d)\n", 
reset_level);
            /* Set DLAB 0. */
            soutp(UART_LCR, sinp(UART_LCR) & (~UART_LCR_DLAB));
            /* First of all, disable all interrupts */
            soutp(UART_IER, sinp(UART_IER)&
                  
(~(UART_IER_MSI|UART_IER_RLSI|UART_IER_THRI|UART_IER_RDI)));
            free_irq(irq, NULL);

            
result=request_irq(irq,irq_handler,SA_INTERRUPT,LIRC_DRIVER_NAME,NULL);
            /* Set DLAB 0. */
            soutp(UART_LCR, sinp(UART_LCR) & (~UART_LCR_DLAB));
            soutp(UART_IER, sinp(UART_IER)|UART_IER_MSI);
        break;
        default:
          reset_level = 0;
          return(n);
        break;
    }
    ++reset_level;
    return(n);



Dan Eriksen wrote:

>On Sat, 23 Mar 2002 15:33:16 -0600
>Karl Bongers <karl@...> wrote:
>
>>Just try *not* to rub your shoes on the carpet before you
>>do this, static is the one thing I've known to blow
>>out serial port hardware.
>>
>
>	Thanks for the reminder, lots of static in this place.
>
>>This is with lircd and irw running.  My remote continues
>>to work before and after I remove my home brew serial
>>circuit for this test.
>>
>
>	I get interrupts as well when I touch 1 and 7. None for 1 and 4. I can then reconnect my receiver and it functions still (same as you).
>	Interestingly, when it has begun acting up, terminating 1 and 7 with a 1k resister does nothing. Same with 1 and 4. I then disconnect all lirc clients, connect irw and it generates interrupts again. Hooking-up my receiver will show it working properly.
>
>	Ouch, would have been easier if it was definately a hardware problem, eh?
>	Still could be hardware, but this problem started right about when I upgraded my motherboard/CPU.
>
>So, are you able to help me hack kernel drivers :)
>
>Dan
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Sun, 24 Mar 2002 01:59:39 -0600
Karl Bongers <karl@...> wrote:

>     static int reset_level=0;

	I assume you wanted this placed outside of lirc_write?
I am not much of a coder, but your code suggests that you want to send a command to toggle the next level of reset, until one of them works.

	After some fighting with xrc (I don't have/want the qt libs), and a driver recompile, I can "rc" the driver into displaying each of the reset_levels. Now I just have to wait...

	I also tried another serial port (still same motherboard though), but it was the same.

	Should we be taking this off-list? I don't really care, but others might.

Dan

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

	Well, here are the results.
When it began acting up, I ran irw and followed the logs (sys.log and lircd).
I ran "rc" and tried buttons on the remote, (lather, rinse, repeat)
When It executed case 2 "Clear Regs(2)" it began working again.

Is there anything I should've done differently?
Or is this useful?

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

I think these are good results, we were able to "fix" the problem with
a few pokes at the hardware.  I am guessing that the registers that
latch the interrupt are not getting properly reset.  Normally when an
interrupt happens it causes the interrupt routine to run, and this routine
then reads these latches(IIR, MSR) which resets them to allow for the
next interrupt.

We don't know why these are getting in this state, it could be a lot of
reasons everything from bad UART hardware to our lirc_serial driver
not reseting these in the right sequence.

What I'd like to do is compare how the lirc_serial driver resets these
interrupt registers with other drivers.  Make sure the sequence is proper
and no race conditions exist.  We could probably build a work around
to your problem with what we know, but it would be nice to try and
figure out what the source of the problem is.

I made another set of code that would allow you to poke and peek
at the serial port hardware from an application(rather than driver 
hacking).  See
http://www.turbobit.com/software/codeman-0.0.2.tgz
Its like the old DOS debug program which allows you to do:

i 3fe
or :
o 3f8 55

(is this function built into GDB or some other common tool?  Anyone?)

You need to run it as root.  Run make to compile it.  run as codeman -c3
I could send you some serial port register doc and some suggestions
on what to try.  My guess is that if you do a:
i 3fa   ; IIR, read interrupt identification reg
i 3fe   ; MSR, read modem status reg
That should get your interrupts running again I think.  These are the 
latches
that need to be read before another interrupt can occur.

Or change our driver test where this is the first case:
           sinp(UART_IIR);
           sinp(UART_MSR);
and see if that does the trick.

We can take some of this off-list too If you like...


Karl.

--------------------------

Dan Eriksen wrote:

>	Well, here are the results.
>When it began acting up, I ran irw and followed the logs (sys.log and lircd).
>I ran "rc" and tried buttons on the remote, (lather, rinse, repeat)
>When It executed case 2 "Clear Regs(2)" it began working again.
>
>Is there anything I should've done differently?
>Or is this useful?
>
>Dan
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

> We can take some of this off-list too If you like...

	I don't really care. Does anyone else care?
 
> We don't know why these are getting in this state, it could be a lot of
> reasons everything from bad UART hardware to our lirc_serial driver
> not reseting these in the right sequence.

	Another interesting thing. I have discovered that it is "crapping" out faster now (every 5-25 minutes, as opposed to 3-8hours). I think it is related to the change I made from having sending disabled to now having it enabled (I did have to re-configure lirc for sending). I will play around with it a bit more to narrow it's source.

	I will be away for a couple of days, but I will try some more on Tuesday. Like you said, we could fix "my" problem, but finding the source of the problem is much better.

Dan

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Sun, 24 Mar 2002 14:40:33 -0600
Karl Bongers <karl@...> wrote:

> Or change our driver test where this is the first case:
>            sinp(UART_IIR);
>            sinp(UART_MSR);
> and see if that does the trick.
> 
	The above works. I couldn't narrow down the source of the more frequent "crap-outs". I was wondering if it happens when I do the above "fix", but cannot confirm.
	Sorry it took me so long to reply. I have codeman 0.2, but just hacked the source for the above.
	What next?

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

I don't know of a good way to troubleshoot this further.  My speculation
would be that either the hardware is broken or some kernel
level code is broke.  The lirc_serial drivers interrupt handling
looks good to me.

Karl.


Dan Eriksen wrote:

>On Sun, 24 Mar 2002 14:40:33 -0600
>Karl Bongers <karl@...> wrote:
>
>>Or change our driver test where this is the first case:
>>           sinp(UART_IIR);
>>           sinp(UART_MSR);
>>and see if that does the trick.
>>
>	The above works. I couldn't narrow down the source of the more frequent "crap-outs". I was wondering if it happens when I do the above "fix", but cannot confirm.
>	Sorry it took me so long to reply. I have codeman 0.2, but just hacked the source for the above.
>	What next?
>
>Dan
>
>

Re: funny lirc behavior

[Dan Eriksen <eriksen@ca...>]

On Thu, 28 Mar 2002 23:38:49 -0600
Karl Bongers <karl@...> wrote:

> 
> I don't know of a good way to troubleshoot this further.  My speculation
> would be that either the hardware is broken or some kernel
> level code is broke.  The lirc_serial drivers interrupt handling
> looks good to me.
> 

	There is no guarentee that this will narrow it down, but I think I will build another receiver from scratch and see if it exibits the same behaviour.
	Thank-you Karl for all your help and explanations. It was very helpful. Are you interested in hearing about my future results?

Dan

Re: funny lirc behavior

[Karl Bongers <karl@tu...>]

Dan,

I'd be interested to know if you discover anything new.  Some things
to try might include using a different kernel, turning off various 
kernel level
drivers(USB, ...), toying with some of the BIOS settings(bus speed,..).
Try compiling your serial driver as a module.
Maybe your motherboard has broken hardware(try swapping motherboards,
try using a PCI card with serial ports..)
These are just wild guesses, and probably lead to just a lot of frustration.

If we could automate or quicken the bug reproduction this would be useful.
We could make a program to toggle an rs232 output and see if we can invoke
the error quicker.  So run a wire from an output
pin on COM2 over to the DCD input pin on COM1, then toggle the COM2
output pin like mad and watch /proc/interrupts.  Let me know if you want
to try this and I'll whip something like this up.

Karl.

Dan Eriksen wrote:

>On Thu, 28 Mar 2002 23:38:49 -0600
>Karl Bongers <karl@...> wrote:
>
>>I don't know of a good way to troubleshoot this further.  My speculation
>>would be that either the hardware is broken or some kernel
>>level code is broke.  The lirc_serial drivers interrupt handling
>>looks good to me.
>>
>
>	There is no guarentee that this will narrow it down, but I think I will build another receiver from scratch and see if it exibits the same behaviour.
>	Thank-you Karl for all your help and explanations. It was very helpful. Are you interested in hearing about my future results?
>
>Dan
>
>