[Asterisk-Dev] FW: hdlc link down errors

[Carey Jung]

Hi,

I'm having problems with Wildcard T100/T400 cards shutting down every day or
two when running in hdlc/cisco mode.  I've got 3 T100 cards connected to a
T400 card over T1 loops.  Loops have all tested clean by my telco provider
and, in one case, has been up for over a year with no errors.  (The other
two loops are fairly new.)  Note that I have seen this on all 4 cards at
various points in time.  I noticed that the lock-ups seem to be coincident
with heavy loads (indicated by lots of iptables packet filter log entries),
so I turned off the packet filter logging; this mitigated the problem, but
hasn't made it go away.

The problem is that every so often, I'll get a 'Link down' error in
/var/log/messages, and the card stops.  I haven't figured out how to restart
it, so I have to reboot.  Digium support thought it might be in the kernel
hdlc driver code, but I talked to the hdlc maintainer, and he had the
following to say.

By the way, I'm running Redhat's kernel 2.4.18-27.7.x, with hdlc support
compiled in.  I'm using the stock "ifup-hdlc" and "zaptel" init scripts that
come in the zaptel source.

I'm not familiar at all with debugging driver code, so if any of you helpful
folks have any ideas for how I should proceed, I'd greatly appreciate it.

BTW, in answer to Krzysztof's questions below, I'm not sure if the card
continues to request interrupts after the link error (/proc/interrupts), and
it doesn't appear that RX/TX packets are logged at all (ifconfig).

thanks for any help,
Carey

> -----Original Message-----
> From: Krzysztof Halasa [mailto:khc at pm.waw.pl]
> Sent: Monday, May 05, 2003 10:51 AM
> To: Carey Jung
> Subject: Re: hdlc link down errors
>
>
> Hi,
>
> "Carey Jung" <carey at itfreedom.com> writes:
>
> > I'm not sure if you can help me, but here goes.  I'm running a
> Digium T100
> > card (www.digium.com) in hdlc mode -- actually, several cards in three
> > different machines.  Kernel is Redhat's 2.4.18-27x kernel.
> Everything works
> > great, except that occasionally I'll get a 'kernel: hdlc0: Link
> down' error
> > and have to restart the box.
>
> I assume it "Cisco" HDLC mode.
>
> > I've noticed that the failure seems to occur in the midst of a
> lot of kernel
> > packet logs from iptables, from outsiders apparently running port scans
> > against my box.  So, I'm thinking that the hdlc code may not be able
> > handling kernel logging very well or something like that.  I
> can otherwise
> > load the connection down continuously with no problems.
>
> I don't think so. The hdlc code doesn't handle kernel logging in fact.
> I would rather look at underlying hardware driver - it might be hardware
> problem, or something like a bug in transmit or interrupt service routine.
> I would check first if both TX and RX operations halt, if only RX fails
> it might be transmit routine, for example bad transmitter locking.
>
> > I know this isn't much to go on, but do you have any ideas how
> I can pursue
> > this?  For now, I've turned off iptables packet logging, and the link is
> > staying up.
>
> Turning it off probably reduces interrupt overhead, but you can never be
> sure it will survive under heavy load (from other devices).
> Does the card request its interrupts after it halts (the counter can
> be examined by 'cat /proc/interrupts')?
> Do TX and/or RX counters (ifconfig) increment?
>
> I don't know if the driver has any debugging capabilities, but
> you may want
> to check it (I have never seen such a card).
>
>
> Such problems are usually easy to fix _after_ they are spotted - author
> of the hardware driver / customer support etc. may be able to better help
> you.
> --
> Krzysztof Halasa
> Network Administrator
>