[Asterisk-Dev] Missing audio from Zaptel channels

[Rod Bacon]

I have cross-posted this all over the place, and sent a copy directly to digium
support, in the hope of getting to the bottom of a problem that has me pulling
my hair out.

I currently have 2 production PSTN gateway servers, running asterisk 1.2beta and
TE406P cards (upgraded 405 cards, with hardware echo cancelers that we recently
purchased on recommendation). We went to the beta version after installing the
cancelers, as 1.0.9 kept segfaulting with the cancelers installed. Our PRIs
terminate on a DMS100, at the same premises where our servers are co-located.

Also in my farm, I have a dedicated IVR server, a VOIP gateway (SIP/IAX/H.323)
and clustered MySQL servers running as FastAGI servers, to remove processor load
from the PSTN servers. All servers are connected via gigabit Ethernet, and
use IAX trunking for inter-server communications.

I have been through _everything_ possible to be sure that I don't have any
zaptel timing/irq problems (framebuffer, apic, acpi, smp irq affinity, irq
latency, etc. etc) and have good zttest results with no frame slips, pops or clicks.

After my PSTN gateway servers have been running for a few hours, I notice that
some missing audio creeps into the start of each call (makes no difference if
the call is ZAP-ZAP native bridge or ZAP-IAX). At best, you miss the first
syllable of the first word. At worst, you can miss the first 3 or 4 seconds of
audio. Further investigation shows that asterisk is lagging after the second leg
of the call is answered (i.e. the time taken to bridge the channels gets 
longer). If the resultant call is a Zaptel native bridge, then the remaining 
audio is fine. If the resultant call is not zaptel natively bridged (eg. call is 
routed via another server, or asterisk remains in the media stream for another 
reason) then significant delay exists from one end of the call to another 
(simply put, asterisk seems to slow down).

If I restart asterisk (even without removing and reloading zaptel drivers), 
calls are OK again for a period (typically around 12 hours). A workaround is to 
simply to install a cron job that periodically restarts asterisk when it's idle, 
  but this is a less than ideal solution from my perspective.

Something is definitely changing over time. A memory leak? Runaway process? I
really need help in trying to troubleshoot this, as I've run completely out of
both patience and ideas.


-- 
==========================================
Rod Bacon
Empowered Communications
Ground Floor, 102 York St. South Melbourne
Victoria, Australia. 3205
Phone: +613 99401600 Fax: +613 99401650
FWD: 512237 ICQ: 5662270
==========================================