[asterisk-users] Asterisk 13.22.0 under very high load conditions - freezes in H exten and blocks new calls

[Stefan Viljoen]

Hi all

I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7 -
32GB RAM. 
When I approach about 320 channels, I -sometimes- get thousands of these
messages suddenly streamed in the CLI / Asterisk log:

WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dbea;2
WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dd10;2
WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
queuing to Local/xxxx at local-0002dd30;1

Top reports system load as 30.0, but the box is still very responsive over
SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
close to it) and doesn't consume more as this state comes into effect.

Top also shows that HT CPU instances are idle 40% of their time.

What then happens when these messages appear is Asterisk stops accepting
originates on the AMI, and that no CDRs write through to the CDR DB anymore
- MySQL itself is idle and has no locks or running inserts indicated in
"show processlist" in MySQL CLI.

The Asterisk instance will then effectively freeze, never hanging up
terrminated calls, and never allowing new cals to start, keeping around 320
channels open, forever.

Thing is I can sometimes peak at over 350 channels, then come back down as
answers, hangups, etc. take place - no problem. But -sustaining- more than
320 channels over a two or three minutes seems to reliably cause the above
to happen.

Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
sighup / cli "core stop now"

CLI does work and is still repsonsive in this state, until I do a "core stop
now", which doesn't stop the instance even after 20 to 30 minutes.

Asterisk max load is set to 40 and max calls to 350.

Any idea what I can look at? Any tips for running specifically 13 under high
load?

Thanks

Stefan