[asterisk-users] Asterisk 13.22.0 under very high load conditions - freezes in H exten and blocks new calls

[Dovid Bender]

Are you using NFS? Any ODBC connections?

On Tue, Apr 21, 2020 at 10:23 AM Stefan Viljoen <viljoens at verishare.co.za>
wrote:

> Hi all
>
> I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7 -
> 32GB RAM.
> When I approach about 320 channels, I -sometimes- get thousands of these
> messages suddenly streamed in the CLI / Asterisk log:
>
> WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dbea;2
> WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dd10;2
> WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxx at local-0002dd30;1
>
> Top reports system load as 30.0, but the box is still very responsive over
> SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
> close to it) and doesn't consume more as this state comes into effect.
>
> Top also shows that HT CPU instances are idle 40% of their time.
>
> What then happens when these messages appear is Asterisk stops accepting
> originates on the AMI, and that no CDRs write through to the CDR DB anymore
> - MySQL itself is idle and has no locks or running inserts indicated in
> "show processlist" in MySQL CLI.
>
> The Asterisk instance will then effectively freeze, never hanging up
> terrminated calls, and never allowing new cals to start, keeping around 320
> channels open, forever.
>
> Thing is I can sometimes peak at over 350 channels, then come back down as
> answers, hangups, etc. take place - no problem. But -sustaining- more than
> 320 channels over a two or three minutes seems to reliably cause the above
> to happen.
>
> Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
> sighup / cli "core stop now"
>
> CLI does work and is still repsonsive in this state, until I do a "core
> stop
> now", which doesn't stop the instance even after 20 to 30 minutes.
>
> Asterisk max load is set to 40 and max calls to 350.
>
> Any idea what I can look at? Any tips for running specifically 13 under
> high
> load?
>
> Thanks
>
> Stefan
>
>
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at:
> https://community.asterisk.org/
>
> New to Asterisk? Start here:
>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200421/577bc875/attachment.html>