[asterisk-dev] Asterisk Load Performance

[Richard Mudgett]

On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello <
michael.petruzzello at civi.com> wrote:

> Hello,
>
> I am currently working on determining bottlenecks in Asterisk and a Stasis
> App. I'm currently trying to handle 83.3 calls/second. For the most part,
> Asterisk and the Stasis APP handle that well, but there is a 60+ second
> delay in response time.
>
> On the Asterisk side, I am seeing the following warnings. [Jun 17
> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The
> 'subm:cdr_engine-00000003' task processor queue reached 500 scheduled tasks.
> [Jun 17 12:00:18] WARNING[25477][C-00000068]: taskprocessor.c:803
> taskprocessor_push: The 'subm:devService-test-00000038' task processor
> queue reached 500 scheduled tasks.
> [Jun 17 12:00:21] WARNING[26298][C-000000a3]: taskprocessor.c:803
> taskprocessor_push: The 'subp:PJSIP/sippeer-00000022' task processor queue
> reached 500 scheduled tasks.
> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: taskprocessor.c:803
> taskprocessor_push: The 'subm:ast_channel_topic_all-cached-00000032' task
> processor queue reached 500 scheduled tasks.
> [Jun 17 12:01:32] WARNING[31697][C-000003b2]: taskprocessor.c:803
> taskprocessor_push: The 'subm:ast_channel_topic_all-00000036' task
> processor queue reached 500 scheduled tasks.
> [Jun 17 12:05:55] WARNING[23280]: taskprocessor.c:803 taskprocessor_push:
> The 'SIP' task processor queue reached 500 scheduled tasks.
>
> I have not seen a configuration setting on Asterisk to prevent these
> warnings from occurring (I'm trying to avoid modifying Asterisk source code
> if possible). Looking at the task processors, I see the queue to the stasis
> app bottlenecks:
> subm:devService-test-00000038                    4560990          0
> 1041689. It does clear up relatively quickly. The CDR engine also bottle
> necks (extremely badly), but I don't use that. Nothing else comes close to
> having a large queue.
>
> The stasis app itself is extremely streamlined and is very capable of
> handling a large number of messages at a time. The app runs with the JVM so
> I am also researching into that as well as the netty library I am using for
> the websocket connections.
>
> Any insight into Asterisk's side of the equation and how it scales on 40
> vCPUs would be greatly appreciated.
>

There are no options to disable those taskprocessor queue size warnings.
They are a
symptom of the system being severely stressed.  If the stress continues it
is possible
that the system could consume all memory in those taskprocessor queues.

Recent changes to the Asterisk v13 branch were made to help throttle back
incoming
SIP requests on PJSIP when the taskprocessors become backlogged like you
are seeing.
These changes will be in the forthcoming v13.10.0 release.  If you want,
you can test with
the current v13 branch to see how these changes affect your stress testing.

If you don't need CDR's then you really need to disable them as they
consume a lot of
processing time and the CDR taskprocessor queue backlog can take minutes to
clear.

Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20160617/3a85f6f6/attachment.html>