[asterisk-dev] Asterisk Load Performance

Tue Jun 21 11:12:59 CDT 2016

>On Fri, Jun 17, 2016 at 1:37 PM, Richard Mudgett <rmudgett at digium.com>
wrote:
>>
>>
>> On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello
>> <michael.petruzzello at civi.com> wrote:
>>>
>>> Hello,
>>>
>>> I am currently working on determining bottlenecks in Asterisk and a
Stasis
>>> App. I'm currently trying to handle 83.3 calls/second. For the most
part,
>>> Asterisk and the Stasis APP handle that well, but there is a 60+ second
>>> delay in response time.
>>>
>>> On the Asterisk side, I am seeing the following warnings. [Jun 17
>>> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The
>>> 'subm:cdr_engine-00000003' task processor queue reached 500 scheduled
tasks.
>>> [Jun 17 12:00:18] WARNING[25477][C-00000068]: taskprocessor.c:803
>>> taskprocessor_push: The 'subm:devService-test-00000038' task processor
queue
>>> reached 500 scheduled tasks.
>>> [Jun 17 12:00:21] WARNING[26298][C-000000a3]: taskprocessor.c:803
>>> taskprocessor_push: The 'subp:PJSIP/sippeer-00000022' task processor
queue
>>> reached 500 scheduled tasks.
>>> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: taskprocessor.c:803
>>> taskprocessor_push: The 'subm:ast_channel_topic_all-cached-00000032'
task
>>> processor queue reached 500 scheduled tasks.
>>> [Jun 17 12:01:32] WARNING[31697][C-000003b2]: taskprocessor.c:803
>>> taskprocessor_push: The 'subm:ast_channel_topic_all-00000036' task
processor
>>> queue reached 500 scheduled tasks.
>>> [Jun 17 12:05:55] WARNING[23280]: taskprocessor.c:803
taskprocessor_push:
>>> The 'SIP' task processor queue reached 500 scheduled tasks.
>>>
>>> I have not seen a configuration setting on Asterisk to prevent these
>>> warnings from occurring (I'm trying to avoid modifying Asterisk source
code
>>> if possible). Looking at the task processors, I see the queue to the
stasis
>>> app bottlenecks:
>>> subm:devService-test-00000038                    4560990          0
>>> 1041689. It does clear up relatively quickly. The CDR engine also bottle
>>> necks (extremely badly), but I don't use that. Nothing else comes close
to
>>> having a large queue.
>>>
>>> The stasis app itself is extremely streamlined and is very capable of
>>> handling a large number of messages at a time. The app runs with the
JVM so
>>> I am also researching into that as well as the netty library I am using
for
>>> the websocket connections.
>>>
>>> Any insight into Asterisk's side of the equation and how it scales on 40
>>> vCPUs would be greatly appreciated.
>>
>>
>> There are no options to disable those taskprocessor queue size warnings.
>> They are a
>> symptom of the system being severely stressed.  If the stress continues
it
>> is possible
>> that the system could consume all memory in those taskprocessor queues.
>>
>> Recent changes to the Asterisk v13 branch were made to help throttle back
>> incoming
>> SIP requests on PJSIP when the taskprocessors become backlogged like you
are
>> seeing.
>> These changes will be in the forthcoming v13.10.0 release.  If you want,
you
>> can test with
>> the current v13 branch to see how these changes affect your stress
testing.
>>
>> If you don't need CDR's then you really need to disable them as they
consume
>> a lot of
>> processing time and the CDR taskprocessor queue backlog can take minutes
to
>> clear.
>>
>
>To echo what Richard said, because Asterisk is now sharing state
>across the Stasis message bus, turning off subscribers to that bus
>will help performance. Some easy ones to disable, if you aren't using
>them, are CDRs, CEL, and AMI. Those all do a reasonable amount of
>processing, and you can get some noticeable improvement by disabling
>them.
>
>Once you get past that, you can start fiddling with some of the lower
>level options. To start, you can throttle things back further by
>disabling certain internal messages in stasis.conf. As stasis.conf
>notes, functionality within Asterisk can break (or just not happen) if
>some messages are removed. For example, disabling
>'ast_channel_snapshot_type' would break ... most things. You may
>however be able to streamline your application by looking at what ARI
>messages it cares about, what messages it doesn't, inspecting the
>code, and disabling those that you don't care about. Lots of testing
>should occur before doing this, of course.
>
>You may also be able to get some different performance characteristics
>by changing the threadpool options for the message bus in stasis.conf.
>This may make a difference, depending on the underlying machine.

Thank you for the suggestions.

I'm running Asterisk on 40 vCPUs with 120 GB of RAM. Changing the thread
pool options to many more threads is not increasing performance, and at a
certain point it decreases performance.

With further testing and having implemented your suggestions, I am
realizing the subm:devService-test-00000038 task processor is a major
bottleneck. I have always read that Asterisk can handle as many calls as
the hardware allows, but I'm not seeing that.

I can go with a different architecture of multiple Asterisk servers working
together, but I would prefer one server to reduce complexity.

What I am doing is not really a stress test because I really need Asterisk
to handle 1000s of callers calling in within a few minutes.

*Michael J. Petruzzello*
Software Engineer
P.O. Box 4689
Greenwich, CT 06831
203-618-1811 ext.289 (office)
www.civi.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20160621/5c6f5d9a/attachment.html>