[asterisk-dev] Asterisk Load Performance

[Richard Mudgett]

On Tue, Jun 21, 2016 at 11:12 AM, Michael Petruzzello <
michael.petruzzello at civi.com> wrote:

> >On Fri, Jun 17, 2016 at 1:37 PM, Richard Mudgett <rmudgett at digium.com>
> wrote:
> >>
> >>
> >> On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello
> >> <michael.petruzzello at civi.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am currently working on determining bottlenecks in Asterisk and a
> Stasis
> >>> App. I'm currently trying to handle 83.3 calls/second. For the most
> part,
> >>> Asterisk and the Stasis APP handle that well, but there is a 60+ second
> >>> delay in response time.
> >>>
> >>> On the Asterisk side, I am seeing the following warnings. [Jun 17
> >>> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The
> >>> 'subm:cdr_engine-00000003' task processor queue reached 500 scheduled
> tasks.
> >>> [Jun 17 12:00:18] WARNING[25477][C-00000068]: taskprocessor.c:803
> >>> taskprocessor_push: The 'subm:devService-test-00000038' task processor
> queue
> >>> reached 500 scheduled tasks.
> >>> [Jun 17 12:00:21] WARNING[26298][C-000000a3]: taskprocessor.c:803
> >>> taskprocessor_push: The 'subp:PJSIP/sippeer-00000022' task processor
> queue
> >>> reached 500 scheduled tasks.
> >>> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: taskprocessor.c:803
> >>> taskprocessor_push: The 'subm:ast_channel_topic_all-cached-00000032'
> task
> >>> processor queue reached 500 scheduled tasks.
> >>> [Jun 17 12:01:32] WARNING[31697][C-000003b2]: taskprocessor.c:803
> >>> taskprocessor_push: The 'subm:ast_channel_topic_all-00000036' task
> processor
> >>> queue reached 500 scheduled tasks.
> >>> [Jun 17 12:05:55] WARNING[23280]: taskprocessor.c:803
> taskprocessor_push:
> >>> The 'SIP' task processor queue reached 500 scheduled tasks.
> >>>
> >>> I have not seen a configuration setting on Asterisk to prevent these
> >>> warnings from occurring (I'm trying to avoid modifying Asterisk source
> code
> >>> if possible). Looking at the task processors, I see the queue to the
> stasis
> >>> app bottlenecks:
> >>> subm:devService-test-00000038                    4560990          0
> >>> 1041689. It does clear up relatively quickly. The CDR engine also
> bottle
> >>> necks (extremely badly), but I don't use that. Nothing else comes
> close to
> >>> having a large queue.
> >>>
> >>> The stasis app itself is extremely streamlined and is very capable of
> >>> handling a large number of messages at a time. The app runs with the
> JVM so
> >>> I am also researching into that as well as the netty library I am
> using for
> >>> the websocket connections.
> >>>
> >>> Any insight into Asterisk's side of the equation and how it scales on
> 40
> >>> vCPUs would be greatly appreciated.
> >>
> >>
> >> There are no options to disable those taskprocessor queue size warnings.
> >> They are a
> >> symptom of the system being severely stressed.  If the stress continues
> it
> >> is possible
> >> that the system could consume all memory in those taskprocessor queues.
> >>
> >> Recent changes to the Asterisk v13 branch were made to help throttle
> back
> >> incoming
> >> SIP requests on PJSIP when the taskprocessors become backlogged like
> you are
> >> seeing.
> >> These changes will be in the forthcoming v13.10.0 release.  If you
> want, you
> >> can test with
> >> the current v13 branch to see how these changes affect your stress
> testing.
> >>
> >> If you don't need CDR's then you really need to disable them as they
> consume
> >> a lot of
> >> processing time and the CDR taskprocessor queue backlog can take
> minutes to
> >> clear.
> >>
> >
> >To echo what Richard said, because Asterisk is now sharing state
> >across the Stasis message bus, turning off subscribers to that bus
> >will help performance. Some easy ones to disable, if you aren't using
> >them, are CDRs, CEL, and AMI. Those all do a reasonable amount of
> >processing, and you can get some noticeable improvement by disabling
> >them.
> >
> >Once you get past that, you can start fiddling with some of the lower
> >level options. To start, you can throttle things back further by
> >disabling certain internal messages in stasis.conf. As stasis.conf
> >notes, functionality within Asterisk can break (or just not happen) if
> >some messages are removed. For example, disabling
> >'ast_channel_snapshot_type' would break ... most things. You may
> >however be able to streamline your application by looking at what ARI
> >messages it cares about, what messages it doesn't, inspecting the
> >code, and disabling those that you don't care about. Lots of testing
> >should occur before doing this, of course.
> >
> >You may also be able to get some different performance characteristics
> >by changing the threadpool options for the message bus in stasis.conf.
> >This may make a difference, depending on the underlying machine.
>
> Thank you for the suggestions.
>
> I'm running Asterisk on 40 vCPUs with 120 GB of RAM. Changing the thread
> pool options to many more threads is not increasing performance, and at a
> certain point it decreases performance.
>
> With further testing and having implemented your suggestions, I am
> realizing the subm:devService-test-00000038 task processor is a major
> bottleneck. I have always read that Asterisk can handle as many calls as
> the hardware allows, but I'm not seeing that.
>
> I can go with a different architecture of multiple Asterisk servers
> working together, but I would prefer one server to reduce complexity.
>
> What I am doing is not really a stress test because I really need Asterisk
> to handle 1000s of callers calling in within a few minutes.
>

The subm:devService-test-00000038 taskprocessor is servicing the stasis
message bus
communication with your devService-test ARI application.  Since each
taskprocessor is
executed by one thread, that is going to be a bottleneck.  One thing you
can try is to
register multiple copies of your ARI application and randomly spread the
calls to the
different copies of the application.  (devService-test1,
devService-test2,...)

Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20160621/a1d87822/attachment.html>