[asterisk-dev] Asterisk Load Performance

Tue Jul 5 15:43:12 CDT 2016

On Wed, Jun 29 at 11:14:04 AM, Richard Mudgett<rmudgett at digium.com>
wrote:
> Each softmix bridge has only one thread performing all of the media mixing
> for the bridge.  To
> get better mixing performance for such a large conference, you will need
to
> create several
> softmix bridges in a hierarchy with the bridges linked by local channels.

A bridge is only able to handle around 2000-2500 channels, so I created 15
bridges with 14 channels bridging the bridges together.

When doing this an error I see a lot is WARNING[98920]: channel.c:1101
__ast_queue_frame: Exceptionally long voice queue length queuing to
Local/**********@default-00000000;2, which then turns into WARNING[47525]:
pjproject:0 <?>:      sip_transactio .Unable to register INVITE transaction
(key exists) and ERROR[47525]: res_pjsip.c:2777 ast_sip_create_dialog_uas:
Could not create dialog with endpoint sippeer. Object already exists
(PJ_EEXISTS). Finally the following repeats over and over again, [Jun 30
12:22:21] ERROR[84189][C-00000958]: netsock2.c:305 ast_sockaddr_resolve:
getaddrinfo("domain.name", "(null)", ...): Temporary failure in name
resolution
[Jun 30 12:22:21] WARNING[84189][C-00000958]: acl.c:800 resolve_first:
Unable to lookup 'domain.name'.

The last error just keeps on repeating and calls can no longer join (only
around 3,500 make it on before this starts to occur). Calling in manually I
receive an "all circuits are busy" message.

I'm going to try halving the number of bridges, but is there anything else
I can do to improve performance? This seems to be the last hurdle to use
one server for 10,000 callers.

*Michael J. Petruzzello*
Software Engineer
P.O. Box 4689
Greenwich, CT 06831
203-618-1811 ext.289 (office)
www.civi.com

On Wed, Jun 29, 2016 at 10:55 AM, Michael Petruzzello <
michael.petruzzello at civi.com> wrote:

> It is very interesting how threading issues on both a stasis application
> and Asterisk escalate each other. Using 15 websockets in one stasis
> application and removing all thread locking from the application have made
> the ARI messages flow smoothly. Right now I am using about 900 threads to
> process messages from Asterisk and Asterisk has at least 320 in stasis,
> though that can increase to infinity.
>
> I have also disabled the channel_varset from stasis because it becomes
> really unwieldy. When having thousands of callers in a bridge, every time a
> channel is added to a bridge or removed, every channel receives a channel
> var set message because of the BridgePeer variable.
>
> As of now, I have two remaining problems:
>
> 1. At around having 5,000 channels in a bridge (whether majority are muted
> or not), the audio breaks down. Anyone talking can only be heard in 3
> second bursts approximately every 5-10 seconds. At 10,000 channels only
> static can be heard in these 3 second bursts.
>
> Is there anything I can optimize so that Asterisk can handle all these
> channels in a bridge?
>
> 2. Every time a channel joins the bridge, the websocket responsible for
> that channel is then subscribed to the bridge. Then any events that occur
> on that bridge (such as another channel entering or exiting it) are sent to
> that websocket. Because every websocket then ends up receiving these
> messages, it defeats the point of having multiple websockets. To get around
> this I have been unsubscribing the websockets from the bridge anytime a
> channel from that websocket enters the bridge, but this isn't perfect as
> timing is an issue.
>
> Is there anyway to disable this automatic subscription behavior to a
> bridge?
>
>
> *Michael J. Petruzzello*
> Software Engineer
> P.O. Box 4689
> Greenwich, CT 06831
> 203-618-1811 ext.289 (office)
> www.civi.com
>
> On Tue, Jun 21, 2016 at 3:29 PM, Michael Petruzzello <
> michael.petruzzello at civi.com> wrote:
>
>> On Tue, Jun 21, 2016 at 12:16 PM, Richard Mudgett <rmudgett at digium.com>
>> wrote:
>> > The subm:devService-test-00000038 taskprocessor is servicing the stasis
>> > message bus
>> > communication with your devService-test ARI application.  Since each
>> > taskprocessor is
>> > executed by one thread, that is going to be a bottleneck.  One thing you
>> > can try is to
>> > register multiple copies of your ARI application and randomly spread the
>> > calls to the
>> > different copies of the application.  (devService-test1,
>> > devService-test2,...)
>>
>> Ah, that explains it! Everything else has been running well in Asterisk
>> as far as handling the actual channels and the SIP messaging with the large
>> calls / second.
>>
>> I was thinking about the potential of parallelizing the stasis message
>> bus communication to use multiple task processors, but that would introduce
>> other issues. Messages would be sent out of order, and the ARI application
>> would need to handle that.
>>
>> Your suggestion sounds like the best approach. That way I still have only
>> one application with multiple connections to Asterisk. No need to have
>> multiple applications and servers that would need to communicate together.
>>
>> Thank you for the insight.
>>
>> On Tue, Jun 21, 2016 at 1:03 PM, Matthew Jordan<mjordan at digium.com>
>> wrote:
>> > To follow up with Richard's suggestion:
>> >
>> > Events being written out (either over a WebSocket in ARI or over a
>> > direct TCP socket in AMI) have to be fully written before the next
>> > event is written. That means that the client application processing
>> > the events can directly slow down the rate at which events are sent if
>> > the process that is reading the event does not keep reading from the
>> > socket as quickly as possible. You may already be doing this - in
>> > which case, disregard the suggestion - but you may want to have one
>> > thread/process read from the ARI WebSocket, and farm out the
>> > processing of the events to some other thread/process.
>>
>> I am already doing this, but thank you for the suggestion. Database
>> access really slows down the processing of events so I have had to do this
>> from the start of my project.
>>
>>
>> *Michael J. Petruzzello*
>> Software Engineer
>> P.O. Box 4689
>> Greenwich, CT 06831
>> 203-618-1811 ext.289 (office)
>> www.civi.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20160705/a7c421cd/attachment.html>