[asterisk-dev] Asterisk scalability

[John Todd]

On Feb 17, 2009, at 4:10 AM, Benny Amorsen wrote:

> Johansson Olle E <oej at edvina.net> writes:
>
>> With SIP (for some reason, I did not test IAX2 ;-) ) we clearly see
>> that the system dies around 1600 simultaneous channels being trashed
>> by all the RTP traffic using the p2p rtp bridge. irqbalance fails
>> miserably at that point and lack of call quality makes all calls
>> sound like the Swedish chef. Unusable is a good description.
>
>> My conclusion so far is the same as Stephen Uhler's at the Atlanta
>> Astridevcon - the IP stack is our problem here.
>
> The IP stack itself can only do so much here... At 1600 simultaneous
> calls and 20ms packets, that is 80000pps. The ethernet card will do
> interrupt mitigation at that point, but AFAIK no ethernet card has any
> kind of UDP receive offload.
>
> If you have as many ethernet cards as CPU cores, and you balance the
> traffic across the cards, you should in theory get almost perfect
> scaling with number of cores (unless Asterisk has problems, but they
> can be fixed). If you only have one ethernet card, you
> will never go beyond single-core performance -- unless the card does
> multiqueue receive, but that is very new so that probably wasn't in
> use for your test.


So this raises another interesting point which is not related to  
Asterisk directly, but may influence the final "scalability" of  
Asterisk on certain hardware.  As many of you know, scaling Asterisk  
is a subject near and dear to my heart, so any tricks that make it  
possible to reduce system footprint and architectural density are a  
Good Thing as far as I'm concerned.   I have an all-expenses-paid  
Morton's steak dinner at Astricon waiting for the person who cracks  
10,000 G.711 call legs on a single machine on a single instance of  
Asterisk.

We have a given fact, which we must figure out a way around:  There is  
a limit to the number of packets that can be transmitted/received over  
an Ethernet interface.  This is a limit that, eventually, we will  
reach in the goal of getting ever-higher numbers of SIP/RTP sessions  
on a single backplane.  So let's put aside the Asterisk limits for the  
moment and go directly to the question of "What can be done to  
increase PPS throughput to a single machine?"  It seems that any  
efforts to improve RTP packet relay would eventually be constrained by  
PPS through a single ethernet card long before the bandwidth limits of  
that card would be reached.  Of course, any issues with Asterisk  
should be addressed, and need to be addressed before we hit this  
distant wall.   But I think the wall is not-too-distant any more, and  
I'd like to hear discussions on what to do before we get there.

There is one line of attack which uses interrupt coalescence to reduce  
the number of IRQs sent out by the ethernet card onto the PCI bus.   
This is a card-specific feature, though many modern Ethernet NICs  
support it.  There have been discussions on this topic on -dev and I  
think also on -users, and hopefully there will be some test results  
soon with real data (though I've been hoping for that real data using  
Asterisk or other high-PPS RTP traffic for a few YEARS now.)  Since  
this is at least understood and previously discussed, I'll leave it  
alone, close my eyes, and hope that it generates good results all by  
itself.

The second method that comes to mind has not been (to my recollection)  
discussed here or on asterisk-users, though with the thousands of  
posts on both forums please forgive me if this has already been a  
topic that has seen reference.  This second method would be "Interface  
Bonding" (aka Interface Teaming, aka Ethernet trunking, aka 802.3ad)  
which most (all?) modern Linux variations support, as well as most of  
the BSD derivatives.   It seems fairly trivial to do - instructions  
seem simple, and it seems like the right thing to eliminate overload  
on IRQs on a single card by distributing the load out across N cards  
in the backplane and treating them as a single interface with a single  
IP address.   While the layer 3 protocol thinks it would be a single  
address, the layer 2 protocol distributes packets across the multiple  
cards, which should in theory distribute the IRQ load across the cards  
and cores in the system.  In fact, it seems like a great idea in  
general to have multiple cards even under moderate load, since even  
with IRQ balancing (which apparently doesn't actually work) there is  
one CPU that takes all the interrupts from a single card.  Plus,  
802.3ad seems like a quick way to get some wiring redundancy even  
though you're keeping your other single points of failure intact  
(primary system, upstream switch.)

After some reading, I am unclear if Cisco devices work as expected for  
transmitting inbound load - I can't find any methods by which  
something like "round-robin" can be specified on an 802.3ad group -  
Linux how-tos are pretty clear how it can be done on the host side.   
It seems that Cisco gear only supports failover capabilities.  This, I  
hope, is just my lack of time to search for better answers.  If you  
know that Ciscos can semi-evenly transmit load across two or more  
802.3ad interfaces using some heuristic, let's hear about it.  Or  
other network gear, for that matter.

Random references:
http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php
http://my.opera.com/blu3c4t/blog/2008/02/01/openbsd-nic-bonding
http://www.linux.com/feature/133849


PS: This seems a little off-topic for asterisk-dev, but I suspect  
those who have the most significant interest in getting their Asterisk  
systems to use preposterous density solutions are probably more likely  
to pay close attention to -dev.

JT

--
John Todd                       email:jtodd at digium.com
Digium, Inc. | Asterisk Open Source Community Director
445 Jan Davis Drive NW -  Huntsville AL 35806  -   USA
direct: +1-256-428-6083         http://www.digium.com/