[asterisk-dev] Asterisk scalability

[Kaloyan Kovachev]

On Fri, 20 Feb 2009 15:48:29 -0800, John Todd wrote
> On Feb 20, 2009, at 12:00 PM, Kevin P. Fleming wrote:
> 
> > John Todd wrote:
> >
> >> Hypothetical: My big router receives a 10gpbs ethernet from "the
> >> Internet".  Then, that router has three 1gbps interfaces that feed
> >> into my Asterisk box (which has three NICs.)  Thus, all of the  
> >> traffic
> >> appears to come from the MAC address on the distant side of the  
> >> 10gbps
> >> link.  I could generate the traffic from a zillion different
> >> endpoints, so hopefully UDP port hashing would work.  Would this work
> >> appropriately for spreading load on 802.3ad?
> >
> > If your 'big router' has three NICs connected to you, it will have  
> > three
> > MAC addresses for those NICs, and that's what you will see on your
> > incoming packets, not the MAC address of the router on the other side.
> >
> > If your 'big router' is actually a switch (which I think is what you
> > really meant), then your logic is correct.
> 
> I had assumed we were talking about seeing MAC addresses from the  
> perspective of the device doing the load sharing on packets that were  
> inbound from the Internet towards the Asterisk server.  That is the  
> device that would have to "share" those packets across the three NICs  
> of the Asterisk server, and therefore would have to come up with some  
> algorithm to distribute evenly (or at least predictably) the traffic  
> to the three interfaces.  Return traffic from the Asterisk server to  
> the router/switch is the problem of the Asterisk box, but it looks  
> like there are lots of knobs to turn on on the open-source side of the  
> world with 802.3ad implementations that make that possible.
> 
> However, I'm speaking out of turn.  If others here think that 802.3ad  
> works well and can "share" load across multiple NICs that are on a  
> single host (versus just doing failover) then I'll believe them, but I  
> haven't heard a "Yes absolutely it can be done" reply.  I strongly  
> suspect that it does in fact work, but I've just never set it up to  
> learn about the devil in the details.  It seems like 802.3ad would be  
> a good way to share load across multiple NICs to reduce IRQ contention  
> on the PCI backplane with high packet counts, and it also seems pretty  
> universally supported by most mid-to-high end router/switch gear as  
> well as Linux/BSD kernels.  It has the added benefit of reducing some  
> single wiring points of failure, but introduces complexity.
> 
> Who has tried it?  Speak up!  Your experiences make you the one-eyed  
> man in this land of the blind.
> 

Yes absolutely it can be done :)

There are different modes for bonding in Linux, but the most important part
for "sharing the load" is the hash algorithm used. Some switches have options
for including the incoming port for the hash algorithm, which is good in the
case of 3 cards on the router going to the Asterisk via switch. Even without
this option there is IP/port hashing algorithm (see [1] layer3+4), which will
definitely share the traffic between the 3 cards equally enough to spreed the
IRQ's between them because each RTP stream will use different ports even going
to the same destination IP (of the Asterisk server). If the ports selected are
even only - the number of NICs should be odd.

If the 3 cards are only for VoIP and going directly to the Asterisk server
then instead of 802.3ad it is best using round robin mode for the bonding. The
drawback of RR is that the packets may come out of order, but for the VoIP
traffic it is not a problem, because the packets from the same connection are
coming slower (even at 10ms intervals) than they will be received on the other
side.

[1] http://www.linuxfoundation.org/en/Net:Bonding

> JT
> 
> ---
> John Todd                       email:jtodd at digium.com
> Digium, Inc. | Asterisk Open Source Community Director
> 445 Jan Davis Drive NW -  Huntsville AL 35806  -   USA
> direct: +1-256-428-6083         http://www.digium.com/
> 
> _______________________________________________
> --Bandwidth and Colocation Provided by http://www.api-digital.com--
> 
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev