[asterisk-users] Breaking news, but what happened? 11.000 channels on one server

[John Todd]

On Aug 25, 2009, at 5:59 AM, Olle E. Johansson wrote:

> Hello Asterisk users around the world!
>
> Recently, I have been working with pretty large Asterisk
> installations. 300 servers running Asterisk and Kamailio (OpenSER).
> Replacing large Nortel systems with just a few tiny boxes and other
> interesting solutions. Testing has been a large part of these
> projects. How much can we put into one Asterisk box? Calls per euro
> invested matters.
>
> So far, we've been able to reach about 2000 channels of G.711 with
> quad core CPU's and Intel Pro/1000 network cards in IBM servers. At
> that point, we see that IRQ balancer gives up and goes to bed, and all
> the traffic is directed to one core and the system gives up. We've
> been running these tests on several systems, with different NICs and
> have been working hard to tweak capacity. New drivers, new cards, new
> stuff. But all indications told us that the problem was the CPU
> context switching between handling network traffic (RTP traffic) and
> Asterisk. This was also confirmed from a few different independent
> software development teams.
>
> Imaging my surprise this Monday when I installed a plain old Asterisk
> 1.4 on a new HP server, a DL380 G6, and could run in circles around
> the old IBM servers. Three servers looping calls between them and we
> bypassed 10.000 channels without any issues.  SIP to SIP calls, the
> p2p RTP bridge, basically running a media proxy. At that point, our
> cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit
> was more than enough. The CPU's (we had 16 of them with
> hyperthreading) was not very stressed. Asterisk was occupying a few of
> them in a nice way, but we had a majority of them idling around
> looking for something to do.I'll
>
> So, please help me. I need answers to John Todds questions while he's
> treating me with really good expensive wine at Astricon. How did this
> happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or
> a combination? Or maybe just the cheap Netgear switch...
>
> I hope to get more access to these boxes, three of them, to run tests
> with the latest code. In that version we have the new hashtables, all
> the refcounters and fancy stuff that the Digium team has reworked on
> the inside of Asterisk. The trunk version will propably behave much,
> much better than 1.4 when it comes to heavy loads and high call setup
> rates.
>
> We're on our way to build a new generation of Asterisk, far away from
> the 1.0 platform. At the same time, the hardware guys have obviously
> not been asleep. They're giving us inexpensive hardware that makes our
> software shine. Now we need to test other things and see how the rest
> of Asterisk scales, apart from the actual calls. Manager, events,
> musiconhold, agi/fastagi... New interesting challenges.
>
> So take one of these standard rack servers from HP and run a telco for
> a small city on one box. While you're at it, buy a spare one, hardware
> can fail ( ;-) ).
> But don't say that Asterisk does not scale well. Those times are gone.
>
> /Olle
>
> ---
> * Olle E Johansson - oej at edvina.net
> * Open Unified Communication - SIP & XMPP projects
>


Your dinner awaits, along with a very nice bottle of wine. (or port,  
or whatever it is you prefer.)  But, just a few questions..  ;-)

Moving back away from Layer 3 discussion that erupted on this thread,  
let's get back to what you actually did.

It seems odd to me (though I'm hopeful!) that this would "just work"  
without any changes.  Especially on 1.4.  So being somewhat scientific  
about it, I'd say that the first thing to do is to examine your  
measurements with an assumption that there is a flaw in your  
observations.  If you find no errors with that hypothesis, then you've  
deduced that things are as they seem.  :-)

1) Are you certain that the media was actually being routed?  I know  
you said that the switch and NICs gave up because of traffic, but did  
you choose a random channel and record the media between two servers?   
In other words, are you 100% certain that there was valid RTP being  
exchanged?  (I typically make one call out of every 100 or 1000 a  
"monkeys" call, and record it instead of just routing to Echo()  
directly, then play back later to ensure media was actually happening.)

2) Were the SIP transactions completing normally?  What was the rate  
of ramp-up?

3) Can you post your dialplan and a few snippets of "core show  
channels" at peak usage?

4) What was the sampling rate for the media?  20ms? 30ms? 40ms?

5) Any summary stats on RTP packet loss, etc? (from  
"CHANNEL(rtpqos,audio,all)") on channels?

JT

---
John Todd                       email:jtodd at digium.com
Digium, Inc. | Asterisk Open Source Community Director
445 Jan Davis Drive NW -  Huntsville AL 35806  -   USA
direct: +1-256-428-6083         http://www.digium.com/