[asterisk-users] Breaking news, but what happened? 11.000 channels on one server

[Olle E. Johansson]

Hello Asterisk users around the world!

Recently, I have been working with pretty large Asterisk  
installations. 300 servers running Asterisk and Kamailio (OpenSER).  
Replacing large Nortel systems with just a few tiny boxes and other  
interesting solutions. Testing has been a large part of these  
projects. How much can we put into one Asterisk box? Calls per euro  
invested matters.

So far, we've been able to reach about 2000 channels of G.711 with  
quad core CPU's and Intel Pro/1000 network cards in IBM servers. At  
that point, we see that IRQ balancer gives up and goes to bed, and all  
the traffic is directed to one core and the system gives up. We've  
been running these tests on several systems, with different NICs and  
have been working hard to tweak capacity. New drivers, new cards, new  
stuff. But all indications told us that the problem was the CPU  
context switching between handling network traffic (RTP traffic) and  
Asterisk. This was also confirmed from a few different independent  
software development teams.

Imaging my surprise this Monday when I installed a plain old Asterisk  
1.4 on a new HP server, a DL380 G6, and could run in circles around  
the old IBM servers. Three servers looping calls between them and we  
bypassed 10.000 channels without any issues.  SIP to SIP calls, the  
p2p RTP bridge, basically running a media proxy. At that point, our  
cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit  
was more than enough. The CPU's (we had 16 of them with  
hyperthreading) was not very stressed. Asterisk was occupying a few of  
them in a nice way, but we had a majority of them idling around  
looking for something to do.

So, please help me. I need answers to John Todds questions while he's  
treating me with really good expensive wine at Astricon. How did this  
happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or  
a combination? Or maybe just the cheap Netgear switch...

I hope to get more access to these boxes, three of them, to run tests  
with the latest code. In that version we have the new hashtables, all  
the refcounters and fancy stuff that the Digium team has reworked on  
the inside of Asterisk. The trunk version will propably behave much,  
much better than 1.4 when it comes to heavy loads and high call setup  
rates.

We're on our way to build a new generation of Asterisk, far away from  
the 1.0 platform. At the same time, the hardware guys have obviously  
not been asleep. They're giving us inexpensive hardware that makes our  
software shine. Now we need to test other things and see how the rest  
of Asterisk scales, apart from the actual calls. Manager, events,  
musiconhold, agi/fastagi... New interesting challenges.

So take one of these standard rack servers from HP and run a telco for  
a small city on one box. While you're at it, buy a spare one, hardware  
can fail ( ;-) ).
But don't say that Asterisk does not scale well. Those times are gone.

/Olle

---
* Olle E Johansson - oej at edvina.net
* Open Unified Communication - SIP & XMPP projects