[asterisk-users] Scaling Asterisk: High volume benchmarks (0 to 450
calls)
Matthew J. Roth
mroth at imminc.com
Wed Jun 6 14:35:24 CDT 2007
List users,
This post contains the benchmarks for Asterisk at high call volumes on a
4 CPU, dual-core (8 cores total) server. It's a continuation of the
posts titled "Scaling Asterisk: Dual-Core CPUs not yielding gains at
high call volumes". They contain a fair amount of information,
including details about our servers and the software on them. I'm happy
to answer any questions you might have, but please take a moment to
review those posts to make sure they don't contain the information
you're seeking.
Thank you,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
Conclusions
-----------
Once again, I'm presenting the conclusions first. Scroll down if you're
more interested in the raw data.
1. Asterisk scales quite well up to a certain number of calls. At
this point, the cost in CPU cycles per call starts to increase more
drastically. A graph of the Avg Used% values can be used to demonstrate
this. It can be described as consisting of two roughly linear
segments. The first segment is from 0 to 110 calls. The rest of the
graph is a second, steeper segment. This is not entirely true, as in
fact each new call costs a little more than the last, but it is a useful
simplification.
2. Even at very high call volumes, Asterisk uses less than 512 KB of
memory. 2 GB of RAM would probably avoid swapping and excessive disk
activity on most Asterisk installations.
3. Future benchmarks should be based on the number of active channels,
not active calls.
I'm relying on you to point out my mistakes and omissions, so please
take a look at the data and respond with your own analysis and conclusions.
Benchmarking Methodology
------------------------
The benchmarks are based on data I collected over the period of
5/12/2007 to 05/30/2007 from two production servers used in our inbound
call center. The servers are identical 8-core Dell PowerEdge 6850s as
documented in my prior posts. They are meant to be used as a
primary/backup pair, but both were used in production in order to rule
out a hardware failure as the cause of our scaling issues.
The data was collected by a bash script executed from cron every 2
minutes. This script utilizes some basic Linux tools (such as sar,
free, df, and the proc filesystem) to record information about the
system, and 'asterisk -rx "show channels"' to record information about
the number of active calls and channels within Asterisk.
Unfortunately, the sample sizes this produced are relatively small for
the 300-450 call range. This is due to two factors:
1. The majority of the time we don't operate at such high call volumes.
2. Asterisk intermittently fails to report call and channel statistics
when the CPU idle is low.
This means that the benchmarking results are somewhat erratic for the
300-450 call range. The good news is that they are pretty consistent
for 0 to 300 calls, and I'd imagine that covers the range most people
are interested in.
Keep in mind that the impetus behind this benchmarking was the lack of a
performance boost on the dual-core server at high call volumes, so the
high call range may also be skewed by whatever bottleneck is being hit
on the 8-core servers. In the near future, we will be adding one of our
4-core PowerEdge 6850s to our production environment. I'll collect and
analyze the same data, which I believe will show similar performance (as
defined by cumulative idle CPU percentage) at around 200-300 calls.
In the end, I hope to understand this problem well enough to overcome it
or determine what the optimal point is for achieving the highest call
volume without over-dimensioning the hardware.
Call Types and a Note on Channels
---------------------------------
All of the calls are SIP-to-SIP using the uLaw codec. The vast majority
of the calls are either in queue or connected to an agent, but there are
also a small number of regular outbound calls and transfers. Every call
that is connected to an agent is recorded via the Monitor() application
in PCM format to a RAM disk. In short, there was no transcoding,
protocol bridging, or TDM hardware involved on the servers being
benchmarked.
At any given time, the makeup of the calls varied (ie. calls in queue
vs. calls connected to agents). The calls connected to agents involve
bridging two SIP channels, so they are more resource intensive. This
means that the number of active channels is probably a better base
benchmarking unit than the number of active calls. Fortunately, the
ratio of calls to channels is somewhat consistent so this round of
benchmarking still produced useful results.
Future Tests
------------
I'm aware that using a live environment isn't ideal for testing. I have
some ideas for setting up more controlled tests using SIPp, VICIDIAL, or
call files. I think I have the necessary hardware, but I haven't had
the time to do much research, let alone implement anything.
If anyone has any ideas or suggestions in this realm, I'd really
appreciate hearing them.
The Numbers
-----------
DC - Production Inbound Call Center Environment
===============================================
Calls Avg Chans Calls:Chans Avg Idle% Avg Used% MHz/Call MHz/Chan
Avg Mem (KB) Sample Size
----- --------- ----------- --------- --------- -------- --------
------------ -----------
0 0.06 ----- 99.61 0.39 -----
----- 302509 6441
10 13.28 1:1.3 99.45 0.55 3.84
2.89 350184 71
20 31.72 1:1.6 98.80 1.20 9.72
6.13 382557 54
30 48.52 1:1.6 97.93 2.07 13.44
8.31 338624 65
40 63.97 1:1.6 97.16 2.84 14.70
9.19 325619 34
50 80.23 1:1.6 96.45 3.55 15.17
9.45 344405 40
60 89.97 1:1.5 96.15 3.85 13.84
9.23 375178 34
70 99.69 1:1.4 94.96 5.04 15.94
11.19 377010 35
80 114.84 1:1.4 94.07 5.93 16.62
11.58 377977 56
90 130.43 1:1.4 93.30 6.70 16.83
11.61 408516 30
100 145.13 1:1.5 91.71 8.29 18.96
13.06 377776 24
110 167.15 1:1.5 90.36 9.64 20.18
13.28 372274 26
120 198.24 1:1.7 87.23 12.77 24.76
14.99 366577 21
130 220.78 1:1.7 84.07 15.93 28.69
16.89 400740 18
140 229.40 1:1.6 83.92 16.08 26.90
16.41 423768 15
150 253.00 1:1.7 82.17 17.83 27.90
16.54 399702 8
160 279.40 1:1.7 75.98 24.02 35.45
20.30 411208 10
170 300.81 1:1.8 75.50 24.50 34.04
19.24 409009 21
180 315.45 1:1.8 74.08 25.92 34.04
19.42 442198 22
190 331.94 1:1.7 69.01 30.99 38.65
22.12 432958 17
200 354.04 1:1.8 66.22 33.78 40.07
22.63 426874 27
210 366.30 1:1.7 63.84 36.16 40.88
23.44 405001 43
220 384.72 1:1.7 60.36 39.64 42.82
24.49 407868 46
230 397.58 1:1.7 57.65 42.35 43.78
25.33 404447 36
240 404.92 1:1.7 57.29 42.71 42.32
25.08 399253 26
250 428.28 1:1.7 52.84 47.16 44.90
26.21 427255 25
260 426.58 1:1.6 53.05 46.95 42.98
26.20 434245 19
270 437.23 1:1.6 53.14 46.86 41.31
25.51 409633 13
280 449.38 1:1.6 50.79 49.21 41.85
26.07 431533 13
290 469.30 1:1.6 44.58 55.42 45.54
28.14 393659 10
300 473.71 1:1.6 46.47 53.53 42.51
26.92 402571 7
310 478.13 1:1.5 42.84 57.16 43.95
28.50 409747 8
320 493.63 1:1.5 36.50 63.50 47.33
30.68 392383 8
330 512.75 1:1.6 36.55 63.45 45.86
29.52 417522 4
340 525.67 1:1.5 25.72 74.28 52.16
33.74 394472 3
350 527.00 1:1.5 30.42 69.58 47.44
31.51 422372 2
360 538.29 1:1.5 27.83 72.17 47.85
32.00 400915 7
370 544.80 1:1.5 31.16 68.84 44.40
30.15 418727 5
380 573.00 1:1.5 22.22 77.78 48.88
32.41 408265 3
390 594.00 1:1.5 8.98 91.02 55.77
36.62 405802 2
400 ------ ----- ----- ----- -----
----- ------ 0
410 601.00 1:1.5 14.07 85.93 50.07
34.16 379880 1
420 586.00 1:1.4 25.87 74.13 42.14
30.20 375936 1
430 ------ ----- ----- ----- -----
----- ------ 0
440 ------ ----- ----- ----- -----
----- ------ 0
450 ------ ----- 4.33 95.67 50.82
----- 413496 2
More information about the asterisk-users
mailing list