[asterisk-users] Scaling Asterisk: High volume benchmarks (0
to 450 calls)
Remco Post
remco at pipsworld.nl
Thu Jun 7 01:35:57 CDT 2007
Matthew J. Roth wrote:
> List users,
>
> This post contains the benchmarks for Asterisk at high call volumes on a
> 4 CPU, dual-core (8 cores total) server. It's a continuation of the
> posts titled "Scaling Asterisk: Dual-Core CPUs not yielding gains at
> high call volumes". They contain a fair amount of information,
> including details about our servers and the software on them. I'm happy
> to answer any questions you might have, but please take a moment to
> review those posts to make sure they don't contain the information
> you're seeking.
I guess that if I read these stats correctly, the bottleneck for * is
not so much cpu power, it's the cpu cache. As I see it, the cpu cache
becomes far less efficient for larger call volumes, eg. the cache is
unable to keep the most frequently used code and data in cache, due to
the sheer amount of call date going through the cpu. I guess that you do
have some gain from going from single core to dual-core but is dwarfed
by the very limited effect on the cache.
But that is just a guess. Maybe for pure voip solutions cpu's with a
huge cache like eg Power5+ would perform much better that ia32/x64 cpu's.
>
> Thank you,
>
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
>
>
> Conclusions
> -----------
> Once again, I'm presenting the conclusions first. Scroll down if you're
> more interested in the raw data.
>
> 1. Asterisk scales quite well up to a certain number of calls. At this
> point, the cost in CPU cycles per call starts to increase more
> drastically. A graph of the Avg Used% values can be used to demonstrate
> this. It can be described as consisting of two roughly linear
> segments. The first segment is from 0 to 110 calls. The rest of the
> graph is a second, steeper segment. This is not entirely true, as in
> fact each new call costs a little more than the last, but it is a useful
> simplification.
> 2. Even at very high call volumes, Asterisk uses less than 512 KB of
> memory. 2 GB of RAM would probably avoid swapping and excessive disk
> activity on most Asterisk installations.
> 3. Future benchmarks should be based on the number of active channels,
> not active calls.
>
> I'm relying on you to point out my mistakes and omissions, so please
> take a look at the data and respond with your own analysis and conclusions.
>
> Benchmarking Methodology
> ------------------------
> The benchmarks are based on data I collected over the period of
> 5/12/2007 to 05/30/2007 from two production servers used in our inbound
> call center. The servers are identical 8-core Dell PowerEdge 6850s as
> documented in my prior posts. They are meant to be used as a
> primary/backup pair, but both were used in production in order to rule
> out a hardware failure as the cause of our scaling issues.
>
> The data was collected by a bash script executed from cron every 2
> minutes. This script utilizes some basic Linux tools (such as sar,
> free, df, and the proc filesystem) to record information about the
> system, and 'asterisk -rx "show channels"' to record information about
> the number of active calls and channels within Asterisk.
>
> Unfortunately, the sample sizes this produced are relatively small for
> the 300-450 call range. This is due to two factors:
>
> 1. The majority of the time we don't operate at such high call volumes.
> 2. Asterisk intermittently fails to report call and channel statistics
> when the CPU idle is low.
>
> This means that the benchmarking results are somewhat erratic for the
> 300-450 call range. The good news is that they are pretty consistent
> for 0 to 300 calls, and I'd imagine that covers the range most people
> are interested in.
>
> Keep in mind that the impetus behind this benchmarking was the lack of a
> performance boost on the dual-core server at high call volumes, so the
> high call range may also be skewed by whatever bottleneck is being hit
> on the 8-core servers. In the near future, we will be adding one of our
> 4-core PowerEdge 6850s to our production environment. I'll collect and
> analyze the same data, which I believe will show similar performance (as
> defined by cumulative idle CPU percentage) at around 200-300 calls.
>
> In the end, I hope to understand this problem well enough to overcome it
> or determine what the optimal point is for achieving the highest call
> volume without over-dimensioning the hardware.
>
> Call Types and a Note on Channels
> ---------------------------------
> All of the calls are SIP-to-SIP using the uLaw codec. The vast majority
> of the calls are either in queue or connected to an agent, but there are
> also a small number of regular outbound calls and transfers. Every call
> that is connected to an agent is recorded via the Monitor() application
> in PCM format to a RAM disk. In short, there was no transcoding,
> protocol bridging, or TDM hardware involved on the servers being
> benchmarked.
>
> At any given time, the makeup of the calls varied (ie. calls in queue
> vs. calls connected to agents). The calls connected to agents involve
> bridging two SIP channels, so they are more resource intensive. This
> means that the number of active channels is probably a better base
> benchmarking unit than the number of active calls. Fortunately, the
> ratio of calls to channels is somewhat consistent so this round of
> benchmarking still produced useful results.
>
> Future Tests
> ------------
> I'm aware that using a live environment isn't ideal for testing. I have
> some ideas for setting up more controlled tests using SIPp, VICIDIAL, or
> call files. I think I have the necessary hardware, but I haven't had
> the time to do much research, let alone implement anything.
>
> If anyone has any ideas or suggestions in this realm, I'd really
> appreciate hearing them.
>
> The Numbers
> -----------
>
> DC - Production Inbound Call Center Environment
> ===============================================
>
> Calls Avg Chans Calls:Chans Avg Idle% Avg Used% MHz/Call MHz/Chan
> Avg Mem (KB) Sample Size
> ----- --------- ----------- --------- --------- -------- --------
> ------------ -----------
> 0 0.06 ----- 99.61 0.39 -----
> ----- 302509 6441
> 10 13.28 1:1.3 99.45 0.55 3.84
> 2.89 350184 71
> 20 31.72 1:1.6 98.80 1.20 9.72
> 6.13 382557 54
> 30 48.52 1:1.6 97.93 2.07 13.44
> 8.31 338624 65
> 40 63.97 1:1.6 97.16 2.84 14.70
> 9.19 325619 34
> 50 80.23 1:1.6 96.45 3.55 15.17
> 9.45 344405 40
> 60 89.97 1:1.5 96.15 3.85 13.84
> 9.23 375178 34
> 70 99.69 1:1.4 94.96 5.04 15.94
> 11.19 377010 35
> 80 114.84 1:1.4 94.07 5.93 16.62
> 11.58 377977 56
> 90 130.43 1:1.4 93.30 6.70 16.83
> 11.61 408516 30
> 100 145.13 1:1.5 91.71 8.29 18.96
> 13.06 377776 24
> 110 167.15 1:1.5 90.36 9.64 20.18
> 13.28 372274 26
> 120 198.24 1:1.7 87.23 12.77 24.76
> 14.99 366577 21
> 130 220.78 1:1.7 84.07 15.93 28.69
> 16.89 400740 18
> 140 229.40 1:1.6 83.92 16.08 26.90
> 16.41 423768 15
> 150 253.00 1:1.7 82.17 17.83 27.90
> 16.54 399702 8
> 160 279.40 1:1.7 75.98 24.02 35.45
> 20.30 411208 10
> 170 300.81 1:1.8 75.50 24.50 34.04
> 19.24 409009 21
> 180 315.45 1:1.8 74.08 25.92 34.04
> 19.42 442198 22
> 190 331.94 1:1.7 69.01 30.99 38.65
> 22.12 432958 17
> 200 354.04 1:1.8 66.22 33.78 40.07
> 22.63 426874 27
> 210 366.30 1:1.7 63.84 36.16 40.88
> 23.44 405001 43
> 220 384.72 1:1.7 60.36 39.64 42.82
> 24.49 407868 46
> 230 397.58 1:1.7 57.65 42.35 43.78
> 25.33 404447 36
> 240 404.92 1:1.7 57.29 42.71 42.32
> 25.08 399253 26
> 250 428.28 1:1.7 52.84 47.16 44.90
> 26.21 427255 25
> 260 426.58 1:1.6 53.05 46.95 42.98
> 26.20 434245 19
> 270 437.23 1:1.6 53.14 46.86 41.31
> 25.51 409633 13
> 280 449.38 1:1.6 50.79 49.21 41.85
> 26.07 431533 13
> 290 469.30 1:1.6 44.58 55.42 45.54
> 28.14 393659 10
> 300 473.71 1:1.6 46.47 53.53 42.51
> 26.92 402571 7
> 310 478.13 1:1.5 42.84 57.16 43.95
> 28.50 409747 8
> 320 493.63 1:1.5 36.50 63.50 47.33
> 30.68 392383 8
> 330 512.75 1:1.6 36.55 63.45 45.86
> 29.52 417522 4
> 340 525.67 1:1.5 25.72 74.28 52.16
> 33.74 394472 3
> 350 527.00 1:1.5 30.42 69.58 47.44
> 31.51 422372 2
> 360 538.29 1:1.5 27.83 72.17 47.85
> 32.00 400915 7
> 370 544.80 1:1.5 31.16 68.84 44.40
> 30.15 418727 5
> 380 573.00 1:1.5 22.22 77.78 48.88
> 32.41 408265 3
> 390 594.00 1:1.5 8.98 91.02 55.77
> 36.62 405802 2
> 400 ------ ----- ----- ----- -----
> ----- ------ 0
> 410 601.00 1:1.5 14.07 85.93 50.07
> 34.16 379880 1
> 420 586.00 1:1.4 25.87 74.13 42.14
> 30.20 375936 1
> 430 ------ ----- ----- ----- -----
> ----- ------ 0
> 440 ------ ----- ----- ----- -----
> ----- ------ 0
> 450 ------ ----- 4.33 95.67 50.82
> ----- 413496 2
--
Remco Post
"I didn't write all this code, and I can't even pretend that all of it
makes sense." -- Glen Hattrup
More information about the asterisk-users
mailing list