[asterisk-users] Scaling Asterisk: Dual-Core CPUs not yielding gains at high call volumes

[Matthew J. Roth]

John Hughes wrote:
> Matthew J. Roth wrote:
>   
>> As far as Asterisk is concerned, at low call volumes the dual-core
>> server outperforms the single-core server at a similar rate.
>>     
> Outperforms in what sense?
>   
At low call volumes the cumulative CPU utilization, expressed as a 
percentage of available processor, is lower on the dual-core server.  
This is the expected behavior.  What I'm proposing (and hope to back up 
with numbers in the near future) is that as the number of calls rises to 
the 300-400 range, the cumulative CPU utilization starts to approach the 
same number on both servers.

Unfortunately, I wasn't collecting as much data when the single-core 
server was in production so some of this is speculation based on my 
memory of the system's performance.  The environment is also different, 
because we have added agents so the ratio of calls connected vs. calls 
in queue has changed.  Nonetheless, the dual-core server is not 
performing anywhere near our expectations.

Here is something we recently noticed that may explain why the dual-core 
server is under-performing at high call volumes.  The following numbers 
were collected off both servers while they were in production.  Note 
that while they have similar cumulative idle values, the ratio of system 
time to user time on the single-core server is roughly 2.3 to 1, but on 
the dual-core server it is roughly 19.6 to 1.  I'm not quite sure what 
to make of this, but it seems to be very relevant to the problem.

  Mon Apr  2 12:15:01 EDT 2007
  Idle (sar -P ALL 60 14) (60 seconds 14 slices)
  Linux 2.6.12-1.1376_FC3smp (4core.imminc.com)         04/02/07

  12:24:01          CPU     %user     %nice   %system   %iowait     %idle
  12:25:02          all     14.97      0.03     34.25      0.92     49.82
  12:25:02            0      8.83      0.05     33.60      1.28     56.24
  12:25:02            1     17.50      0.02     34.60      0.57     47.32
  12:25:02            2     19.94      0.02     33.52      1.31     45.22
  12:25:02            3     13.62      0.02     35.29      0.52     50.55

  Thu May 10 15:30:01 EDT 2007
  Idle (sar -P ALL 60 14) (60 seconds 14 slices)
  Linux 2.6.12-1.1376_FC3smp (8core.imminc.com)         05/10/07

  15:38:01          CPU     %user     %nice   %system   %iowait     %idle
  15:39:01          all      2.47      0.01     48.29      0.00     49.23
  15:39:01            0      2.92      0.00     53.17      0.00     43.91
  15:39:01            1      2.98      0.00     48.68      0.02     48.33
  15:39:01            2      2.47      0.02     48.61      0.00     48.91
  15:39:01            3      2.27      0.00     48.35      0.00     49.38
  15:39:01            4      2.38      0.02     47.38      0.00     50.22
  15:39:01            5      2.37      0.02     46.94      0.00     50.67
  15:39:01            6      2.23      0.02     46.63      0.00     51.12
  15:39:01            7      2.17      0.02     46.54      0.00     51.27

>> I'm working on a follow-up post that will demonstrate this with some
>> benchmarks for a small number of calls in various scenarios on each
>> machine.  However, to our surprise as the number of concurrent calls
>> increases, the performance gains begin to flatten out.  In fact, it
>> seems that somewhere between 200 and 300 calls, the two servers start
>> to exhibit similar idle times despite one of them having twice as many
>> cores.
>>     
> What do you mean by "idle" here?
Idle percentage as shown in top's or sar's cumulative view.

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer