[asterisk-users] Scaling Asterisk: Dual-Core CPUs not yielding gains at high call volumes

[Stephen Davies]

On 01/06/07, Matthew J. Roth <mroth at imminc.com> wrote:
>   Mon Apr  2 12:15:01 EDT 2007
>   Idle (sar -P ALL 60 14) (60 seconds 14 slices)
>   Linux 2.6.12-1.1376_FC3smp (4core.imminc.com)         04/02/07
>
>   12:24:01          CPU     %user     %nice   %system   %iowait     %idle
>   12:25:02          all     14.97      0.03     34.25      0.92     49.82
>   12:25:02            0      8.83      0.05     33.60      1.28     56.24
>   12:25:02            1     17.50      0.02     34.60      0.57     47.32
>   12:25:02            2     19.94      0.02     33.52      1.31     45.22
>   12:25:02            3     13.62      0.02     35.29      0.52     50.55
>
>   Thu May 10 15:30:01 EDT 2007
>   Idle (sar -P ALL 60 14) (60 seconds 14 slices)
>   Linux 2.6.12-1.1376_FC3smp (8core.imminc.com)         05/10/07
>
>   15:38:01          CPU     %user     %nice   %system   %iowait     %idle
>   15:39:01          all      2.47      0.01     48.29      0.00     49.23
>   15:39:01            0      2.92      0.00     53.17      0.00     43.91
>   15:39:01            1      2.98      0.00     48.68      0.02     48.33
>   15:39:01            2      2.47      0.02     48.61      0.00     48.91
>   15:39:01            3      2.27      0.00     48.35      0.00     49.38
>   15:39:01            4      2.38      0.02     47.38      0.00     50.22
>   15:39:01            5      2.37      0.02     46.94      0.00     50.67
>   15:39:01            6      2.23      0.02     46.63      0.00     51.12
>   15:39:01            7      2.17      0.02     46.54      0.00     51.27


Have you got, or could you install oprofile?

That will give you a LOT of information as to where your CPUs are
spending their time,

One guess is that you could be hitting contention in the kernel with
all the cores contending for some scarce resource.  So your cores
can't execute because they are waiting on some kernel mutex for access
to some resource.  That would account for the increase in system time
- oprofile would show where in the kernel they are spending time
(where those 50%ishes are going).

Steve Uhler at Sun has been studying this on his big multi-core Sparc
boxes so he can probably contribute some insight.  Hope you don't mind
a cc, Steve.  We're talking about Asterisk/Linux running out of
scaling on an 8 core box.

Steve