[asterisk-users] Asterisk AGI issues (at high load)

[Jeff LaCoursiere]

On Thu, 4 Jun 2009, Deepak wrote:

> Hi, we are experiencing a strange issue and I am hoping someone can point me
> to the right direction or help out with some pointers.
>
> We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4
> T1's for a total of DAHDI 96 channels.
>
> We have an agi application (php) that acts as a kind of a calling card
> application.
>
> All users are SIP users that make calls and asterisk then bridges the calls
> over T1 to the PSTN (after running agi script). Codec between SIP clients
> and Asterisk is G729 (96 G729 licenses installed on Asterisk).
>
> System has 12 GB of RAM.
>
>
> What we are seeing is the following:
>
> 1) All is fine till about 70 calls (70 active calls as displayed by core
> show channels). But as soon as traffic increases, we see the the a lot of
> AGI Processes are running and the *number of active calls *keeps on GROWING
> to 120, 130, ... . THis makes no sense since we can have a maximum of 96
> active calls (4 T1). People calling start experiencing delays (10-15 seconds
> before calls are answered or connected, etc.). Upon doing a "ps -ef" we see
> a lot of AGI processes running that seem to keep growing
>
> 2) At that state, if we "Decrease" traffic, the *number of active
> calls*starts going down and the system becomes "normal" again.
>
> 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI
> processes are not exiting when traffic increases. But as soon as traffic
> decreases things become normal.
>
> 4) We did follow standard suggestions for AGI scripts (exit, Softhangup,
> etc)
>
> 5) We checked and the CPU utilization seems to be 25% idle at high loads..
> so it does not seem like a CPU issue. We have a dual core 3.0GHz intel
> processor and based upon benchmarking info, we should be easily able to
> handle 96 calls (with transcoding).
>
> 6) We see that each AGI process takes up 149 MB of Virtual Memory when it is
> forked for each call.
>
> 7) Asterisk is taking up 500 MB Of VM.
>
>
> *Any* suggestions/help will be appreciated and I'd be happy to even pay if
> someone is ready to help us resolve this issue we are having.
>

If each of 96 processes takes up 150MB you would need 15GB of RAM!  This 
is only because you are trying to do this with PHP, which is crazy. 
Rewrite your AGI in C.  My guess is you are dealing with large swap 
issues, which is degenerating into thrashing your disk.

I also run a calling card operation... my AGI is in C :)

j