[asterisk-users] Asterisk AGI issues (at high load)

[Steve Edwards]

On Thu, 4 Jun 2009, Deepak wrote:

> Hi, we are experiencing a strange issue and I am hoping someone can 
> point me to the right direction or help out with some pointers.
>
> We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 
> T1's for a total of DAHDI 96 channels.
>
> We have an agi application (php) that acts as a kind of a calling card 
> application.
>
> All users are SIP users that make calls and asterisk then bridges the 
> calls over T1 to the PSTN (after running agi script). Codec between SIP 
> clients and Asterisk is G729 (96 G729 licenses installed on Asterisk).
>
> System has 12 GB of RAM.
>
> What we are seeing is the following:
>
> 1) All is fine till about 70 calls (70 active calls as displayed by core 
> show channels). But as soon as traffic increases, we see the the a lot 
> of AGI Processes are running and the *number of active calls *keeps on 
> GROWING to 120, 130, ... . THis makes no sense since we can have a 
> maximum of 96 active calls (4 T1). People calling start experiencing 
> delays (10-15 seconds before calls are answered or connected, etc.). 
> Upon doing a "ps -ef" we see a lot of AGI processes running that seem to 
> keep growing
>
> 2) At that state, if we "Decrease" traffic, the *number of active 
> calls*starts going down and the system becomes "normal" again.
>
> 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI 
> processes are not exiting when traffic increases. But as soon as traffic 
> decreases things become normal.
>
> 4) We did follow standard suggestions for AGI scripts (exit, Softhangup, 
> etc)
>
> 5) We checked and the CPU utilization seems to be 25% idle at high 
> loads.. so it does not seem like a CPU issue. We have a dual core 3.0GHz 
> intel processor and based upon benchmarking info, we should be easily 
> able to handle 96 calls (with transcoding).
>
> 6) We see that each AGI process takes up 149 MB of Virtual Memory when 
> it is forked for each call.
>
> 7) Asterisk is taking up 500 MB Of VM.
>
> *Any* suggestions/help will be appreciated and I'd be happy to even pay 
> if someone is ready to help us resolve this issue we are having.
>
> Thanks

I'm just a 1.2 Luddite, but here's my two cents:

A couple of observations...

1) PHP would not get my vote for a "high load" environment. You can 
execute xxx AGI's in C before a single PHP script loads the interpreter 
and parses. Once you get over the startup "overhead," the difference will 
not be so significant.

2) 12GB strikes me as just a ton of RAM. I have a "chat" system that 
processes several times that number of simultaneous callers, running 
dozens of AGIs (in C) per call. The Asterisk process (according to "top") 
consumes less than 200mb. The AGI's (when I can "catch" one on top -- they 
execute so quickly) take less than 10mb. The host only has 2GB and doesn't 
swap.

3) 500 MB for Asterisk seems huge.

4) 149 MB for an AGI seems huge. Are you reading a bunch of stuff from the 
database?

5) Are you processing the "HUP" signal correctly in your AGI?

I'm guessing you have database issues.

I think your AGI is reading too much data from the database (149MB).

I think your database design is not handling locking well and could 
benefit from a bit of re-design, indexing, pruning, etc.

What can you tell us about the database?

What database is it? (MySQL?)

Is it on the same host?

What statistics have you collected from the database?

How many rows are in your tables?

What does your AGI do?

Thanks in advance,
------------------------------------------------------------------------
Steve Edwards      sedwards at sedwards.com      Voice: +1-760-468-3867 PST
Newline                                             Fax: +1-760-731-3000