[asterisk-users] VERY HIGH LOAD AVERAGE: top - 10:27:57 up 199 days, 5:18, 2 users, load average: 67.75, 62.55, 55.75

[Steve Totaro]

On Wed, Feb 10, 2010 at 5:23 AM, RESEARCH <research at businesstz.com> wrote:
>>
>> snip
>>>>
>>>
>>> You are correct. 8 span which process up to 240 calls at pick time
>>>
>>>> If the system is actually performing fine then I'd just say that there
>>> is something about the Asterisk threads that makes them look runnable
>>> and that
>>>> accounts for the high load average. ?Is the IVR an agi or fastagi or
>>> what? -
>>>
>>> I have the agi scripts not as ivr but to help populate the required
>>> information into mysql db. Probably here is where the problem lies i
>>> have
>>> to connect and disconnect to mysql each time a call is made or a
>>> specific
>>> menu is selected
>>>
>>> Here is the script
>>> *****
>>> #!/usr/bin/perl -w
>>> use strict;
>>> use DBI();
>>> use Scalar::Util qw/weaken/;
>>>
>>> my $cdr_log_file = "/var/log/asterisk/ivr_log";
>>> my $mysql_host = "cdr01";
>>> my $mysql_db = "ivrcdrdb";
>>> my $mysql_table = "tbl_ivrcdr_details";
>>> my $mysql_user = "ivruser";
>>> my $mysql_pwd = "a09876a";
>>>
>>>
>>> my $sth;
>>>
>>> my $data0= $ARGV[0];
>>> my $data1= $ARGV[1];
>>> my $data2= $ARGV[2];
>>> my $data3= $ARGV[3];
>>> my $data4= $ARGV[4];
>>> my $data5= $ARGV[5];
>>> my $data6= $ARGV[6];
>>> my $data7= $ARGV[7];
>>>
>>>
>>> # Connect to database
>>> # print "Connecting to database...\n\n";
>>> my $dbh =
>>>
> DBI->connect("DBI:mysql:database=$mysql_db;host=$mysql_host","$mysql_user","
> $mysql_pwd",{'RaiseError'
>>> => 1});
>>>
>>> my $insert_str = "insert into $mysql_table (calldate, language, src,
>>> duration, accountcode, uniqueid, currentmenu, nextmenu) values
>>> (\"$data0\", \"$data1\", \"$data2\", \"$data3\", ?\"$data4\",
>>> \"$data5\",
>>> \"$data6\", \"$data7\");\n";
>>> ? ? ? $sth = $dbh->prepare($insert_str);
>>> ? ? ? $sth->execute();
>>>
>>> # print "\n\nOK.\n";
>>>
>>> $sth->finish();
>>> $dbh->disconnect();
>>>
>>>
>>> # Trying to resolve memory leak should it happen
>>> delete($ARGV[0]);
>>> delete($ARGV[1]);
>>> delete($ARGV[2]);
>>> delete($ARGV[3]);
>>> delete($ARGV[4]);
>>> delete($ARGV[5]);
>>> delete($ARGV[6]);
>>> delete($ARGV[7]);
>>>
>>>
>>> exit;
>>> *********************
>>>
>>>> the code path may have a "spinlock" logic to it that means that many
>>> threads
>>>> are runnable but when scheduled just go back to sleep. ?That would
>>> account for high load average with lots of spare CPU. ?If that's what is
>>> happening then I wouldn't worry much more about it.
>>>>
>>>> Regards,
>>>> Steve
>>>
>>> Regards
>>> Sam
>>
>> If I were you, and I am not and never will be, I would move over to
>> fastagi and offload all that Perl and database stuff off to a
>> designated server just to handle that stuff.
>>
>> I have had the EXACT same problem and that is how it was fixed,
>> fastagi running to a Windows box that had a process developed (written
>> in C something) by the M$ developers to hit the M$SQL databases.
>>
>> We were also doing a ton of things with the AMI which we figured out
>> how to do the same end result without banging on the AMI, such as
>> using call files rather than AMI to originate a call.
>>
>> Load avg dropped to one or under if I remember correctly.
>>
>> Thanks,
>> Steve Totaro
>>
>
> Thank you Steve for your recommendation. Ofcoz i have separate server that
> is hosting the db and i will consider doing fastagi and see it it will help
> @Phil. The credintials displayed there are dummy, so don't worry unless you
> mean something else
>
> @Steve Edward. Can you share your C agi codes? I presume what you want me to
> do is rewrite the script in C and use it as compiled binary
>
> @Tzafrir. How about this
> [ivr4 ~]# ps aux | grep D
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root      1975  0.0  0.0   3920   688 pts/4    S+   13:17   0:00 grep D
> root      3413  0.0  0.0   1832   576 ?        Ss    2009  80:58
> /usr/sbin/mDNSResponder -b -f /etc/services_mDNS
>
> I have killed that process but no changes
>
>
> @All, looks like the conclusion has been made that this is to do with AGI.
> Let me address it and see how it reacts. I shall feedback
>
> Thanks
> Sam
>

Simple experiment, move to fastagi, perl calls are killing you.