<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE></TITLE>
<META content="MSHTML 5.00.3700.6699" name=GENERATOR></HEAD>
<BODY bgColor=#ffffff text=#000000>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN class=529265714-12082004>We
have the luxury(or curse depending on your point of view) of having over 100
phones connected to Sipura SIP adapters and another few dozen other SIP phones,
and when we do load tests we will actually use live SIP calls from SIP devices.
Another way to simulated traffic (if you have a quad T1 card) would be to loop
two T1 ports into the other two T1 ports and use the manager to dial out one Zap
channel that would go into another Zap channel as an incoming call. This would
at least give you 96 Zap channels of load that would help you do some basic load
testing. You may be able to do the same thing with SIP if you use a machine as a
port forwarder and have it send SIP calls back to the Asterisk server, but I'm
not exactly sure how you would set that up.</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN class=529265714-12082004>As for
where the actual bottleneck that locks the manager up is, I really don't know.
The last time I tried to figure it out I had full debugging turned on with 50
SIP channels running and I ended up with hundreds of lines of debug output per
second and I never was able to figure out exactly what caused it except for it
seemed to be something to do with an Asterisk routine being run on all SIP
channels at the same time intermittantly. At that point I made my Queue system a
lot more fault tolerant and I haven't had any problems since.
</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN class=529265714-12082004>And I
don't think I mentioned it yet, but our asterisk servers run on P4 2.6GHz with
2GB RAM. We are going to try using a Prescott P4 3.2GHZ (1MB cache) next week,
I'll post if there are any significant differences between the two
setups.</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN class=529265714-12082004>Well,
I'm in the Tampa Bay area of Florida and we have a hurricane coming, so I won't
be able to stay on this thread for the next few days. I'm off to the store to
get supplies and I have to get ready to board up the
windows.</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN class=529265714-12082004>Let me
know how the testing goes,</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004>MATT---</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=529265714-12082004></SPAN></FONT> </DIV>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px">
<DIV align=left class=OutlookMessageHeader dir=ltr><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> David Pollak
[mailto:dpp-asterisk@projectsinmotion.com]<BR><B>Sent:</B> Wednesday, August
11, 2004 6:11 PM<BR><B>To:</B>
asterisk-dev@lists.digium.com<BR><B>Subject:</B> Re: [Asterisk-Dev]
Integration of AGI and Management API<BR><BR></DIV></FONT>Matt,<BR><BR>Thanks
for your very clear, well reasoned comments. <BR><BR>mattf wrote:
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<META content="MSHTML 5.00.3700.6699" name=GENERATOR>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>OK, I think I understand what you're shooting for a
little better now, you're looking to have a single app server act as a
dynamic remote dialplan for 25 asterisk servers. I like the idea, but I'd
really like to see what your results are running it on a loaded server(with
100 SIP channels running).</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT><BR></DIV></BLOCKQUOTE>Can you point me
to a load generator? Being able to simulate the 100 call load would be a
GoodThing (tm)<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>The reason I asked if you had load tested is
because the manager will spit out a large amount of linear output even at
just 30 channels of concurrent use. At 100 channels and with passing and
parsing more data because of AGI, will your app server be able to keep up
with all of the data coming in while also sending commands out and keeping
track of all of it? And will the Asterisk server be able to keep up with no
significant manager output pauses?</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT><BR></DIV></BLOCKQUOTE>My app server
can handle the traffic. The socket read thread just reads the message
from the socket, puts the message in a queue that's handled by a worker
thread. The socket read threads can be prioritized over the worker
threads. Keeping the socket read buffer clear is not a problem.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>Here are some numbers to look
at:</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>Using the example of a calling card AGI app. you
have the customer dial in, enter their phone number as a passcode, then
enter the phone number they want to call. let's say that for load testing's
sake we have 100 people dial in to use the calling card program at once. in
the first 3 seconds you will receive 3000 lines of manager output and have
to send out at least 1500 lines of manager commands and receive back another
600 lines to play a sound and prepare to collect DTMF input. Then for each
of the 20 DTMF signals sent for each of those channels you will have to
receive 5 lines output and send 5 lines input for each of the 100
channels(another 5000 lines of output and 5000 lines of input). Then you
will initiate the new call by Redirecting the SIP channel to an outbound
channel creating 10 more output lines and 6 more input lines each. and when
the calls terminate you will have another 26 lines of output. So for the
first 20 seconds at least you are looking at 15,000 lines of input/output
for the manager interface at a bare minimum. This doesn't seem like much
data when you compare it to a database or a web server, but moving 300,000
Bytes through the manager interface in 20 seconds is a rather large amount
because it has not been optimized to handle it. In this scenario you may
even run into the buffer limiter built into the manager to prevent kernel
panics, which would result in non-delivered
data.</SPAN></FONT></DIV></BLOCKQUOTE>Yep. I looked at the code. If the
write blocks for more than 1 ms, the message is thrown away. I don't
think the Asterisk write/App Server read socket will block (see the comments
above.)<BR><BR>I did a test sending MAGI commands from the App Server to
Asterisk. I was able to sustain 500 messages per second (500 AGI no-op
commands) sent to Asterisk with the response sent back to my app server.
There was no significant CPU utilization indicating that the parsing of the
text buffers was not taking much time. I sustained the 500 messages per
second over 60 seconds with no lost traffic. That's about 10K bytes per
second through the Manager. The seems like a reasonable amount of
traffic. Granted, the traffic was not on a loaded machine, but given the
lack of CPU utilization, I think it should pass the tests you've outlined
above.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>Another thing to consider is the SIP channel system
load spikes that occur on Asterisk. At intermittant times(anywhere from 5
minutes to 2 hours apart), Asterisk will perform some actions on running SIP
channels all at once. If you are running more than 24 channels at the time
this happens you will see the system load usage spike and the manager output
will pause for an amount of time proportional to the number of active
channels you have going(anywhere from 1-15 seconds on my systems). When the
connection unpausess all of the commands sent to the manager while it was
paused will all be executed at once and you will receive a flood of manager
output data really fast. Now this does not affect the dialplan operation but
it does mess with the manager. I have been able to reproduce this and have
proven that this does happen to SIP channels even in a controlled lab
environment.</SPAN></FONT></DIV></BLOCKQUOTE>Can you send me a way to
reproduce this situation? Looking at the Manager code, I can't seem to
find a place in the manager that would block based on what's happening on SIP
channels. Have you reproduced this under the debugger and looked at
what's blocking where?<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>Your suggestion is a great idea for how to have
Asterisk function more easily in a larger environment, and if it works on
loaded systems it is something that I would love to use on my systems. But I
think you need to do some load testing and see what the Asterisk Manager can
handle before you put more effort into finalizing an application that may
not work on the scale you are planning on.</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT><BR></DIV></BLOCKQUOTE>Once again,
thanks for your well reasoned comments. Once I get a load tester, I'll
try running the puppy with 100 SIP channels (maybe all running meetme) and
then send a ton of MAGI traffic and see if I can get the 500 commands per
second.<BR><BR>Thanks,<BR><BR>David<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502261@vicimail.vicimarketinggroup.com">
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004>MATT---</SPAN></FONT></DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT> </DIV>
<DIV><FONT color=#0000ff face=Arial size=2><SPAN
class=861010419-11082004></SPAN></FONT> </DIV>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px">
<DIV align=left class=OutlookMessageHeader dir=ltr><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> David Pollak [<A
class=moz-txt-link-freetext
href="mailto:dpp-asterisk@projectsinmotion.com">mailto:dpp-asterisk@projectsinmotion.com</A>]<BR><B>Sent:</B>
Wednesday, August 11, 2004 2:08 PM<BR><B>To:</B> <A
class=moz-txt-link-abbreviated
href="mailto:asterisk-dev@lists.digium.com">asterisk-dev@lists.digium.com</A><BR><B>Subject:</B>
Re: [Asterisk-Dev] Integration of AGI and Management
API<BR><BR></FONT></DIV>Matt,<BR><BR>I think we agree on the goals:<BR>
<UL>
<LI>To minimize the number of threads and sockets on the machine running
Asterisk to maximize the performance of the machine running Asterisk
<LI>To minimize the changes to Asterisk, but maximize the flexibility of
Asterisk
<LI>To minimize the number of open Manager API connections </LI></UL>My
proposal (and the code that I have already developed) allows the posting
of AGI commands from the Manager API to a given channel and sending the
response code for the commands from res_agi.c to the Manager. The
code simply pipes AGI commands from a source other than a pipe that's
dedicated to a single AGI session.<BR><BR>Put another way, AGI is a lot
like early CGI on web servers. Over time, systems like mod_perl,
etc. were developed to reduce load on systems. What the merger
between Manager API and AGI does is allows a single manager connection to
manage the execution of applications on an arbitrary number of
channels.<BR><BR>mattf wrote:<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">When you say 100 concurrent channels do you mean 50 SIP phones connected to
50 Zap channels or 100 SIP phones in conversations with 100 other end
points?
</PRE></BLOCKQUOTE>The initial system that we're building primarily
performs IVR. All of the channels will be terminated SIP calls.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">Have you done any load testing ? What kind of system configuration are you
planning on?
</PRE></BLOCKQUOTE>Each of the Asterisk servers will be P4 3.x Ghz
systems running Intel's 875 chipset.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">How exactly would you go about "creating" a calling card AGI through the
manager?
</PRE></BLOCKQUOTE>Sending (or piping) the AGI commands through the
Manager API so that they are posted to a Channel. The exec_agi()
routine in res_agi.c dequeues the AGI command from the channel rather than
reading it from a file descriptor. Once the command is read or
dequeue, it's executed and the results are sent back to the fd or sent as
a manager_event.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">What's wrong with just using AGI scripts?
</PRE></BLOCKQUOTE>Please see my original posting. With AGI
scripts, there's a seperate process created for each AGI that's
executed. Hooking the AGI process to my central J2EE server means 2
open sockets for each running AGI. Monitoring 100 channels on 25
machines means 2,500 open sockets to my J2EE server. On the other
hand, sending AGI commands via the Manager API means only 25 open sockets
for my J2EE server and only 1 open connection from the Asterisk instance
to my server.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">A busy Asterisk server with 100 concurrent manager connections will not
work, especially if you want to actually use those manager connecitons for
anything.
</PRE></BLOCKQUOTE>I think you're missing the point. There will
only be 1 open manager instance per Asterisk instance. That single
manager instance will control all the open channels.<BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">Attempting to run interactive programs like AGIs through the manager API
would complicate the managerAPI by adding many new inputs, outputs and
parsing rules to it and would slow it down even more, not to mention the
fact that you are depending on that client socket connection to keep flowing
perfectly in order to run your phone system and I can tell you that the
manager API is not fault tolerant enough to handle that kind of volume,
complexity and data-delivery-assurance with any kind of reliability.
</PRE></BLOCKQUOTE>Nope. Here's the total change to manager.c
(okay, there's one other change where the action_magi function is
registered, but that's an additional line in an array of
functions):<BR><TT>static int action_magi(struct mansession *s, struct
message *m)<BR>{<BR> char *command = astman_get_header(m,
"Command");<BR> char *channel = astman_get_header(m,
"Channel");<BR> char *id = astman_get_header(m,
"Uniqueid");<BR><BR> if (!command || ast_strlen_zero(command))
{<BR> astman_send_error(s, m, "Command not
specified");<BR> return 0;<BR> }<BR><BR> if
(!channel || ast_strlen_zero(channel)) {<BR>
astman_send_error(s, m, "Channel not specified");<BR>
return 0;<BR> }<BR><BR> if (!id || ast_strlen_zero(id))
{<BR> astman_send_error(s, m, "UniqueID not
specified");<BR> return 0;<BR> }<BR><BR>
struct ast_channel *theChan =
ast_get_channel_by_name_locked(channel);<BR><BR> if (!theChan)
{<BR> astman_send_error(s, m, "Channel not
found");<BR> return 0;<BR> }<BR><BR> //
allocate the structure here... it will be<BR> // free'ed if
enqueuing fails or if <BR> struct ast_magi *magi =
ast_magi_new();<BR><BR> if (!magi) {<BR>
astman_send_error(s, m, "MAGI not allocated -- out of
memory");<BR>
ast_mutex_unlock(&theChan->lock);<BR> return
0;<BR> }<BR><BR> // being smart, cmd is 1 char longer than
MAX_AGI_CMD_LEN<BR> // so we strncpy, and then set the last char to
0<BR> strncpy(magi->cmd, command, MAX_AGI_CMD_LEN);<BR>
magi->cmd[MAX_AGI_CMD_LEN] = 0;<BR><BR> // see above re field
lengths<BR> strncpy(magi->uniqueid, id,
MAX_MAGI_UNIQUE_ID);<BR> magi->uniqueid[MAX_MAGI_UNIQUE_ID] =
0;<BR><BR><BR> // if the add fails, this routine cleans up<BR>
// the magi<BR> int added = ast_add_magi_to_channel(theChan, magi,
0);<BR><BR> ast_mutex_unlock(&theChan->lock);<BR><BR>
if (!added) {<BR> astman_send_error(s, m, "MAGI not
added to channel");<BR> return 0;<BR>
}<BR><BR> ast_cli(s->fd, "Response:
Success\r\n"<BR> "Uniqueid:
%s\r\n\r\n",<BR> id);<BR><BR> return
0;<BR>}<BR></TT><BR>All of the parsing for the commands is done is reg_agi
and the commands are treated <B>exactly</B> like the commands that came in
from a pipe/file descriptor that the current AGI monitors.<BR><BR>If you
can find anything in this code that will reduce the stability of the
Manager or create a bottleneck, please let me
know.<BR><BR>Thanks,<BR><BR>David<BR><BR>
<BLOCKQUOTE type="cite"
cite="midDB43F516702AAF4392AA45573F181819502258@vicimail.vicimarketinggroup.com"><PRE wrap="">MATT---
-----Original Message-----
From: David Pollak [<A class=moz-txt-link-freetext href="mailto:dpp-asterisk@projectsinmotion.com">mailto:dpp-asterisk@projectsinmotion.com</A>]
Sent: Tuesday, August 10, 2004 10:58 PM
To: <A class=moz-txt-link-abbreviated href="mailto:asterisk-dev@lists.digium.com">asterisk-dev@lists.digium.com</A>
Subject: Re: [Asterisk-Dev] Integration of AGI and Management API
My application will initially have 25 Asterisk servers managed by 1
application server. That number must be scalable to at least 500 Asterisk
servers.
Given the initial 25 servers, with 100 open channels each, that's 2,500
channels to manage.
So, I've been looking over the Asterisk source and I had an idea... what
if there was a merger of the Manager API and AGI. Here's specifically
what I was thinking:
It looks like you've already done that above. You already
started using the AGI and the AMI, so that's where the
merger is. I could be wrong on this, but the rest of your
message concerns how to put these two things back together.
So you are making a circle, when a straight line would do.
I have not found a straight line from the Manager API that allows the
execution of an application on a given channel with all the normal responses
(e.g., waiting for DTMF, etc.) If you can show me how to do that, I'd
really appreciate it.
Yep... I can see originating a call to an application, but I cannot see how
to respond to an existing channel. For example, how would one create the
Calling Card sample AGI application by originating a call?
To my mind, all AGI scripts represent dynamic dial plans. I haven't done
the full analysis, but it seems that dial plans themselves are Turing
complete. However, it's really, really hard to build complex applications
using dial plans. Thus, AGI script allow for a better language to write
dynamic dial plans. My changes are simply to allow the AGI commands to be
sent to a channel via the Manager API rather than via a pipe to a separate
process. In res_agi.c, the next command is recalled from the Channel rather
than by reading from the pipe. That's the fundimental difference.
I don't think that works. I think the AGI was added to Asterisk because the
ability to control a channel via the Manager API is limitted. My changes
have simply added a new way in which an AGI script can send commands to a
Channel that's expecting AGI commands.
Thanks,
David
_______________________________________________
Asterisk-Dev mailing list
<A class=moz-txt-link-abbreviated href="mailto:Asterisk-Dev@lists.digium.com">Asterisk-Dev@lists.digium.com</A>
<A class=moz-txt-link-freetext href="http://lists.digium.com/mailman/listinfo/asterisk-dev">http://lists.digium.com/mailman/listinfo/asterisk-dev</A>
To UNSUBSCRIBE or update options visit:
<A class=moz-txt-link-freetext href="http://lists.digium.com/mailman/listinfo/asterisk-dev">http://lists.digium.com/mailman/listinfo/asterisk-dev</A>
</PRE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>