[Asterisk-Users] Agent/Queue Scalability (Formerly: UPDATE - 512 Calls...)

[Matt Roth]

List users,

Please review this exchange between myself and Matt Florell.  I am 
looking for ANY data concerning Asterisk servers running standard Agents 
and Queues.  Hardware configurations, software configurations, Asterisk 
configurations (especially the number of agents and queues), and 
descriptions of identified bottlenecks would be ideal.

For perspective, we have scaled a singler Asterisk server (4 x 3.16 GHz 
Xeon processors, 4 GB RAM) to 256 simultaneous calls with digital 
recording at 70% idle and to 512 simultaneous calls with digital 
recording at 20% idle.  I would like to know what to expect when we add 
agent channels to handle these calls directed to them via queues in a 
standard inbound call center environment.

I would GREATLY appreciate any user experiences or knowledge you can share.

============================================================

Matt,

Thank you so much for all of your help.  I hope I can reciprocate in the 
future.

============================================================

> On 10/4/05, *Matt Roth* <mroth at imminc.com <mailto:mroth at imminc.com>> 
> wrote:
>
>     > We do not use Asterisk Queues or Agents because we found them
>     limiting
>     > in terms of functionality and scalability as well as not being
>     as open
>     > to call manipulation as the system we built is. Because of this we
>     > haven't really used the Queue stat analysis tools out there any more
>     > than to look at the kind of stats they generate.
>
>     I figured as much, but with your experience I didn't think it
>     would hurt
>     to ask.  Your statement concerning scalability being a problem with
>     Asterisk Queues and Agents concerns me, because we are planning on
>     using
>     both in our large scale (250-500 concurrent calls) call center setup.
>     Could you expound on it?  Are they resource intensive at large
>     numbers
>     of calls, does the code have hard limits, etc?  We've dealt with the
>     scalability issue of the Monitor application and thought that would be
>     the last such hurdle.  The Asterisk source code looks to be very well
>     written and I was hoping that the other applications that aren't bound
>     to disk I/O do not introduce other bottlenecks.
>
>
> I don't know of any installation that has over 100 agents on a single 
> Asterisk server. You may want to contact the GnuDialer guys, because 
> they thought that they would be able to take a quad Xeon server and 
> put 150 agents using agent/queue on it but they never were able to 
> reliably get over 50 agents on that server. the agent/queue functions 
> on Asterisk really are much more CPU-intensive than just a standard 
> SIP stream. In our initial test 2 years ago the agent/queue load was 
> actually not much less than the meetme architecture that we ended up 
> using and with meetme you have a lot more control and functionality. 
> Your best bet may be to have several agent servers that get their 
> calls sent to them from your main server through IAX2 channels or 
> something like that.

It may be wishful thinking, but 50 seemed to be the magic number 
associated with digital recording scalability.  We have contacted the 
GNU Dialer team but if they do not respond, is anyone aware if they were 
trying to record the agent channels?

If this is strictly an Agents/Queues issue, does anyone know where the 
CPU spikes occur?  For example, will we see significant resource 
consumption by simply logging in a large number of agents to the 
Asterisk server or does the load occur as calls are routed through the 
queues?

We'd like to avoid a server farm if possible.

>     Are there other Asterisk applications that present serious scaling
>     issues?  We were hoping that our hardware (four 3.16 GHz Intel Xeon
>     processors and 4 GB of RAM) would help us overcome most of them.
>
>
> agent/queue  has a lot of functionality (you can see that in the code) 
> and it is not exactly designed for  a low memory/CPU footprint. Other 
> than that, if you want to be as optimized as possible, don't use AGI 
> scripts at all and deactivate every Asterisk module you can.

Done and done.

Are there any metrics concerning Agents/Queues and the required 
memory/CPU for varying numbers of each?  Do they scale linearly?  It 
seems that a large problem with scaling Asterisk is that numbers such as 
these are hard to come by.

>     > Could you tell me a little more about what it is exactly that
>     you are
>     > trying to build?
>
>     Sure.  We are developing a three server system to handle inbound
>     calling
>     in a call center environment:
>
>     1) Asterisk Server
>     2) Digital Recording Server
>     3) Reporting Server
>
>     The Asterisk Server itself performs no codec translations and no DSP.
>     It will be connected to a Cisco AS5400HPX Universal Gateway that
>     terminates the Ts and handles all TDM to VoIP (SIP)
>     processing.  As you
>     know it's a pretty beefy box and we've tried to reserve as many
>     resources on it as possible for running Asterisk and its applications.
>     We won't be doing call conferencing on it, but we are planning on
>     having
>     multiple queues with music on hold (our music files are converted
>     to raw
>     files and do not require any decoding) and queue announcements.  The
>     majority of calls will be digitally recorded as well, but this
>     involves
>     no transcoding and no writes to the local disk.  Some calls will also
>     involve transfers to outsides lines, three way calling, and unattended
>     monitoring via the app_chanspy.so module.  We are looking to scale to
>     500 agents spread across multiple queues, but I believe our initial
>     numbers will be closer to 200 agents.
>
>     Currently, we are running a beta of Asterisk Business Edition with a
>     license limit of 1000 simultaneous calls.  We are using Asterisk's
>     standard queues, but would like to move to realtime/dynamic queues in
>     the future.
>
>     The Digital Recording Server receives the leg files at the end of each
>     call and is responsible for mixing, indexing, and eventually
>     archiving
>     them.  The digital recordings will be available for downloading
>     via FTP
>     from this server as well.
>
>     The Reporting Server is the biggest unknown at this point.  Currently,
>     it has a mySQL database that receives call detail records from
>     Asterisk
>     via the cdr_addon_mysql.so module.  AstManProxy also runs on this box
>     and may be used to add additional data to the mySQL database.  Any
>     real-time or next-day reporting will be created from this database, so
>     that Asterisk itself is not affected by the reporting process.
>
>     Please let me know if you would like any more specific details.  I
>     would
>     be happy to supply them to you.
>
>
>
> In VICIDIAL we really haven't optimized it for inbound centers of your 
> size, but we are planning on implementing a way of having a single 
> large inbound queue that would allow agents from any server to take 
> calls from it.

Good luck with your project!

> You certainly have your work cut out for you, lwt me know how it goes.

Remember when people said computers would make our lives easier?   = )

Sincerely,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer