[asterisk-users] Scaling/Loadbalancing a Call Center and Redundancy

[Steve Totaro]

Right now we are all inbound and every call is recorded.

Matt Florell wrote:
> Hello,
>
> At this point in time VICIDIAL is more focused on outbound features,
> but inbound and blended capcbilities have been part of VICIDIAL for
> about two years. The most I have done inbound-only with it is 3 T1s
> with 60 agents. But for outbound and inbound agents together we have
> had upto 120 agents on one setup with 20 T1s(spread across 8 Asterisk
> servers, 2 web servers and 1 MySQL server) handling over 200,000 calls
> a day(mostly outbound of course).
>
> The inbound portion of VICIDIAL does not have customized hold music or
> periodic announcements yet, but we plan on adding those features in a
> future version as we begin to focus more on inbound in the project.
>
> We do use meetme rooms in VICIDIAL. This allows for easy third-party
> calls and multiple monitoring/manager intrusion into an agent session.
>
> The load balancing works by having the Database keep track of agents
> and calls for all of the servers and then use IAX to move the calls
> from where they originated to whoever the next available agent is, no
> matter what server they are on. So the calls can come from anywhere
> and the agents can be logged in anywhere.
>
> If you receive inbound callerID on these calls you will also be able
> to have caller information appear on the Agent's web interface. And if
> they are a customer that is already in the system it will bring up
> their existing record.
>
> As for reliability, we have not had a total system failure in the last
> 2 years(aside from long power outages and hurricane interruptions).
> MySQL can handle a tremendous volume and it is the only
> total-system-single-point-of-failure in VICIDIAL, ours never crashes.
> The web servers can be load balanced(no need for session-awareness)
> and you can use any Apache/PHP webserver that may be on your system to
> serve the AJAX-based agent web interface. As for Asterisk, we have had
> servers crash periodically(a couple crashes a month across 8 servers),
> but that is to be expected when you push tens of thousands of calls
> through each one per day.
>
> Will you be recording all calls in this setup?
>
> MATT---
>
> On 9/16/06, Steve Totaro <stotaro at asteriskhelpdesk.com> wrote:
>> Matt,
>>
>> I am sure this is a RTFM and I am pretty sure you are using meetme
>> rooms.  Just not too sure how you do the magic.
>>
>> 28 T1s with NFAS so 95 channels per trunk group, seven trunk groups =
>> 665 lines.  My client's call volume has shot from 5,000 to about 10,000
>> calls a day.  Due to recent product offerings/advertising, I expect to
>> be eating up 6 T1 (peak) by the end  of October.  They will eventually
>> have every channel in use during peaks, whether that is in November or
>> December, I am not sure.  I just know it can't break at that point due
>> the the sheer expense of revenue lost for downtime.
>>
>> Thanks,
>> Steve
>>
>>
>> Matt Florell wrote:
>> > How many lines and agents are you looking at?
>> >
>> > What kind of call volume?
>> >
>> > Average expected hold time?
>> >
>> > VICIDIAL could be an option for you since it does not use Asterisk
>> > Queues and can already easily scale across many servers.
>> >
>> > MATT---
>> >
>> >
>> > On 9/15/06, Steve Totaro <stotaro at asteriskhelpdesk.com> wrote:
>> >> I have been tossing around some ideas about scaling a call center 
>> with
>> >> load balancing and redundancy and would like the comunities input,
>> >> thoughts, criticism and anything anyone wants to toss in.
>> >>
>> >> The most evident thing is to start with beefy servers and only run 
>> procs
>> >> that are required.  All of the TDM boxes run stripped down 
>> versions of
>> >> Linux and Asterisk, they just take the call from the PRIs and convert
>> >> them to SIP, everything stays ulaw end to end.
>> >>
>> >> *Shared queues across multiple servers would be ideal*.  I don't 
>> think
>> >> it is possible in asterisk, as is.  Maybe DUNDI could be useful 
>> but I am
>> >> not up to speed on it enough to really know.
>> >>
>> >> I was toying with a concept of a DB server tracking the number of 
>> calls
>> >> to queue(s), number of agents logged into the queue(s).  Some agents
>> >> will be logged into multiple queues and providing the logic to a 
>> series
>> >> of Asterisk servers.   Calls could be made to the db to determine 
>> which
>> >> queue/server to route the call to.  In this situation, duplicate 
>> queues
>> >> would exist on several servers, so balancing would work somewhat 
>> if the
>> >> DB made the selection on which box to route the call to and which 
>> box an
>> >> agent should log into.  FastAGI and the manager interface will 
>> provide
>> >> the routing and DB updates.
>> >>
>> >> Another thought was to have one central server with all of the queues
>> >> and agents, then somehow the central server would cause a 
>> "recording/CDR
>> >> server" to send re-invites to the two SIP endpoints so that the 
>> call/RTP
>> >> stream is moved to another asterisk server which would record the 
>> call
>> >> and keep the CDR info.  Again, this would be done with a DB to decide
>> >> which asterisk (recording/CDR) box has the lightest load.  It 
>> would take
>> >> the burden of maintaining the call from the "Queue" server.  I/O 
>> is the
>> >> first bottleneck in scaling when you record each and every call.
>> >>
>> >> Would it be difficult to have asterisk send two SIP endpoints 
>> re-invites
>> >> and then bridge the call?  Then it is just a matter of the "Queue"
>> >> server checking the DB which recording/CDR server the call should 
>> go to
>> >> and send it a message to re-invite and bridge the endpoints.  A 
>> transfer
>> >> to a meetme is another possiblility but I want the "Queue" server 
>> out of
>> >> the stream.
>> >>
>> >> Has anybody else thought through the best way to scale something like
>> >> this.  I have a DS3 and will be using all of the channels in the
>> >> semi-near future.  I need to come up with a workable plan before 
>> then.
>> >>
>> >> Thanks,
>> >> Steve
>>
>
>