[asterisk-users] Scaling/Loadbalancing a Call Center and Redundancy

[Steve Totaro]

Much higher, maybe double but that is when the agents start to complain 
that their conversations start "cutting in and out".

This is the main reason I am looking into building a re-invite solution 
that moves the call into "recording/cdr" server's media path.  Then we 
just cap these servers at 50-70 calls and keep track of it in a DB.  I 
think my developers could re-write asterisk to handle this.  It will 
also allow the main "Queue" server(s) to go down, so long as the 
"recording/cdr" server is still up no ongoing calls would get lost.

Thanks,
Steve Totaro

Matt Florell wrote:
> To maintain high recording quality with no audio skips we have found
> that you should not go over 50 conversations being recorded on a
> single server. What have you found is your limit while maintaining
> very good audio quality?
>
> MATT---
>
> On 9/16/06, Steve Totaro <stotaro at asteriskhelpdesk.com> wrote:
>> Right now we are all inbound and every call is recorded.
>>
>> Matt Florell wrote:
>> > Hello,
>> >
>> > At this point in time VICIDIAL is more focused on outbound features,
>> > but inbound and blended capcbilities have been part of VICIDIAL for
>> > about two years. The most I have done inbound-only with it is 3 T1s
>> > with 60 agents. But for outbound and inbound agents together we have
>> > had upto 120 agents on one setup with 20 T1s(spread across 8 Asterisk
>> > servers, 2 web servers and 1 MySQL server) handling over 200,000 calls
>> > a day(mostly outbound of course).
>> >
>> > The inbound portion of VICIDIAL does not have customized hold music or
>> > periodic announcements yet, but we plan on adding those features in a
>> > future version as we begin to focus more on inbound in the project.
>> >
>> > We do use meetme rooms in VICIDIAL. This allows for easy third-party
>> > calls and multiple monitoring/manager intrusion into an agent session.
>> >
>> > The load balancing works by having the Database keep track of agents
>> > and calls for all of the servers and then use IAX to move the calls
>> > from where they originated to whoever the next available agent is, no
>> > matter what server they are on. So the calls can come from anywhere
>> > and the agents can be logged in anywhere.
>> >
>> > If you receive inbound callerID on these calls you will also be able
>> > to have caller information appear on the Agent's web interface. And if
>> > they are a customer that is already in the system it will bring up
>> > their existing record.
>> >
>> > As for reliability, we have not had a total system failure in the last
>> > 2 years(aside from long power outages and hurricane interruptions).
>> > MySQL can handle a tremendous volume and it is the only
>> > total-system-single-point-of-failure in VICIDIAL, ours never crashes.
>> > The web servers can be load balanced(no need for session-awareness)
>> > and you can use any Apache/PHP webserver that may be on your system to
>> > serve the AJAX-based agent web interface. As for Asterisk, we have had
>> > servers crash periodically(a couple crashes a month across 8 servers),
>> > but that is to be expected when you push tens of thousands of calls
>> > through each one per day.
>> >
>> > Will you be recording all calls in this setup?
>> >
>> > MATT---
>> >
>> > On 9/16/06, Steve Totaro <stotaro at asteriskhelpdesk.com> wrote:
>> >> Matt,
>> >>
>> >> I am sure this is a RTFM and I am pretty sure you are using meetme
>> >> rooms.  Just not too sure how you do the magic.
>> >>
>> >> 28 T1s with NFAS so 95 channels per trunk group, seven trunk groups =
>> >> 665 lines.  My client's call volume has shot from 5,000 to about 
>> 10,000
>> >> calls a day.  Due to recent product offerings/advertising, I 
>> expect to
>> >> be eating up 6 T1 (peak) by the end  of October.  They will 
>> eventually
>> >> have every channel in use during peaks, whether that is in 
>> November or
>> >> December, I am not sure.  I just know it can't break at that point 
>> due
>> >> the the sheer expense of revenue lost for downtime.
>> >>
>> >> Thanks,
>> >> Steve
>> >>
>> >>
>> >> Matt Florell wrote:
>> >> > How many lines and agents are you looking at?
>> >> >
>> >> > What kind of call volume?
>> >> >
>> >> > Average expected hold time?
>> >> >
>> >> > VICIDIAL could be an option for you since it does not use Asterisk
>> >> > Queues and can already easily scale across many servers.
>> >> >
>> >> > MATT---
>> >> >
>> >> >
>> >> > On 9/15/06, Steve Totaro <stotaro at asteriskhelpdesk.com> wrote:
>> >> >> I have been tossing around some ideas about scaling a call center
>> >> with
>> >> >> load balancing and redundancy and would like the comunities input,
>> >> >> thoughts, criticism and anything anyone wants to toss in.
>> >> >>
>> >> >> The most evident thing is to start with beefy servers and only run
>> >> procs
>> >> >> that are required.  All of the TDM boxes run stripped down
>> >> versions of
>> >> >> Linux and Asterisk, they just take the call from the PRIs and 
>> convert
>> >> >> them to SIP, everything stays ulaw end to end.
>> >> >>
>> >> >> *Shared queues across multiple servers would be ideal*.  I don't
>> >> think
>> >> >> it is possible in asterisk, as is.  Maybe DUNDI could be useful
>> >> but I am
>> >> >> not up to speed on it enough to really know.
>> >> >>
>> >> >> I was toying with a concept of a DB server tracking the number of
>> >> calls
>> >> >> to queue(s), number of agents logged into the queue(s).  Some 
>> agents
>> >> >> will be logged into multiple queues and providing the logic to a
>> >> series
>> >> >> of Asterisk servers.   Calls could be made to the db to determine
>> >> which
>> >> >> queue/server to route the call to.  In this situation, duplicate
>> >> queues
>> >> >> would exist on several servers, so balancing would work somewhat
>> >> if the
>> >> >> DB made the selection on which box to route the call to and which
>> >> box an
>> >> >> agent should log into.  FastAGI and the manager interface will
>> >> provide
>> >> >> the routing and DB updates.
>> >> >>
>> >> >> Another thought was to have one central server with all of the 
>> queues
>> >> >> and agents, then somehow the central server would cause a
>> >> "recording/CDR
>> >> >> server" to send re-invites to the two SIP endpoints so that the
>> >> call/RTP
>> >> >> stream is moved to another asterisk server which would record the
>> >> call
>> >> >> and keep the CDR info.  Again, this would be done with a DB to 
>> decide
>> >> >> which asterisk (recording/CDR) box has the lightest load.  It
>> >> would take
>> >> >> the burden of maintaining the call from the "Queue" server.  I/O
>> >> is the
>> >> >> first bottleneck in scaling when you record each and every call.
>> >> >>
>> >> >> Would it be difficult to have asterisk send two SIP endpoints
>> >> re-invites
>> >> >> and then bridge the call?  Then it is just a matter of the "Queue"
>> >> >> server checking the DB which recording/CDR server the call should
>> >> go to
>> >> >> and send it a message to re-invite and bridge the endpoints.  A
>> >> transfer
>> >> >> to a meetme is another possiblility but I want the "Queue" server
>> >> out of
>> >> >> the stream.
>> >> >>
>> >> >> Has anybody else thought through the best way to scale 
>> something like
>> >> >> this.  I have a DS3 and will be using all of the channels in the
>> >> >> semi-near future.  I need to come up with a workable plan before
>> >> then.
>> >> >>
>> >> >> Thanks,
>> >> >> Steve
>> >>
>> >
>> >
>>
>