[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

[Matt Roth]

Hi everyone,

This is just another attempt to address everybody in one place and 
consolidate the thread.

============================================================

Zoa,

- I think its the best you can do.
If something pops into your head, even if it's off the wall, don't 
hesitate to share it.  Any idea is better than no idea if things go south.

- Maybe there should be some option to be set for the monitor command to
- buffer, with a warning that it will eat memory.
- Its also not needed to buffer the complete call at once, just buffering
- and writing to disk every 10 seconds would already be a big improvement,
- and since the calls won't all start at the same time, the writing to
- disk will be spread over time.
This would be a nice option. res_monitor.c is using ast_writefile() to 
handle writing each call leg to file. Maybe a more specialized file 
writing function that includes buffering or threading is all that is needed.

- Good luck with the implementation, let us know how it turns out!
- Try to find out what happens if the nfs share is gone, or overloaded too.
Thanks. The information provided by members of this list is invaluable, 
so I'll be sure to keep you informed of our progress. The plan is to 
have the application that is called via the MONITOR_EXEC hook verify the 
NFS mount before the copy. If it's not there it'll have to write locally 
until the problem is resolved.

The NFS mount will have it's own dedicated network interface (an Intel 
Pro 1000 MT) communicating to the digital recording server over a 
Gigabit connection.  That should be sufficient.

- If there is no zaptel hardware involved, i don't think you have to be
- too concerned about the network cards interrupts.
Nope, we're terminating our Ts in a Cisco AS5400HPX Universal Gateway 
and using ztdummy as a timing source for music on hold. Our Asterisk 
server (a Dell PowerEdge 6850) has configurable IRQs, so I can fix any 
conflicts that may arise with other hardware relatively easily.

============================================================

Matt,

- Have you tried recording directly to GSM format? It will help reduce the
- bottleneck on disk IO although it will use more CPU cycles(in your case
- on a RAM drive this may not help at all)
We don't want to do any transcoding on the Asterisk server because of 
the drain on CPU. The RAM disk seems to be the best solution for our 
scenario.

- (Regarding "Avoided deadlock" warnings and their effects)
- There aren't always audio skips but they do happen more when you get more
- ast_channel_walk warnings. The audio gaps are usually less than a quarter
- second in our experience but can be upto a second depending on the
- severity of the IO problem at that instance. It's very hard to test for
- until you get into production and you have real conversations and real
- people listening to them that can hear the audio skips.
Thank you for this information. It seems like the warnings won't cause 
any unacceptable results, but we'll be listening to the audio regularly 
after we go into production.

- We have sevaral call centers as well, and we just restrict a single server
- to 50 recordings at once and then we would pass the next recording as an
- IAX2 channel to another recording server. It's a scalable system for us
- that is relatively cheap and works well since we can mix and gsm-encode
- the audio on these multiple servers at night when not in production
- leaving the NSF server just for storage and not audio processing.
That is a smart solution. Thanks for sharing it. If we have any problems 
with our setup we may start looking in this direction.

- (Regarding queue management, call volume, and system architecture of
-  his Asterisk setup)
- We wrote VICIDIAL(part of the GPL astGUIclient suite
- http://astguiclient.sf.net) for our call center operations. Yes it does
- use a central queue and does not use Asterisk queues or agents. The
- system is based on a MySQL database and meetme rooms for the agents that
- use a web-client app for lead information and call control.
We'll have to take a look at your work. We have to manage multiple 
queues and plan on using Asterisk queues and agents on our single 
Asterisk server.

- We mostly do outbound and the volume is split across several servers, and
- for inbound we do have forwarding to other servers if the defined
- capacity is exceeded a certain point.
How are you calculating this capacity?
 
- As for our distributed recording approach, it's easy with meetme rooms to
- record a call on one server from another server, you just drop a call into
- the meetme room that is a monitor exten on another server over IAX2, TDMoE
- or a crossover Zap T1 connection.
Great idea. Once again, if we have problems with our implementation it's 
great to have this alternative path to explore.

- As for phone calls at one time: for inbound we almost never exceed 50
- agents on a single server with no more than 72 incoming lines live at 
once.
- Our average is actually much less than that. For outbound we usually have
- about 15-40 agents per server with upto 96 lines dialing out 
concurrently.
- At our main office location we've had over 100 agents on at one time 
across
- 6 Asterisk servers handling over 350 calls at once with a total of 
more than
- 550 live channels on our Asterisk servers(includes recording, client and
- trunk channels).
I hope you don't mind answering some questions about your setup.

It seems like you have really solid numbers to work with. We're 
implementing our own reporting (both real-time and next-day) using 
Asterisk's call detail records (stored in a MySQL database) and 
information captured from the management interface (via AstManProxy). 
Are we reinventing the wheel? Do you have any tips for us in regards to 
our reporting software?

I'm assuming you are using VICIDIAL as a predictive dialer in 
conjunction with Asterisk for your outbound calling. We are looking for 
an outbound dialing solution. Could you please provide a list of the 
abilities and limitations of VICIDIAL?

In your experience, how stable is Asterisk at high call volumes? Do you 
have any issues with dropped calls, call quality, or losing calls in queue?

How are you calculating the number of live channels on your Asterisk 
server?  For instance, if we have 256 simultaneous calls each being 
digitally recorded, how many live channels is that? Is there a number of 
live channels where Asterisk starts experiencing stability problems?

- We mostly use cheaper single CPU systems instead of more expensive 
multi-CPU
- systems which is more cost effective in a larger setup and allows for 
greater
- flexibility, redundancy and scalability.
We'll keep these points in mind as we move forward with our development.

- I'll also be presenting a case-study on our whole setup at Astricon next
- month. The presentation file will be available on the astGUIclient 
project
- website after I present it for those who are interested.
Good luck with that. Maybe we'll see you there in 2006.

============================================================

Waldo,

- Is your solution 100% Asterisk or are you using other "helpers" such 
as SER or XXXproxy or whatever?

We are not using SER or XXXproxy.  We are using a Cisco AS5400HPX 
Universal Gateway to terminate our Ts. The gateway sends SIP traffic to 
the Asterisk server, from which point we are 100% Asterisk.

We are also a 100% open implementation, from hardware specs to software 
configurations.  Right now I'm a little bogged down with development and 
I have a deadline looming over my head, but in general if you want to 
know something just ask and I'll do my best to provide a quick response.

============================================================

Thank you all for your responses,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer