[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

[Matt Florell]

On 9/22/05, Matt Roth <mroth at imminc.com> wrote:

> Matt,
>
> - Have you tried recording directly to GSM format? It will help reduce the
> - bottleneck on disk IO although it will use more CPU cycles(in your case
> - on a RAM drive this may not help at all)
> We don't want to do any transcoding on the Asterisk server because of
> the drain on CPU. The RAM disk seems to be the best solution for our
> scenario.


That's pretty much what I though, wanted to mention it though.

- (Regarding "Avoided deadlock" warnings and their effects)
> - There aren't always audio skips but they do happen more when you get
> more
> - ast_channel_walk warnings. The audio gaps are usually less than a
> quarter
> - second in our experience but can be upto a second depending on the
> - severity of the IO problem at that instance. It's very hard to test for
> - until you get into production and you have real conversations and real
> - people listening to them that can hear the audio skips.
> Thank you for this information. It seems like the warnings won't cause
> any unacceptable results, but we'll be listening to the audio regularly
> after we go into production.


When we traced the recording skips back to an exact time there is always a
ast_channel_walk warning in the logs, although there is not always an audio
skip if the warning appears, but the more of those warnings, the more skips
we got.

- We mostly do outbound and the volume is split across several servers, and
> - for inbound we do have forwarding to other servers if the defined
> - capacity is exceeded a certain point.
> How are you calculating this capacity?


Trial and error, astGUIclient/VICIDIAL has a very detailed system
performance logging utility built into it that tracks open asterisk
channels, ram, CPU use %, system load and a few other things and we were
best able to find a good and safe capacity at which to limit our systems to
ensure reliability:
http://astguiclient.sourceforge.net/VDreports/performance_new.gif

- As for phone calls at one time: for inbound we almost never exceed 50
> - agents on a single server with no more than 72 incoming lines live at
> once.
> - Our average is actually much less than that. For outbound we usually
> have
> - about 15-40 agents per server with upto 96 lines dialing out
> concurrently.
> - At our main office location we've had over 100 agents on at one time
> across
> - 6 Asterisk servers handling over 350 calls at once with a total of
> more than
> - 550 live channels on our Asterisk servers(includes recording, client and
> - trunk channels).
> I hope you don't mind answering some questions about your setup.
>
> It seems like you have really solid numbers to work with. We're
> implementing our own reporting (both real-time and next-day) using
> Asterisk's call detail records (stored in a MySQL database) and
> information captured from the management interface (via AstManProxy).
> Are we reinventing the wheel? Do you have any tips for us in regards to
> our reporting software?


We don't use any Asterisk-generated logs, they didn't offer enough
information for us, so we created several of our own MySQL logs for calls
and Manager actions that we use to track every action that happens across
all systems as well as all live calls that are occuring in real-time on all
systems. This allows VICIDIAL to have real-time status screens and
datetime-level statistical performance reports available any time you want
to pull them. As for tips, just log everything, you may not think you need
it now but you will eventually want just about every little piece of call
and action information that you can get your hands on for reporting.

I'm assuming you are using VICIDIAL as a predictive dialer in
> conjunction with Asterisk for your outbound calling. We are looking for
> an outbound dialing solution. Could you please provide a list of the
> abilities and limitations of VICIDIAL?


Well, since we wrote VICIDIAL we are a little biased :) it is free and GPL
so you won't get many of the pretty features you get with those expensive
dialer solutions, but you can tinker with it all you like. We've been using
it for about 2 years in production and now have it on over 200 seats across
4 locations for our company. There are also over 50 production VICIDIAL
installations in over a dozen countries that we know about. It works better
for us than any of the dialer systems we used to use and we have total
control over it which makes it easier to change things on the fly. There are
limitations and things that we have not gotten around to writing yet, but
there is an active community around it and we are developing new features
for it all the time and releasing a new version every 2-8 weeks. One
limitation is Answering Machine detection. We haven't found a good method of
doing this and we don't like the delay that all commercial systems have when
doing AM detection so we just do a dial timeout by default, about 4-5 rings
will eliminate 90% of the answering machines you receive without any delay
in the calls that are answered by a human. For other features see the
product page:
http://astguiclient.sourceforge.net/vicidial.html

In your experience, how stable is Asterisk at high call volumes? Do you
> have any issues with dropped calls, call quality, or losing calls in
> queue?


What really matters is the load on your system and the type of channels you
are using. Zap channels are more reliable at extremely high loads than any
VOIP channel. Manager Actions can fail at extremely high loads. All VOIP
channels will suffer quality loss at extremely high loads. By extremely high
loads I mean about 5.00 and higher of system load(10.00 on dual CPU machine)
There are other factors, but this has proven fairly reliable for us. System
Load by channels in use is different from server to server depending on the
CPU/RAM/Motherboard and other application running on the server.

How are you calculating the number of live channels on your Asterisk
> server? For instance, if we have 256 simultaneous calls each being
> digitally recorded, how many live channels is that? Is there a number of
> live channels where Asterisk starts experiencing stability problems?


With VICIDIAL we have a real-time count of all active channels (including
recordings) on each server so it is easy to limit them. 256 channels being
recorded would be 256 recording channels, and 256 agent channels and at
least 256 customer channels(768 channels total) which is many more than
we've ever had going on a single server. The maximum channels per server
really depends on the server, it's hard to specilate in general without
testing first.

Hope this helps,

MATT---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.digium.com/pipermail/asterisk-users/attachments/20050922/c5018804/attachment.htm