[Asterisk-Users] update - 512 Simultaneous Calls with Digital Recording

[Matt Florell]

> >That is what we do actually. One drive for Linux/Asterisk and a SCSI
> >RAID for /var/spool/asterisk/monitor
> >
> >
> I'm really surprised (and impressed) that this is working for you,
> Matt.  What are the specs of the RAID array (filesystem, drive speeds,
> RAID level, etc.)?

We use LSILogic MegaRAID 320-1 with four 320U 15kRPM 78GB SCSI drives
RAID 1 across two logical partitions

> Taking that into consideration, I didn't think that a dedicated drive
> would make much of a difference.  Do you agree that the problem is that
> the frames are written to disk in the same thread that is responsible
> for bridging them between the end-points?  If so, why do you feel that
> Linux's file buffering is inadequate?  Have you considered using
> MixMonitor instead, as it is supposed to address this problem?

I don't think it's Linux's file buffer, it's Asterisk. There is not
much of a buffer there and once you get 100 streams running through
it, it begins to have problems.

As for MixMonitor, we haven't really messed around with tweaking
recording since what we have now works wonderfully and 9 months ago
MixMonitor was not production-ready 9 months ago.

You need to keep in mind that we have VERY stringent requirements for
our audio recordings. A audio recording drop rate of 0.00005% is too
high for us and is not even noticable by other Asterisk users so we
had to figure out how to totally eliminate any skips or audio gaps in
recordings and that's the solution that works for our needs. Others
might get by with a fwe skips and have a much higher concurrent
recording capacity.

> >Yes, you will have to wait until off-hours to mix the recordings, but
> >while switching to GSM does help reduce the amount of data written to
> >the drives, the gains are mostly cancelled out by the compression that
> >is needed to convert each stream to GSM. In the case of 60 calls you
> >would be converting 120 streams to GSM.
> >
> I agree with you.  We mix our PCM recordings down to GSM, but we do it
> on a dedicated server.  We've hijacked soxmix so that the recording is
> available almost immediately after the call has completed.  Check out
> this post
> <http://lists.digium.com/pipermail/asterisk-users/2006-April/147202.html>
> for details.

We started doing it off-server as well recently, it does help to have
a machine to do just the mixing now for all of our Asterisk servers'
audio.

> >The problem that happens with a large amount of concurrent recordings
> >is audio skips, from a quarter second to two seconds. There isn't much
> >of a recording buffer built-in to Asterisk so if anything causes a
> >delay in packet delivery there will be an audio skip. No matter what
> >kind of setup we used we have not been able to guarantee skip-free
> >recording files in any case above 60 concurrent recordings over the
> >course of a week. It is very hard to test for, but if you have all of
> >your recordings listened to you will be able to know if there is a
> >problem. For us in one case(70 concurrent recordings) the skips only
> >occured very infrequently, maybe 0.00005% of the time(10 skips in 55
> >hours of recordings) which doesn't sound bad, but if it happens during
> >the wrong part of a critical sales confirmation that half second skip
> >could cost my company hundreds of dollars.
> >
> >
> >
> Do these skips coincide with drops in the actual call audio?  What kind
> of overall call quality are you experiencing (dropped audio, dropped
> calls, etc)?  What does the load average look like on your system at peak?

We are on only Zap channels on the recording servers, so no issues
with audio dropping. The load is low, peaking at 50%.

> >The best reliable solution that I have been able to depend on is to
> >limit the number to 60 concurrent recordings on a single P4 3.4GHz
> >server with a LSILogic MegaRAID 320-1 and four 15k-RPM SCSI drives in
> >a RAID 10. With this we experience usually no audio skips across the
> >course of a week(and thousands of recordings), and it has worked this
> >way reliably for the last nine months.
> >
> >If you have less strict quality standards you could probably push that
> >number up, but you will get more and more audio skips the higher you
> >go.
> >
> Thanks for another very informative post, Matt.  It looks like we share
> a lot of similar goals.  Feel free to contact me off-list if I can ever
> be of any help to you.

I wish I had the cash for a nice big RAM disk to play with :)

MATT---