[Asterisk-Users] update - 512 Simultaneous Calls with Digital Recording

[Matt Roth]

Matt Florell wrote:

>>Matthew, thanks for your feedback and advice.
>>    
>>
>>>what I actually experienced was the complete breakdown of Asterisk at
>>>around 60 concurrent recordings without it (the reality).
>>>      
>>>
>>The drive for saving your voice recordings is the same as your OS
>>(Asterisk)? What do you think that save the voice recordings to a
>>dedicated drive rather than the one which Asterisk program (OS) locates?
>>    
>>
>
>That is what we do actually. One drive for Linux/Asterisk and a SCSI
>RAID for /var/spool/asterisk/monitor
>  
>
I'm really surprised (and impressed) that this is working for you, 
Matt.  What are the specs of the RAID array (filesystem, drive speeds, 
RAID level, etc.)?

Once we starting recording the calls to a RAM disk, overall disk 
activity on the system dropped to next to nothing.  Asterisk really 
doesn't do much writing to disk (depending on your log level...debug can 
produce quite a bit of output) and its been my experience that MOH, 
announcements, and other frequently accessed files are being cached by 
Linux (there is almost no read activity on our production server, even 
with 100 calls in queue all using native MOH).

Taking that into consideration, I didn't think that a dedicated drive 
would make much of a difference.  Do you agree that the problem is that 
the frames are written to disk in the same thread that is responsible 
for bridging them between the end-points?  If so, why do you feel that 
Linux's file buffering is inadequate?  Have you considered using 
MixMonitor instead, as it is supposed to address this problem?

>>I also think about using GSM format (Monitor(gsm,${CALLFILENAME}, mb))
>>rather than WAV, PCM. In this case, it will use more CPU, but I/O of
>>hard disk is reduced dramatically as you mentioned that it is I/O
>>bottleneck issue, not CPU (In my case, I want to use P4 Dual core CPU or
>>extreme edition). In order to reduce the CPU usage, we can have two leg
>>files mixed after peak time.
>>    
>>
>
>Yes, you will have to wait until off-hours to mix the recordings, but
>while switching to GSM does help reduce the amount of data written to
>the drives, the gains are mostly cancelled out by the compression that
>is needed to convert each stream to GSM. In the case of 60 calls you
>would be converting 120 streams to GSM.
>
>  
>
I agree with you.  We mix our PCM recordings down to GSM, but we do it 
on a dedicated server.  We've hijacked soxmix so that the recording is 
available almost immediately after the call has completed.  Check out 
this post 
<http://lists.digium.com/pipermail/asterisk-users/2006-April/147202.html> 
for details.

>>Matt mentioned about fragmented free space. I googled about Linux
>>defragment topic. People always talk about that Linux doesn't need to
>>defragment, it can handle it by itself very well. Not sure how true it
>>is.
>>    
>>
>
>It is something we ran into before we switched to SCSI RAID setups. It
>happened quicker with IDE drives than with SATA(and so far not at all
>with SCSI). The problem does not show up until you have deleted files
>several times from the drives and it gets worse over time until you 
>wipe and reformat the recording drive.
>
>The problem that happens with a large amount of concurrent recordings
>is audio skips, from a quarter second to two seconds. There isn't much
>of a recording buffer built-in to Asterisk so if anything causes a
>delay in packet delivery there will be an audio skip. No matter what
>kind of setup we used we have not been able to guarantee skip-free
>recording files in any case above 60 concurrent recordings over the
>course of a week. It is very hard to test for, but if you have all of
>your recordings listened to you will be able to know if there is a
>problem. For us in one case(70 concurrent recordings) the skips only
>occured very infrequently, maybe 0.00005% of the time(10 skips in 55
>hours of recordings) which doesn't sound bad, but if it happens during
>the wrong part of a critical sales confirmation that half second skip
>could cost my company hundreds of dollars.
>
>  
>
Do these skips coincide with drops in the actual call audio?  What kind 
of overall call quality are you experiencing (dropped audio, dropped 
calls, etc)?  What does the load average look like on your system at peak?

>>I am looking a solution to record expanding simultaneous calls in the
>>future in a call centre which accepts calls from our global branches. If
>>I find the good solution, I definitely post it to the community.
>>    
>>
>
>The best reliable solution that I have been able to depend on is to
>limit the number to 60 concurrent recordings on a single P4 3.4GHz
>server with a LSILogic MegaRAID 320-1 and four 15k-RPM SCSI drives in
>a RAID 10. With this we experience usually no audio skips across the
>course of a week(and thousands of recordings), and it has worked this
>way reliably for the last nine months.
>
>If you have less strict quality standards you could probably push that
>number up, but you will get more and more audio skips the higher you
>go.
>
Thanks for another very informative post, Matt.  It looks like we share 
a lot of similar goals.  Feel free to contact me off-list if I can ever 
be of any help to you.

Sincerely,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer