[Asterisk-Users] update - 512 Simultaneous Calls with Digital Recording

[Matt Roth]

>On 4/3/06, Isaac Xiao <isaac.x at kvbkunlun.com> wrote:
>  
>
>>
>>Hi All,
>>
>>
>>
>>In previous mail lists, people talked about a solution to record large
>>amount of simultaneous calls. And then it seems that RAM disk solution was
>>the best choice due to the I/O bottleneck of Hard disk (System). Please find
>>the previous discussion as follows:
>>
>>http://lists.digium.com/pipermail/asterisk-users/2005-October/120930.html
>>
>>
>>
>>http://thread.gmane.org/gmane.comp.telephony.pbx.asterisk.user/118497
>>
>>
>>
>>I noticed that an Intel server platform comes with 4 ports SATA-II RAID0/1
>>controller. Please check this link:
>>
>>http://developer.intel.com/design/servers/boards/se7230nh1-e/index.htm
>>
>>SATA-II is 3Gps I/O. so I am thinking implement this solution for recording
>>large amount of simultaneous calls. My proposed solution is:
>>
>>
>>
>>CPU: Prentium 4 Dual core 3.2G or higher
>>
>>RAM: 2G or more
>>
>>OS HDD: 2 x 80G SATA-II HDD for RAID 1 (for OS)
>>
>>Separate HDD for voice recording: 1 x 200G SATA-II HDD standalone
>>
>>Voice recording format: WAV
>>
>>
>>
>>I did a test on my Asterisk PBX. One leg of 61 seconds WAV voice recording
>>use 947564 bytes of hard disk. I did an optimal estimate of numbers of
>>simultaneous calls with 3Gps without thinking of CPU and system limitation.
>>
>>
>>
>>Two legs: 947564 x 2 x 8 = 1895128 Bytes x 8 = 15161024 bits
>>
>>
>>
>>15161024/61 = 248541 bits/s
>>
>>
>>
>>So the number of simultaneous calls 3Gps I/O can handle:
>>
>>
>>
>>3(Gps) x 1024(Mb/G) x 1024 (Kb/M) x 1024 (b/Kb) / 248541 = 12960 calls.
>>
>>
>>
>>512 are far less than 12960. So I think the hard disk I/O should not have
>>any bottleneck problem for 512 simultaneous calls.
>>
>>
>>
>>Welcome to add your comments, do test and give feedback.
>>
>>
>>
>>Cheers,
>>
>>Isaac Xiao
>>    
>>
>Matt Florell wrote:
>
>  Hello,
>
>What happens a few weeks into this when you've fragmented the free
>space on the drives by deleting files periodically and rewriting in
>some places? The consistent performance of SATA drives goes down
>dramatically when this happens, much more so than on a SCSI-based
>drive system.
>
>Also, you don't seem to take into account the read-write-seek time of
>SATA drives into your calculations
>
>Another thing to consider is that if anything else is being done on
>these drives(like running Linux or Asterisk or anything else) the
>write performance will also be effected.
>
>MATT---
>  
>
Isaac,

I'm the author of the original posts and I'd like to share some 
additional information with you.  Our system is a Dell PowerEdge 6850 
and you can see the drive specs here: 
<http://www1.us.dell.com/content/products/productdetails.aspx/pedge_6850?c=us&cs=04&l=en&s=bsd&~page=1#tabtop>.  
The drives that we purchased are 15,000 RPM Ultra320 SCSI drives.  We 
have two 73 GB drives configured in a RAID 1 (mirrored).  The server has 
a PERC hardware RAID controller and we are using the megaraid drivers.  
All of the filesystems are ext3 (except the RAM disk which is ext2).

It was an oversight at the time of the original tests to have the 
partition that the recordings were written to be a journaled 
filesystem.  There is some overhead there, but I'm not clear on exactly 
how much.  The RAID 1 also has some write overhead.  We had limited time 
to use the test equipment, so we went right to the RAM disk without 
changing either of these settings but they are a couple of optimizations 
to keep in mind if you aren't seeing satisfactory performance.

The 512 calls was also the only load on the system during testing.  We 
are actually using the box in production now and with the additional 
load of agents, announcements, MOH, unattended monitoring, NFS transfers 
of the leg files to a remote box for mixing, etc. it looks like a more 
realistic number is somewhere between 250-300 simultaneous calls.  For 
instance, at this moment we have 45 calls in queue and 116 active calls 
(resulting in roughly 250 leg files being generated) and the box is 
about 65% idle.  I'm assuming that the scaling will be somewhere between 
linear and exponential, which is where I'm deriving my projected numbers 
from.

The point of all this is that you need to be careful when comparing the 
ideal situation to the realistic one.  As Matt Florell (who is *very* 
knowledgeable) pointed out, many factors and variables come into play.  
I've been told by some people that the way the Linux kernel buffers disk 
writes should approach the performance of the RAM disk (the ideal), but 
what I actually experienced was the complete breakdown of Asterisk at 
around 60 concurrent recordings without it (the reality).  That seems to 
be the magic number for other users as well, so I don't believe that the 
ext3 filesystem or the RAID 1 are solely to blame.  From the numbers we 
captured, I think that the system could not keep up with the overall 
number of disk writes.  They occur in the same thread as the processing 
of the voice frames, so this leads to a severe decrease in call quality, 
including dropped calls.  I think MixMonitor addresses this problem, but 
it has its own set of issues.

The RAM disk solution also seems a bit daunting because of the amount of 
memory we dedicated to it.  For some perspective, right now we're only 
using 5% of the RAM disk, so it looks like we may be overconfigured.  
The extra space is nice in case the NFS moves fail, but we are thinking 
about moving some other files to it (MOH, announcements, logs, etc.) as 
well.  I have no personal experience with SATA drives, so your mileage 
may vary.  I just wanted to make you're aware that sometimes things 
aren't the same on paper as they are in the server room and that the RAM 
disk solution is being used in a production environment at this time.  
When moving ahead, it's a good idea to have plans to fall back on when 
things don't go as expected.

I wish you luck with your project and hope that you share the results of 
your implementation with the community.

Sincerely,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer