[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

info at beprojects.com info at beprojects.com
Wed Sep 21 11:53:00 MST 2005 

I would think memory would be the limiting factor.  A 3-4 minute wav 
file is what, 30Meg or so?  And there is one for each end of the call, 
so that's 60Meg.  Now let's say it's a 15 minute call and then are 10 of 
them at once.  That's 30Meg x 5 (5 times the length of my estimate) x 2 
(each leg) x 10 simultaneous calls....equals 3 Gig of RAM.  Once you run 
out of RAM, then what does it do?  It would have to try and dump it all 
to disk at once and you are back where you started.  I think the average 
* implementation doesn't have nearly enough free RAM to do this.

The RAMdisk solution seems to be pretty elegant in it's simplicity.



Matt Hess wrote:
> In light of the I/O bottleneck problem I'd have to ask why asterisk 
> can't just buffer incoming audio and then flush a complete audio file to 
> disk.. I'm assuming that recordings vary in length.. the problem with 
> this idea is what happens if 50 recordings all complete at the same 
> time.. a dump like that might not be very pretty (a fast drive plus a 
> little scheduler or limiter so that only x number of files get written 
> to disk at a time would probably help out there) but I can imagine that 
> a single file being written is much more efficient and more 
> disk-friendly.. perhaps I don't know what the heck I'm talking about but 
>  at least in my mind the theory sounds better than the current 
> 'stream-to-file' method employed by asterisk.
> 
> 
> 
> Matt Roth wrote:
> 
>> All,
>>
>> This message has generated a lot of responses, so I'm going to address 
>> each of them here in an attempt to consolidate the thread.
>>
>> ============================================================
>>
>> Matt,
>>
>> - I'm very interested in the specifics of your setup.
>> - How much space is on the RAM disk?
>> Currently it is 10 GB.  We are upgrading it to 16 GB.
>>
>> - What kind of RAM drive is it?
>> The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. 
>> The details for each 1 GB DIMM can be seen here:
>>
>> http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm 
>>
>>
>> The upgrade will involve adding 2 GB DIMMs to the system, but I don't 
>> have the details on these yet.
>>
>> The RAM disk is setup by adding the following kernel command line 
>> option to grub.conf:
>>
>> ramdisk_size=10485760
>>
>> We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
>>
>> By default the RAM disk's block size is 1024 bytes, so we are 
>> formatting it as an ext2 file system with a block size of 1024 bytes 
>> using the following command:
>>
>> mke2fs -b 1024 -m 0 /dev/ram0
>>
>> The block size can easily be changed from the kernel's view (using the 
>> kernel command line option ramdisk_blocksize=XXXX) or from mke2fs's 
>> view (using the -b XXXX argument), so please let me know if I can make 
>> an easy optimization here.
>>
>> Finally, the RAM disk is mounted using the command:
>>
>> mount /dev/ram0 /digrec
>>
>> A good RAMDISK howto exists at:
>>
>> http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
>>
>> - What format are you recording to?
>> - What codec are the SIP calls being placed over?
>> We are recording to the PCM format and using the G711 uLaw codec.  
>> High voice quality is essential to our application (we are a call 
>> center) so we partnered with MCI to configure our network for the 
>> required bandwidth and chose the highest quality, zero compression 
>> codec.  We noload all other codecs in order to avoid transcoding on 
>> the switch, so we must record to PCM. Later (on a separate server) the 
>> recordings are mixed to GSM which provides a 5 to 1 compression ratio 
>> with very little artifacts.
>>
>> - We've run into the "Avoided deadlock" recording issues several times 
>> when trying to do
>> - more than 50 concurrent recordings. Changing the ast_channel_lock 
>> loop from 10 to 20 has
>> - helped somewhat reduce the warnings and reduce audio gaps on the 
>> recordings, but what is
>> - really needed for more robust recording is a configurable recording 
>> buffer that wouldn't
>> - freak out if a 10ms delay occurs.
>> Are you saying that these messages indicate a gap in a digital 
>> recording?  If so, what is the duration of the gap? If it's comparable 
>> to a CD skip, I think we can deal with it until a buffer or another 
>> solution is implemented.
>>
>> - Good luck and please keep us updated on your progress
>> Thank you.  I'll be keeping the list informed of our progress.
>>
>> ============================================================
>>
>> Zoa,
>>
>> - I suppose you are the person from the digium forum
>> That was actually my boss's boss.  We thank you all the way up and 
>> down the line for your suggestion.
>>
>> - The reason i recommended you to use a ramdisk is because i think the
>> - problem with recording to disk is saving 20ms of stream 1, then 20 
>> ms of
>> - stream 2, then 20ms of stream 3 etc etc.... meaning you write everytime
>> - very small things. (with a lot of seeking).
>> Agreed.  This is why we hope that decoupling the copy (memory to disk) 
>> from Asterisk itself and, more importantly, Asterisk's real-time 
>> handling of the call being recorded will be sufficient.
>>
>> For the record, when recording 512 simultaneous calls to the local 
>> disk we saw a peek of about 13,000 blocks written per second.
>>
>> - Our best test results were with:
>> -
>> - - buffering the recordings to a ramdisk, then
>> We're doing that, as per your suggestion.
>>
>> - - on low load (at night) copy the files over the network (easy to shape
>> -   the pipe, so that you dont overload anything), This way, the memory
>> -   buffer will take care of the 'fragmentation' and not your harddisk.
>> If you'll note the format of the recordings and that we'll be 
>> recording up to 200,000 minutes of calls a day, with a little quick 
>> math you'll realize that it would take 80 to 100 GBs of memory for us 
>> to buffer a full day's recordings.  Combined with the fact that a 
>> server failure late in the day would cause us to lose them all, this 
>> is not a desirable solution.
>>
>> Instead, we plan to write an application to call from the MONITOR_EXEC 
>> hook that will be executed at the end of each call.  This application 
>> will be niced down to the lowest priority, and simply copy the leg 
>> files from memory to disk.  Under normal conditions (ie. our NFS 
>> server is up) this will actually be a copy to a remote disk using an 
>> asynchronous NFS transfer.  All actual disk I/O will occur on our 
>> digital recording server and any handling of the digital recordings 
>> will occur only after the call they are bound to is completed.
>>
>> Do you have any suggestions regarding this scenario?
>>
>> - - on the remove server, do all the mixing / indexing etc. (i really
>> -   don't mixing or converting between audio formats on the same 
>> server as
>> -   asterisk).
>> That exactly describes the functionality and purpose of our digital 
>> recording server.  We do no transcoding on the Asterisk server and 
>> perform no DSP on it.  It handles only SIP traffic, as our Ts are 
>> terminated by a Cisco AS5400HPX Universal Gateway.
>>
>> - If you want to go even freakier, run asterisk (or you complete distro)
>> - from a ramdisk.
>> -
>> - Oh, another thing, for the people trying this the performance of hdparm
>> - is not linear with the quality of your calls, tweaking your disks to be
>> - faster will not help for asterisk when you do a copy. (in general).
>> Thanks. I will keep this information in mind.
>>
>> - I thought over your suggestion to use a sniffer to do the recordings,
>> - you might pull it off, but will have to write your own to do so. (or go
>> - to the expensive version of commercial sniffer applications).
>> Trust me, using a sniffer is not our ideal solution. But if push comes 
>> to shove...
>>
>> - Also when you do things over the network, disable your onboard network
>> - card, and go for some more expensive network card.
>> - In our tests with small packets, we could increase the throughput 
>> with a
>> - factor 2. (related to cpu load).
>> -
>> -...the dual nic gigabit intel card was a lot better than all the other
>> - combinations i tried.
>> Done.  We were going to use a Level 5 Networks EF1-21022T NIC, but it 
>> didn't like performing DMA with anything greater than 4 GB of system 
>> RAM.  Level 5 is working on resolving this issue with their drivers, 
>> but until then we are using an Intel Pro 1000 MT NIC installed in a 
>> 64-bit, 133 MHz PCI-X expansion slot.  This Intel NIC is actually the 
>> onboard NIC on our digital recording server (a Dell PowerEdge 1850).
>>
>> ============================================================
>>
>> Trixter,
>>
>> - filesystems are also a consideration with larger scale projects.
>> - Different filesystems add different amounts of overheads on different
>> - types of operations.  Some are faster at moving small files around
>> - others faster with large files.  This adds to the disk latency.
>> - Removing the disk latency itself is a good thing, since that is
>> - typically slower, but to crank out that last little bit of performance
>> - some research into the different filesystems under the specific kernel
>> - that you are using could also be a consideration.  The most obvious
>> - (and easiest to update a running system) is to remove things like 
>> atime,
>> - whih with most linux distros is on by default.  This causes a write
>> - operation for the read of a file to update the last time accessed.  A
>> - couple little things can add up to a few percent improvement and
>> - generally make the cost go down.
>> We may need every percent we can get, so thanks for the suggestions.  
>> As mentioned above, we're running Fedora Core 3 with the latest 2.6 
>> SMP kernel.  Our physical disks are using the ext3 file system, while 
>> our RAM disk is using ext2.
>>
>> - Or have a seperate network set up (dual nic card for example) where the
>> - 2nd network is used just for NFS traffic.  Although NFS generally is
>> - ugly network wise, it is standard and makes things easier.  Just gotta
>> - watch the IO on the system given that the network card itself will 
>> cause
>> - cpu cycles to be used, but lets face it cpu is cheap now.  Different
>> - drivers also work differently, and then with the 2.6 series kernels you
>> - can use device polling instead of interupts which can help a little.
>> This seems like something we may pursue in the short term.  We are 
>> using NFS (which will be configuring for asynchronous transfers), but 
>> not with a dedicated NIC.  Our system has four 3.16 GHz Intel Xeon 
>> processors.  They are hyperthreaded, so the OS sees them as eight 
>> separate processors.  During our 256 call testing they hit 70% idle, 
>> but during our 512 call testing they dropped to 10% idle.  Obviously, 
>> 512 calls is pushing the limits but we only really need to hit around 
>> 300 for our current project.  Nonetheless, the additional CPU cycles 
>> consumed by another NIC are something to watch for.  Luckily, Intel 
>> will be releasing their dual-core processors soon.
>>
>> - When you say ramdisk here I assume you mean using conventional ram, its
>> - cheap yes but its volatile, do you have any plans for failure of the
>> - system or ram?  Or is the data integrity itself not as critical?  The
>> - reason that people like hard drives is because most of the time if the
>> - system goes down for any reason the data is still intact.
>> Please see my responses to Matt regarding our RAM disk configuration.
>>
>> - isnt vomit free?  It was a voip sniffer that worked with some codecs
>> - many years ago (I wanna say mid-late 90s but I may be thinking of
>> - another back then). http://vomit.xtdnet.nl/ does G.711 only.
>> -
>> - The bigger prIoblem that I see is that sniffers dont always get all the
>> - traffic that is on a network particularly when the network has more
>> - traffic on it.  While this generally isnt a concern and I would like to
>> - think that even a poorly configured network could allow for 512 calls,
>> - it is a factor to implement this type of a solution.
>> Hopefully, it won't come to this.  With all of your help, I doubt it 
>> will.
>>
>> ============================================================
>>
>> Richard,
>>
>> - Another project snatched from the Toilet bowl ahead of the flush.
>> With any luck, we'll be able to shine this turd.
>>
>> ============================================================
>>
>> I apologize for the length of this post, but after consideration it 
>> seemed to be the most efficient way to address each of you.
>>
>> Thank you all,
>>
>> Matthew Roth
>> InterMedia Marketing Solutions
>> Software Engineer and Systems Developer
>> _______________________________________________
>> --Bandwidth and Colocation sponsored by Easynews.com --
>>
>> Asterisk-Users mailing list
>> Asterisk-Users at lists.digium.com
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>>
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> --Bandwidth and Colocation sponsored by Easynews.com --
> 
> Asterisk-Users mailing list
> Asterisk-Users at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users

More information about the asterisk-users mailing list