[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

Wed Sep 21 10:02:58 MST 2005

In light of the I/O bottleneck problem I'd have to ask why asterisk 
can't just buffer incoming audio and then flush a complete audio file to 
disk.. I'm assuming that recordings vary in length.. the problem with 
this idea is what happens if 50 recordings all complete at the same 
time.. a dump like that might not be very pretty (a fast drive plus a 
little scheduler or limiter so that only x number of files get written 
to disk at a time would probably help out there) but I can imagine that 
a single file being written is much more efficient and more 
disk-friendly.. perhaps I don't know what the heck I'm talking about but 
  at least in my mind the theory sounds better than the current 
'stream-to-file' method employed by asterisk.

Matt Roth wrote:
> All,
> 
> This message has generated a lot of responses, so I'm going to address 
> each of them here in an attempt to consolidate the thread.
> 
> ============================================================
> 
> Matt,
> 
> - I'm very interested in the specifics of your setup.
> - How much space is on the RAM disk?
> Currently it is 10 GB.  We are upgrading it to 16 GB.
> 
> - What kind of RAM drive is it?
> The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. 
> The details for each 1 GB DIMM can be seen here:
> 
> http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm 
> 
> 
> The upgrade will involve adding 2 GB DIMMs to the system, but I don't 
> have the details on these yet.
> 
> The RAM disk is setup by adding the following kernel command line option 
> to grub.conf:
> 
> ramdisk_size=10485760
> 
> We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
> 
> By default the RAM disk's block size is 1024 bytes, so we are formatting 
> it as an ext2 file system with a block size of 1024 bytes using the 
> following command:
> 
> mke2fs -b 1024 -m 0 /dev/ram0
> 
> The block size can easily be changed from the kernel's view (using the 
> kernel command line option ramdisk_blocksize=XXXX) or from mke2fs's view 
> (using the -b XXXX argument), so please let me know if I can make an 
> easy optimization here.
> 
> Finally, the RAM disk is mounted using the command:
> 
> mount /dev/ram0 /digrec
> 
> A good RAMDISK howto exists at:
> 
> http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
> 
> - What format are you recording to?
> - What codec are the SIP calls being placed over?
> We are recording to the PCM format and using the G711 uLaw codec.  High 
> voice quality is essential to our application (we are a call center) so 
> we partnered with MCI to configure our network for the required 
> bandwidth and chose the highest quality, zero compression codec.  We 
> noload all other codecs in order to avoid transcoding on the switch, so 
> we must record to PCM. Later (on a separate server) the recordings are 
> mixed to GSM which provides a 5 to 1 compression ratio with very little 
> artifacts.
> 
> - We've run into the "Avoided deadlock" recording issues several times 
> when trying to do
> - more than 50 concurrent recordings. Changing the ast_channel_lock loop 
> from 10 to 20 has
> - helped somewhat reduce the warnings and reduce audio gaps on the 
> recordings, but what is
> - really needed for more robust recording is a configurable recording 
> buffer that wouldn't
> - freak out if a 10ms delay occurs.
> Are you saying that these messages indicate a gap in a digital 
> recording?  If so, what is the duration of the gap? If it's comparable 
> to a CD skip, I think we can deal with it until a buffer or another 
> solution is implemented.
> 
> - Good luck and please keep us updated on your progress
> Thank you.  I'll be keeping the list informed of our progress.
> 
> ============================================================
> 
> Zoa,
> 
> - I suppose you are the person from the digium forum
> That was actually my boss's boss.  We thank you all the way up and down 
> the line for your suggestion.
> 
> - The reason i recommended you to use a ramdisk is because i think the
> - problem with recording to disk is saving 20ms of stream 1, then 20 ms of
> - stream 2, then 20ms of stream 3 etc etc.... meaning you write everytime
> - very small things. (with a lot of seeking).
> Agreed.  This is why we hope that decoupling the copy (memory to disk) 
> from Asterisk itself and, more importantly, Asterisk's real-time 
> handling of the call being recorded will be sufficient.
> 
> For the record, when recording 512 simultaneous calls to the local disk 
> we saw a peek of about 13,000 blocks written per second.
> 
> - Our best test results were with:
> -
> - - buffering the recordings to a ramdisk, then
> We're doing that, as per your suggestion.
> 
> - - on low load (at night) copy the files over the network (easy to shape
> -   the pipe, so that you dont overload anything), This way, the memory
> -   buffer will take care of the 'fragmentation' and not your harddisk.
> If you'll note the format of the recordings and that we'll be recording 
> up to 200,000 minutes of calls a day, with a little quick math you'll 
> realize that it would take 80 to 100 GBs of memory for us to buffer a 
> full day's recordings.  Combined with the fact that a server failure 
> late in the day would cause us to lose them all, this is not a desirable 
> solution.
> 
> Instead, we plan to write an application to call from the MONITOR_EXEC 
> hook that will be executed at the end of each call.  This application 
> will be niced down to the lowest priority, and simply copy the leg files 
> from memory to disk.  Under normal conditions (ie. our NFS server is up) 
> this will actually be a copy to a remote disk using an asynchronous NFS 
> transfer.  All actual disk I/O will occur on our digital recording 
> server and any handling of the digital recordings will occur only after 
> the call they are bound to is completed.
> 
> Do you have any suggestions regarding this scenario?
> 
> - - on the remove server, do all the mixing / indexing etc. (i really
> -   don't mixing or converting between audio formats on the same server as
> -   asterisk).
> That exactly describes the functionality and purpose of our digital 
> recording server.  We do no transcoding on the Asterisk server and 
> perform no DSP on it.  It handles only SIP traffic, as our Ts are 
> terminated by a Cisco AS5400HPX Universal Gateway.
> 
> - If you want to go even freakier, run asterisk (or you complete distro)
> - from a ramdisk.
> -
> - Oh, another thing, for the people trying this the performance of hdparm
> - is not linear with the quality of your calls, tweaking your disks to be
> - faster will not help for asterisk when you do a copy. (in general).
> Thanks. I will keep this information in mind.
> 
> - I thought over your suggestion to use a sniffer to do the recordings,
> - you might pull it off, but will have to write your own to do so. (or go
> - to the expensive version of commercial sniffer applications).
> Trust me, using a sniffer is not our ideal solution. But if push comes 
> to shove...
> 
> - Also when you do things over the network, disable your onboard network
> - card, and go for some more expensive network card.
> - In our tests with small packets, we could increase the throughput with a
> - factor 2. (related to cpu load).
> -
> -...the dual nic gigabit intel card was a lot better than all the other
> - combinations i tried.
> Done.  We were going to use a Level 5 Networks EF1-21022T NIC, but it 
> didn't like performing DMA with anything greater than 4 GB of system 
> RAM.  Level 5 is working on resolving this issue with their drivers, but 
> until then we are using an Intel Pro 1000 MT NIC installed in a 64-bit, 
> 133 MHz PCI-X expansion slot.  This Intel NIC is actually the onboard 
> NIC on our digital recording server (a Dell PowerEdge 1850).
> 
> ============================================================
> 
> Trixter,
> 
> - filesystems are also a consideration with larger scale projects.
> - Different filesystems add different amounts of overheads on different
> - types of operations.  Some are faster at moving small files around
> - others faster with large files.  This adds to the disk latency.
> - Removing the disk latency itself is a good thing, since that is
> - typically slower, but to crank out that last little bit of performance
> - some research into the different filesystems under the specific kernel
> - that you are using could also be a consideration.  The most obvious
> - (and easiest to update a running system) is to remove things like atime,
> - whih with most linux distros is on by default.  This causes a write
> - operation for the read of a file to update the last time accessed.  A
> - couple little things can add up to a few percent improvement and
> - generally make the cost go down.
> We may need every percent we can get, so thanks for the suggestions.  As 
> mentioned above, we're running Fedora Core 3 with the latest 2.6 SMP 
> kernel.  Our physical disks are using the ext3 file system, while our 
> RAM disk is using ext2.
> 
> - Or have a seperate network set up (dual nic card for example) where the
> - 2nd network is used just for NFS traffic.  Although NFS generally is
> - ugly network wise, it is standard and makes things easier.  Just gotta
> - watch the IO on the system given that the network card itself will cause
> - cpu cycles to be used, but lets face it cpu is cheap now.  Different
> - drivers also work differently, and then with the 2.6 series kernels you
> - can use device polling instead of interupts which can help a little.
> This seems like something we may pursue in the short term.  We are using 
> NFS (which will be configuring for asynchronous transfers), but not with 
> a dedicated NIC.  Our system has four 3.16 GHz Intel Xeon processors.  
> They are hyperthreaded, so the OS sees them as eight separate 
> processors.  During our 256 call testing they hit 70% idle, but during 
> our 512 call testing they dropped to 10% idle.  Obviously, 512 calls is 
> pushing the limits but we only really need to hit around 300 for our 
> current project.  Nonetheless, the additional CPU cycles consumed by 
> another NIC are something to watch for.  Luckily, Intel will be 
> releasing their dual-core processors soon.
> 
> - When you say ramdisk here I assume you mean using conventional ram, its
> - cheap yes but its volatile, do you have any plans for failure of the
> - system or ram?  Or is the data integrity itself not as critical?  The
> - reason that people like hard drives is because most of the time if the
> - system goes down for any reason the data is still intact.
> Please see my responses to Matt regarding our RAM disk configuration.
> 
> - isnt vomit free?  It was a voip sniffer that worked with some codecs
> - many years ago (I wanna say mid-late 90s but I may be thinking of
> - another back then). http://vomit.xtdnet.nl/ does G.711 only.
> -
> - The bigger prIoblem that I see is that sniffers dont always get all the
> - traffic that is on a network particularly when the network has more
> - traffic on it.  While this generally isnt a concern and I would like to
> - think that even a poorly configured network could allow for 512 calls,
> - it is a factor to implement this type of a solution.
> Hopefully, it won't come to this.  With all of your help, I doubt it will.
> 
> ============================================================
> 
> Richard,
> 
> - Another project snatched from the Toilet bowl ahead of the flush.
> With any luck, we'll be able to shine this turd.
> 
> ============================================================
> 
> I apologize for the length of this post, but after consideration it 
> seemed to be the most efficient way to address each of you.
> 
> Thank you all,
> 
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
> _______________________________________________
> --Bandwidth and Colocation sponsored by Easynews.com --
> 
> Asterisk-Users mailing list
> Asterisk-Users at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mhess.vcf
Type: text/x-vcard
Size: 288 bytes
Desc: not available
Url : http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/f4d709bd/mhess.vcf