[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

Wed Sep 21 12:04:46 MST 2005

The problem is that then it won't work on systems with little memory. 50
streams would eat memory like crazy.

Zoa

Matt Hess wrote:

> In light of the I/O bottleneck problem I'd have to ask why asterisk
> can't just buffer incoming audio and then flush a complete audio file
> to disk.. I'm assuming that recordings vary in length.. the problem
> with this idea is what happens if 50 recordings all complete at the
> same time.. a dump like that might not be very pretty (a fast drive
> plus a little scheduler or limiter so that only x number of files get
> written to disk at a time would probably help out there) but I can
> imagine that a single file being written is much more efficient and
> more disk-friendly.. perhaps I don't know what the heck I'm talking
> about but  at least in my mind the theory sounds better than the
> current 'stream-to-file' method employed by asterisk.
>
>
>
> Matt Roth wrote:
>
>> All,
>>
>> This message has generated a lot of responses, so I'm going to
>> address each of them here in an attempt to consolidate the thread.
>>
>> ============================================================
>>
>> Matt,
>>
>> - I'm very interested in the specifics of your setup.
>> - How much space is on the RAM disk?
>> Currently it is 10 GB.  We are upgrading it to 16 GB.
>>
>> - What kind of RAM drive is it?
>> The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs.
>> The details for each 1 GB DIMM can be seen here:
>>
>> http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm
>>
>>
>> The upgrade will involve adding 2 GB DIMMs to the system, but I don't
>> have the details on these yet.
>>
>> The RAM disk is setup by adding the following kernel command line
>> option to grub.conf:
>>
>> ramdisk_size=10485760
>>
>> We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
>>
>> By default the RAM disk's block size is 1024 bytes, so we are
>> formatting it as an ext2 file system with a block size of 1024 bytes
>> using the following command:
>>
>> mke2fs -b 1024 -m 0 /dev/ram0
>>
>> The block size can easily be changed from the kernel's view (using
>> the kernel command line option ramdisk_blocksize=XXXX) or from
>> mke2fs's view (using the -b XXXX argument), so please let me know if
>> I can make an easy optimization here.
>>
>> Finally, the RAM disk is mounted using the command:
>>
>> mount /dev/ram0 /digrec
>>
>> A good RAMDISK howto exists at:
>>
>> http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
>>
>> - What format are you recording to?
>> - What codec are the SIP calls being placed over?
>> We are recording to the PCM format and using the G711 uLaw codec.
>> High voice quality is essential to our application (we are a call
>> center) so we partnered with MCI to configure our network for the
>> required bandwidth and chose the highest quality, zero compression
>> codec.  We noload all other codecs in order to avoid transcoding on
>> the switch, so we must record to PCM. Later (on a separate server)
>> the recordings are mixed to GSM which provides a 5 to 1 compression
>> ratio with very little artifacts.
>>
>> - We've run into the "Avoided deadlock" recording issues several
>> times when trying to do
>> - more than 50 concurrent recordings. Changing the ast_channel_lock
>> loop from 10 to 20 has
>> - helped somewhat reduce the warnings and reduce audio gaps on the
>> recordings, but what is
>> - really needed for more robust recording is a configurable recording
>> buffer that wouldn't
>> - freak out if a 10ms delay occurs.
>> Are you saying that these messages indicate a gap in a digital
>> recording?  If so, what is the duration of the gap? If it's
>> comparable to a CD skip, I think we can deal with it until a buffer
>> or another solution is implemented.
>>
>> - Good luck and please keep us updated on your progress
>> Thank you.  I'll be keeping the list informed of our progress.
>>
>> ============================================================
>>
>> Zoa,
>>
>> - I suppose you are the person from the digium forum
>> That was actually my boss's boss.  We thank you all the way up and
>> down the line for your suggestion.
>>
>> - The reason i recommended you to use a ramdisk is because i think the
>> - problem with recording to disk is saving 20ms of stream 1, then 20
>> ms of
>> - stream 2, then 20ms of stream 3 etc etc.... meaning you write
>> everytime
>> - very small things. (with a lot of seeking).
>> Agreed.  This is why we hope that decoupling the copy (memory to
>> disk) from Asterisk itself and, more importantly, Asterisk's
>> real-time handling of the call being recorded will be sufficient.
>>
>> For the record, when recording 512 simultaneous calls to the local
>> disk we saw a peek of about 13,000 blocks written per second.
>>
>> - Our best test results were with:
>> -
>> - - buffering the recordings to a ramdisk, then
>> We're doing that, as per your suggestion.
>>
>> - - on low load (at night) copy the files over the network (easy to
>> shape
>> -   the pipe, so that you dont overload anything), This way, the memory
>> -   buffer will take care of the 'fragmentation' and not your harddisk.
>> If you'll note the format of the recordings and that we'll be
>> recording up to 200,000 minutes of calls a day, with a little quick
>> math you'll realize that it would take 80 to 100 GBs of memory for us
>> to buffer a full day's recordings.  Combined with the fact that a
>> server failure late in the day would cause us to lose them all, this
>> is not a desirable solution.
>>
>> Instead, we plan to write an application to call from the
>> MONITOR_EXEC hook that will be executed at the end of each call.
>> This application will be niced down to the lowest priority, and
>> simply copy the leg files from memory to disk.  Under normal
>> conditions (ie. our NFS server is up) this will actually be a copy to
>> a remote disk using an asynchronous NFS transfer.  All actual disk
>> I/O will occur on our digital recording server and any handling of
>> the digital recordings will occur only after the call they are bound
>> to is completed.
>>
>> Do you have any suggestions regarding this scenario?
>>
>> - - on the remove server, do all the mixing / indexing etc. (i really
>> -   don't mixing or converting between audio formats on the same
>> server as
>> -   asterisk).
>> That exactly describes the functionality and purpose of our digital
>> recording server.  We do no transcoding on the Asterisk server and
>> perform no DSP on it.  It handles only SIP traffic, as our Ts are
>> terminated by a Cisco AS5400HPX Universal Gateway.
>>
>> - If you want to go even freakier, run asterisk (or you complete distro)
>> - from a ramdisk.
>> -
>> - Oh, another thing, for the people trying this the performance of
>> hdparm
>> - is not linear with the quality of your calls, tweaking your disks
>> to be
>> - faster will not help for asterisk when you do a copy. (in general).
>> Thanks. I will keep this information in mind.
>>
>> - I thought over your suggestion to use a sniffer to do the recordings,
>> - you might pull it off, but will have to write your own to do so.
>> (or go
>> - to the expensive version of commercial sniffer applications).
>> Trust me, using a sniffer is not our ideal solution. But if push
>> comes to shove...
>>
>> - Also when you do things over the network, disable your onboard network
>> - card, and go for some more expensive network card.
>> - In our tests with small packets, we could increase the throughput
>> with a
>> - factor 2. (related to cpu load).
>> -
>> -...the dual nic gigabit intel card was a lot better than all the other
>> - combinations i tried.
>> Done.  We were going to use a Level 5 Networks EF1-21022T NIC, but it
>> didn't like performing DMA with anything greater than 4 GB of system
>> RAM.  Level 5 is working on resolving this issue with their drivers,
>> but until then we are using an Intel Pro 1000 MT NIC installed in a
>> 64-bit, 133 MHz PCI-X expansion slot.  This Intel NIC is actually the
>> onboard NIC on our digital recording server (a Dell PowerEdge 1850).
>>
>> ============================================================
>>
>> Trixter,
>>
>> - filesystems are also a consideration with larger scale projects.
>> - Different filesystems add different amounts of overheads on different
>> - types of operations.  Some are faster at moving small files around
>> - others faster with large files.  This adds to the disk latency.
>> - Removing the disk latency itself is a good thing, since that is
>> - typically slower, but to crank out that last little bit of performance
>> - some research into the different filesystems under the specific kernel
>> - that you are using could also be a consideration.  The most obvious
>> - (and easiest to update a running system) is to remove things like
>> atime,
>> - whih with most linux distros is on by default.  This causes a write
>> - operation for the read of a file to update the last time accessed.  A
>> - couple little things can add up to a few percent improvement and
>> - generally make the cost go down.
>> We may need every percent we can get, so thanks for the suggestions.
>> As mentioned above, we're running Fedora Core 3 with the latest 2.6
>> SMP kernel.  Our physical disks are using the ext3 file system, while
>> our RAM disk is using ext2.
>>
>> - Or have a seperate network set up (dual nic card for example) where
>> the
>> - 2nd network is used just for NFS traffic.  Although NFS generally is
>> - ugly network wise, it is standard and makes things easier.  Just gotta
>> - watch the IO on the system given that the network card itself will
>> cause
>> - cpu cycles to be used, but lets face it cpu is cheap now.  Different
>> - drivers also work differently, and then with the 2.6 series kernels
>> you
>> - can use device polling instead of interupts which can help a little.
>> This seems like something we may pursue in the short term.  We are
>> using NFS (which will be configuring for asynchronous transfers), but
>> not with a dedicated NIC.  Our system has four 3.16 GHz Intel Xeon
>> processors.  They are hyperthreaded, so the OS sees them as eight
>> separate processors.  During our 256 call testing they hit 70% idle,
>> but during our 512 call testing they dropped to 10% idle.  Obviously,
>> 512 calls is pushing the limits but we only really need to hit around
>> 300 for our current project.  Nonetheless, the additional CPU cycles
>> consumed by another NIC are something to watch for.  Luckily, Intel
>> will be releasing their dual-core processors soon.
>>
>> - When you say ramdisk here I assume you mean using conventional ram,
>> its
>> - cheap yes but its volatile, do you have any plans for failure of the
>> - system or ram?  Or is the data integrity itself not as critical?  The
>> - reason that people like hard drives is because most of the time if the
>> - system goes down for any reason the data is still intact.
>> Please see my responses to Matt regarding our RAM disk configuration.
>>
>> - isnt vomit free?  It was a voip sniffer that worked with some codecs
>> - many years ago (I wanna say mid-late 90s but I may be thinking of
>> - another back then). http://vomit.xtdnet.nl/ does G.711 only.
>> -
>> - The bigger prIoblem that I see is that sniffers dont always get all
>> the
>> - traffic that is on a network particularly when the network has more
>> - traffic on it.  While this generally isnt a concern and I would
>> like to
>> - think that even a poorly configured network could allow for 512 calls,
>> - it is a factor to implement this type of a solution.
>> Hopefully, it won't come to this.  With all of your help, I doubt it
>> will.
>>
>> ============================================================
>>
>> Richard,
>>
>> - Another project snatched from the Toilet bowl ahead of the flush.
>> With any luck, we'll be able to shine this turd.
>>
>> ============================================================
>>
>> I apologize for the length of this post, but after consideration it
>> seemed to be the most efficient way to address each of you.
>>
>> Thank you all,
>>
>> Matthew Roth
>> InterMedia Marketing Solutions
>> Software Engineer and Systems Developer
>> _______________________________________________
>> --Bandwidth and Colocation sponsored by Easynews.com --
>>
>> Asterisk-Users mailing list
>> Asterisk-Users at lists.digium.com
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>>
>_______________________________________________
>--Bandwidth and Colocation sponsored by Easynews.com --
>
>Asterisk-Users mailing list
>Asterisk-Users at lists.digium.com
>http://lists.digium.com/mailman/listinfo/asterisk-users
>To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/3203bd95/signature.pgp