[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording

Zoa zoachien at securax.org
Wed Sep 21 12:49:05 MST 2005 


I think its the best you can do.

Maybe there should be some option to be set for the monitor command to
buffer, with a warning that it will eat memory.
Its also not needed to buffer the complete call at once, just buffering
and writing to disk every 10 seconds would already be a big improvement,
and since the calls won't all start at the same time, the writing to
disk will be spread over time.
Good luck with the implementation, let us know how it turns out!
Try to find out what happens if the nfs share is gone, or overloaded too.

If there is no zaptel hardware involved, i don't think you have to be
too concerned about the network cards interrupts.

greetings,

Zoa.

Matt Roth wrote:

> It's true that the average Asterisk implementation doesn't have enough
> RAM, but we are replacing a legacy NorTel switch in a call center.  If
> you look at the cost of traditional PBXs, the cost of additional
> memory starts to look a little better.  = )
>
> Now for some quick math:
>
> 1 minute of PCM audio = 480 KB
> * 2 leg files ~= 1 MB/minute
> Avail. Mem. = 10 GB = 10,000 MB = 10,000 minutes of digital recordings
> Peak calling = 200,000 minutes/day
> 10,000 / 200,000 = 5%
>
> So our buffer is 5% of our total calling for the day.  We'll be
> bumping the RAM disk up to 16 GB yielding an 8% buffer.
>
> We'll be moving calls out of memory to a remote disk (via NFS) as soon
> as they are finished, so I think we'll be okay.  We'll be monitoring
> memory usage and sharing our data.  It's my hope that this is the
> solution for large scale (250-500 simultaneous calls) Asterisk
> installations.
>
> I hope my math was right,
>
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
>
> Zoa wrote:
>
> - The problem is that then it won't work on systems with little
> memory. 50
> - streams would eat memory like crazy.
> -
> - Zoa
>
> info at beprojects.com wrote:
>
>> I would think memory would be the limiting factor.  A 3-4 minute wav
>> file is what, 30Meg or so?  And there is one for each end of the
>> call, so that's 60Meg.  Now let's say it's a 15 minute call and then
>> are 10 of them at once.  That's 30Meg x 5 (5 times the length of my
>> estimate) x 2 (each leg) x 10 simultaneous calls....equals 3 Gig of
>> RAM.  Once you run out of RAM, then what does it do?  It would have
>> to try and dump it all to disk at once and you are back where you
>> started.  I think the average * implementation doesn't have nearly
>> enough free RAM to do this.
>>
>> The RAMdisk solution seems to be pretty elegant in it's simplicity.
>>
>>
>>
>> Matt Hess wrote:
>>
>>> In light of the I/O bottleneck problem I'd have to ask why asterisk
>>> can't just buffer incoming audio and then flush a complete audio
>>> file to disk.. I'm assuming that recordings vary in length.. the
>>> problem with this idea is what happens if 50 recordings all complete
>>> at the same time.. a dump like that might not be very pretty (a fast
>>> drive plus a little scheduler or limiter so that only x number of
>>> files get written to disk at a time would probably help out there)
>>> but I can imagine that a single file being written is much more
>>> efficient and more disk-friendly.. perhaps I don't know what the
>>> heck I'm talking about but  at least in my mind the theory sounds
>>> better than the current 'stream-to-file' method employed by asterisk.
>>>
>>>
>>>
>>> Matt Roth wrote:
>>>
>>>> All,
>>>>
>>>> This message has generated a lot of responses, so I'm going to
>>>> address each of them here in an attempt to consolidate the thread.
>>>>
>>>> ============================================================
>>>>
>>>> Matt,
>>>>
>>>> - I'm very interested in the specifics of your setup.
>>>> - How much space is on the RAM disk?
>>>> Currently it is 10 GB.  We are upgrading it to 16 GB.
>>>>
>>>> - What kind of RAM drive is it?
>>>> The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked
>>>> DIMMs. The details for each 1 GB DIMM can be seen here:
>>>>
>>>> http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm
>>>>
>>>>
>>>> The upgrade will involve adding 2 GB DIMMs to the system, but I
>>>> don't have the details on these yet.
>>>>
>>>> The RAM disk is setup by adding the following kernel command line
>>>> option to grub.conf:
>>>>
>>>> ramdisk_size=10485760
>>>>
>>>> We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
>>>>
>>>> By default the RAM disk's block size is 1024 bytes, so we are
>>>> formatting it as an ext2 file system with a block size of 1024
>>>> bytes using the following command:
>>>>
>>>> mke2fs -b 1024 -m 0 /dev/ram0
>>>>
>>>> The block size can easily be changed from the kernel's view (using
>>>> the kernel command line option ramdisk_blocksize=XXXX) or from
>>>> mke2fs's view (using the -b XXXX argument), so please let me know
>>>> if I can make an easy optimization here.
>>>>
>>>> Finally, the RAM disk is mounted using the command:
>>>>
>>>> mount /dev/ram0 /digrec
>>>>
>>>> A good RAMDISK howto exists at:
>>>>
>>>> http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
>>>>
>>>> - What format are you recording to?
>>>> - What codec are the SIP calls being placed over?
>>>> We are recording to the PCM format and using the G711 uLaw codec.
>>>> High voice quality is essential to our application (we are a call
>>>> center) so we partnered with MCI to configure our network for the
>>>> required bandwidth and chose the highest quality, zero compression
>>>> codec.  We noload all other codecs in order to avoid transcoding on
>>>> the switch, so we must record to PCM. Later (on a separate server)
>>>> the recordings are mixed to GSM which provides a 5 to 1 compression
>>>> ratio with very little artifacts.
>>>>
>>>> - We've run into the "Avoided deadlock" recording issues several
>>>> times when trying to do
>>>> - more than 50 concurrent recordings. Changing the ast_channel_lock
>>>> loop from 10 to 20 has
>>>> - helped somewhat reduce the warnings and reduce audio gaps on the
>>>> recordings, but what is
>>>> - really needed for more robust recording is a configurable
>>>> recording buffer that wouldn't
>>>> - freak out if a 10ms delay occurs.
>>>> Are you saying that these messages indicate a gap in a digital
>>>> recording?  If so, what is the duration of the gap? If it's
>>>> comparable to a CD skip, I think we can deal with it until a buffer
>>>> or another solution is implemented.
>>>>
>>>> - Good luck and please keep us updated on your progress
>>>> Thank you.  I'll be keeping the list informed of our progress.
>>>>
>>>> ============================================================
>>>>
>>>> Zoa,
>>>>
>>>> - I suppose you are the person from the digium forum
>>>> That was actually my boss's boss.  We thank you all the way up and
>>>> down the line for your suggestion.
>>>>
>>>> - The reason i recommended you to use a ramdisk is because i think the
>>>> - problem with recording to disk is saving 20ms of stream 1, then
>>>> 20 ms of
>>>> - stream 2, then 20ms of stream 3 etc etc.... meaning you write
>>>> everytime
>>>> - very small things. (with a lot of seeking).
>>>> Agreed.  This is why we hope that decoupling the copy (memory to
>>>> disk) from Asterisk itself and, more importantly, Asterisk's
>>>> real-time handling of the call being recorded will be sufficient.
>>>>
>>>> For the record, when recording 512 simultaneous calls to the local
>>>> disk we saw a peek of about 13,000 blocks written per second.
>>>>
>>>> - Our best test results were with:
>>>> -
>>>> - - buffering the recordings to a ramdisk, then
>>>> We're doing that, as per your suggestion.
>>>>
>>>> - - on low load (at night) copy the files over the network (easy to
>>>> shape
>>>> -   the pipe, so that you dont overload anything), This way, the
>>>> memory
>>>> -   buffer will take care of the 'fragmentation' and not your
>>>> harddisk.
>>>> If you'll note the format of the recordings and that we'll be
>>>> recording up to 200,000 minutes of calls a day, with a little quick
>>>> math you'll realize that it would take 80 to 100 GBs of memory for
>>>> us to buffer a full day's recordings.  Combined with the fact that
>>>> a server failure late in the day would cause us to lose them all,
>>>> this is not a desirable solution.
>>>>
>>>> Instead, we plan to write an application to call from the
>>>> MONITOR_EXEC hook that will be executed at the end of each call.
>>>> This application will be niced down to the lowest priority, and
>>>> simply copy the leg files from memory to disk.  Under normal
>>>> conditions (ie. our NFS server is up) this will actually be a copy
>>>> to a remote disk using an asynchronous NFS transfer.  All actual
>>>> disk I/O will occur on our digital recording server and any
>>>> handling of the digital recordings will occur only after the call
>>>> they are bound to is completed.
>>>>
>>>> Do you have any suggestions regarding this scenario?
>>>>
>>>> - - on the remove server, do all the mixing / indexing etc. (i really
>>>> -   don't mixing or converting between audio formats on the same
>>>> server as
>>>> -   asterisk).
>>>> That exactly describes the functionality and purpose of our digital
>>>> recording server.  We do no transcoding on the Asterisk server and
>>>> perform no DSP on it.  It handles only SIP traffic, as our Ts are
>>>> terminated by a Cisco AS5400HPX Universal Gateway.
>>>>
>>>> - If you want to go even freakier, run asterisk (or you complete
>>>> distro)
>>>> - from a ramdisk.
>>>> -
>>>> - Oh, another thing, for the people trying this the performance of
>>>> hdparm
>>>> - is not linear with the quality of your calls, tweaking your disks
>>>> to be
>>>> - faster will not help for asterisk when you do a copy. (in general).
>>>> Thanks. I will keep this information in mind.
>>>>
>>>> - I thought over your suggestion to use a sniffer to do the
>>>> recordings,
>>>> - you might pull it off, but will have to write your own to do so.
>>>> (or go
>>>> - to the expensive version of commercial sniffer applications).
>>>> Trust me, using a sniffer is not our ideal solution. But if push
>>>> comes to shove...
>>>>
>>>> - Also when you do things over the network, disable your onboard
>>>> network
>>>> - card, and go for some more expensive network card.
>>>> - In our tests with small packets, we could increase the throughput
>>>> with a
>>>> - factor 2. (related to cpu load).
>>>> -
>>>> -...the dual nic gigabit intel card was a lot better than all the
>>>> other
>>>> - combinations i tried.
>>>> Done.  We were going to use a Level 5 Networks EF1-21022T NIC, but
>>>> it didn't like performing DMA with anything greater than 4 GB of
>>>> system RAM.  Level 5 is working on resolving this issue with their
>>>> drivers, but until then we are using an Intel Pro 1000 MT NIC
>>>> installed in a 64-bit, 133 MHz PCI-X expansion slot.  This Intel
>>>> NIC is actually the onboard NIC on our digital recording server (a
>>>> Dell PowerEdge 1850).
>>>>
>>>> ============================================================
>>>>
>>>> Trixter,
>>>>
>>>> - filesystems are also a consideration with larger scale projects.
>>>> - Different filesystems add different amounts of overheads on
>>>> different
>>>> - types of operations.  Some are faster at moving small files around
>>>> - others faster with large files.  This adds to the disk latency.
>>>> - Removing the disk latency itself is a good thing, since that is
>>>> - typically slower, but to crank out that last little bit of
>>>> performance
>>>> - some research into the different filesystems under the specific
>>>> kernel
>>>> - that you are using could also be a consideration.  The most obvious
>>>> - (and easiest to update a running system) is to remove things like
>>>> atime,
>>>> - whih with most linux distros is on by default.  This causes a write
>>>> - operation for the read of a file to update the last time
>>>> accessed.  A
>>>> - couple little things can add up to a few percent improvement and
>>>> - generally make the cost go down.
>>>> We may need every percent we can get, so thanks for the
>>>> suggestions.  As mentioned above, we're running Fedora Core 3 with
>>>> the latest 2.6 SMP kernel.  Our physical disks are using the ext3
>>>> file system, while our RAM disk is using ext2.
>>>>
>>>> - Or have a seperate network set up (dual nic card for example)
>>>> where the
>>>> - 2nd network is used just for NFS traffic.  Although NFS generally is
>>>> - ugly network wise, it is standard and makes things easier.  Just
>>>> gotta
>>>> - watch the IO on the system given that the network card itself
>>>> will cause
>>>> - cpu cycles to be used, but lets face it cpu is cheap now.  Different
>>>> - drivers also work differently, and then with the 2.6 series
>>>> kernels you
>>>> - can use device polling instead of interupts which can help a little.
>>>> This seems like something we may pursue in the short term.  We are
>>>> using NFS (which will be configuring for asynchronous transfers),
>>>> but not with a dedicated NIC.  Our system has four 3.16 GHz Intel
>>>> Xeon processors.  They are hyperthreaded, so the OS sees them as
>>>> eight separate processors.  During our 256 call testing they hit
>>>> 70% idle, but during our 512 call testing they dropped to 10%
>>>> idle.  Obviously, 512 calls is pushing the limits but we only
>>>> really need to hit around 300 for our current project.
>>>> Nonetheless, the additional CPU cycles consumed by another NIC are
>>>> something to watch for.  Luckily, Intel will be releasing their
>>>> dual-core processors soon.
>>>>
>>>> - When you say ramdisk here I assume you mean using conventional
>>>> ram, its
>>>> - cheap yes but its volatile, do you have any plans for failure of the
>>>> - system or ram?  Or is the data integrity itself not as critical?
>>>> The
>>>> - reason that people like hard drives is because most of the time
>>>> if the
>>>> - system goes down for any reason the data is still intact.
>>>> Please see my responses to Matt regarding our RAM disk configuration.
>>>>
>>>> - isnt vomit free?  It was a voip sniffer that worked with some codecs
>>>> - many years ago (I wanna say mid-late 90s but I may be thinking of
>>>> - another back then). http://vomit.xtdnet.nl/ does G.711 only.
>>>> -
>>>> - The bigger prIoblem that I see is that sniffers dont always get
>>>> all the
>>>> - traffic that is on a network particularly when the network has more
>>>> - traffic on it.  While this generally isnt a concern and I would
>>>> like to
>>>> - think that even a poorly configured network could allow for 512
>>>> calls,
>>>> - it is a factor to implement this type of a solution.
>>>> Hopefully, it won't come to this.  With all of your help, I doubt
>>>> it will.
>>>>
>>>> ============================================================
>>>>
>>>> Richard,
>>>>
>>>> - Another project snatched from the Toilet bowl ahead of the flush.
>>>> With any luck, we'll be able to shine this turd.
>>>>
>>>> ============================================================
>>>>
>>>> I apologize for the length of this post, but after consideration it
>>>> seemed to be the most efficient way to address each of you.
>>>>
>>>> Thank you all,
>>>>
>>>> Matthew Roth
>>>> InterMedia Marketing Solutions
>>>> Software Engineer and Systems Developer
>>>> _______________________________________________
>>>> --Bandwidth and Colocation sponsored by Easynews.com --
>>>>
>>>> Asterisk-Users mailing list
>>>> Asterisk-Users at lists.digium.com
>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>> To UNSUBSCRIBE or update options visit:
>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> --Bandwidth and Colocation sponsored by Easynews.com --
>>>
>>> Asterisk-Users mailing list
>>> Asterisk-Users at lists.digium.com
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>> To UNSUBSCRIBE or update options visit:
>>>    http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>>
>> _______________________________________________
>> --Bandwidth and Colocation sponsored by Easynews.com --
>>
>> Asterisk-Users mailing list
>> Asterisk-Users at lists.digium.com
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>
> _______________________________________________
> --Bandwidth and Colocation sponsored by Easynews.com --
>
> Asterisk-Users mailing list
> Asterisk-Users at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/740b0003/signature.pgp

More information about the asterisk-users mailing list