[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital
Recording
Matt Roth
mroth at imminc.com
Wed Sep 21 12:29:10 MST 2005
It's true that the average Asterisk implementation doesn't have enough
RAM, but we are replacing a legacy NorTel switch in a call center. If
you look at the cost of traditional PBXs, the cost of additional memory
starts to look a little better. = )
Now for some quick math:
1 minute of PCM audio = 480 KB
* 2 leg files ~= 1 MB/minute
Avail. Mem. = 10 GB = 10,000 MB = 10,000 minutes of digital recordings
Peak calling = 200,000 minutes/day
10,000 / 200,000 = 5%
So our buffer is 5% of our total calling for the day. We'll be bumping
the RAM disk up to 16 GB yielding an 8% buffer.
We'll be moving calls out of memory to a remote disk (via NFS) as soon
as they are finished, so I think we'll be okay. We'll be monitoring
memory usage and sharing our data. It's my hope that this is the
solution for large scale (250-500 simultaneous calls) Asterisk
installations.
I hope my math was right,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
Zoa wrote:
- The problem is that then it won't work on systems with little memory. 50
- streams would eat memory like crazy.
-
- Zoa
info at beprojects.com wrote:
> I would think memory would be the limiting factor. A 3-4 minute wav
> file is what, 30Meg or so? And there is one for each end of the call,
> so that's 60Meg. Now let's say it's a 15 minute call and then are 10
> of them at once. That's 30Meg x 5 (5 times the length of my estimate)
> x 2 (each leg) x 10 simultaneous calls....equals 3 Gig of RAM. Once
> you run out of RAM, then what does it do? It would have to try and
> dump it all to disk at once and you are back where you started. I
> think the average * implementation doesn't have nearly enough free RAM
> to do this.
>
> The RAMdisk solution seems to be pretty elegant in it's simplicity.
>
>
>
> Matt Hess wrote:
>
>> In light of the I/O bottleneck problem I'd have to ask why asterisk
>> can't just buffer incoming audio and then flush a complete audio file
>> to disk.. I'm assuming that recordings vary in length.. the problem
>> with this idea is what happens if 50 recordings all complete at the
>> same time.. a dump like that might not be very pretty (a fast drive
>> plus a little scheduler or limiter so that only x number of files get
>> written to disk at a time would probably help out there) but I can
>> imagine that a single file being written is much more efficient and
>> more disk-friendly.. perhaps I don't know what the heck I'm talking
>> about but at least in my mind the theory sounds better than the
>> current 'stream-to-file' method employed by asterisk.
>>
>>
>>
>> Matt Roth wrote:
>>
>>> All,
>>>
>>> This message has generated a lot of responses, so I'm going to
>>> address each of them here in an attempt to consolidate the thread.
>>>
>>> ============================================================
>>>
>>> Matt,
>>>
>>> - I'm very interested in the specifics of your setup.
>>> - How much space is on the RAM disk?
>>> Currently it is 10 GB. We are upgrading it to 16 GB.
>>>
>>> - What kind of RAM drive is it?
>>> The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked
>>> DIMMs. The details for each 1 GB DIMM can be seen here:
>>>
>>> http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm
>>>
>>>
>>> The upgrade will involve adding 2 GB DIMMs to the system, but I
>>> don't have the details on these yet.
>>>
>>> The RAM disk is setup by adding the following kernel command line
>>> option to grub.conf:
>>>
>>> ramdisk_size=10485760
>>>
>>> We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
>>>
>>> By default the RAM disk's block size is 1024 bytes, so we are
>>> formatting it as an ext2 file system with a block size of 1024 bytes
>>> using the following command:
>>>
>>> mke2fs -b 1024 -m 0 /dev/ram0
>>>
>>> The block size can easily be changed from the kernel's view (using
>>> the kernel command line option ramdisk_blocksize=XXXX) or from
>>> mke2fs's view (using the -b XXXX argument), so please let me know if
>>> I can make an easy optimization here.
>>>
>>> Finally, the RAM disk is mounted using the command:
>>>
>>> mount /dev/ram0 /digrec
>>>
>>> A good RAMDISK howto exists at:
>>>
>>> http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
>>>
>>> - What format are you recording to?
>>> - What codec are the SIP calls being placed over?
>>> We are recording to the PCM format and using the G711 uLaw codec.
>>> High voice quality is essential to our application (we are a call
>>> center) so we partnered with MCI to configure our network for the
>>> required bandwidth and chose the highest quality, zero compression
>>> codec. We noload all other codecs in order to avoid transcoding on
>>> the switch, so we must record to PCM. Later (on a separate server)
>>> the recordings are mixed to GSM which provides a 5 to 1 compression
>>> ratio with very little artifacts.
>>>
>>> - We've run into the "Avoided deadlock" recording issues several
>>> times when trying to do
>>> - more than 50 concurrent recordings. Changing the ast_channel_lock
>>> loop from 10 to 20 has
>>> - helped somewhat reduce the warnings and reduce audio gaps on the
>>> recordings, but what is
>>> - really needed for more robust recording is a configurable
>>> recording buffer that wouldn't
>>> - freak out if a 10ms delay occurs.
>>> Are you saying that these messages indicate a gap in a digital
>>> recording? If so, what is the duration of the gap? If it's
>>> comparable to a CD skip, I think we can deal with it until a buffer
>>> or another solution is implemented.
>>>
>>> - Good luck and please keep us updated on your progress
>>> Thank you. I'll be keeping the list informed of our progress.
>>>
>>> ============================================================
>>>
>>> Zoa,
>>>
>>> - I suppose you are the person from the digium forum
>>> That was actually my boss's boss. We thank you all the way up and
>>> down the line for your suggestion.
>>>
>>> - The reason i recommended you to use a ramdisk is because i think the
>>> - problem with recording to disk is saving 20ms of stream 1, then 20
>>> ms of
>>> - stream 2, then 20ms of stream 3 etc etc.... meaning you write
>>> everytime
>>> - very small things. (with a lot of seeking).
>>> Agreed. This is why we hope that decoupling the copy (memory to
>>> disk) from Asterisk itself and, more importantly, Asterisk's
>>> real-time handling of the call being recorded will be sufficient.
>>>
>>> For the record, when recording 512 simultaneous calls to the local
>>> disk we saw a peek of about 13,000 blocks written per second.
>>>
>>> - Our best test results were with:
>>> -
>>> - - buffering the recordings to a ramdisk, then
>>> We're doing that, as per your suggestion.
>>>
>>> - - on low load (at night) copy the files over the network (easy to
>>> shape
>>> - the pipe, so that you dont overload anything), This way, the memory
>>> - buffer will take care of the 'fragmentation' and not your harddisk.
>>> If you'll note the format of the recordings and that we'll be
>>> recording up to 200,000 minutes of calls a day, with a little quick
>>> math you'll realize that it would take 80 to 100 GBs of memory for
>>> us to buffer a full day's recordings. Combined with the fact that a
>>> server failure late in the day would cause us to lose them all, this
>>> is not a desirable solution.
>>>
>>> Instead, we plan to write an application to call from the
>>> MONITOR_EXEC hook that will be executed at the end of each call.
>>> This application will be niced down to the lowest priority, and
>>> simply copy the leg files from memory to disk. Under normal
>>> conditions (ie. our NFS server is up) this will actually be a copy
>>> to a remote disk using an asynchronous NFS transfer. All actual
>>> disk I/O will occur on our digital recording server and any handling
>>> of the digital recordings will occur only after the call they are
>>> bound to is completed.
>>>
>>> Do you have any suggestions regarding this scenario?
>>>
>>> - - on the remove server, do all the mixing / indexing etc. (i really
>>> - don't mixing or converting between audio formats on the same
>>> server as
>>> - asterisk).
>>> That exactly describes the functionality and purpose of our digital
>>> recording server. We do no transcoding on the Asterisk server and
>>> perform no DSP on it. It handles only SIP traffic, as our Ts are
>>> terminated by a Cisco AS5400HPX Universal Gateway.
>>>
>>> - If you want to go even freakier, run asterisk (or you complete
>>> distro)
>>> - from a ramdisk.
>>> -
>>> - Oh, another thing, for the people trying this the performance of
>>> hdparm
>>> - is not linear with the quality of your calls, tweaking your disks
>>> to be
>>> - faster will not help for asterisk when you do a copy. (in general).
>>> Thanks. I will keep this information in mind.
>>>
>>> - I thought over your suggestion to use a sniffer to do the recordings,
>>> - you might pull it off, but will have to write your own to do so.
>>> (or go
>>> - to the expensive version of commercial sniffer applications).
>>> Trust me, using a sniffer is not our ideal solution. But if push
>>> comes to shove...
>>>
>>> - Also when you do things over the network, disable your onboard
>>> network
>>> - card, and go for some more expensive network card.
>>> - In our tests with small packets, we could increase the throughput
>>> with a
>>> - factor 2. (related to cpu load).
>>> -
>>> -...the dual nic gigabit intel card was a lot better than all the other
>>> - combinations i tried.
>>> Done. We were going to use a Level 5 Networks EF1-21022T NIC, but
>>> it didn't like performing DMA with anything greater than 4 GB of
>>> system RAM. Level 5 is working on resolving this issue with their
>>> drivers, but until then we are using an Intel Pro 1000 MT NIC
>>> installed in a 64-bit, 133 MHz PCI-X expansion slot. This Intel NIC
>>> is actually the onboard NIC on our digital recording server (a Dell
>>> PowerEdge 1850).
>>>
>>> ============================================================
>>>
>>> Trixter,
>>>
>>> - filesystems are also a consideration with larger scale projects.
>>> - Different filesystems add different amounts of overheads on different
>>> - types of operations. Some are faster at moving small files around
>>> - others faster with large files. This adds to the disk latency.
>>> - Removing the disk latency itself is a good thing, since that is
>>> - typically slower, but to crank out that last little bit of
>>> performance
>>> - some research into the different filesystems under the specific
>>> kernel
>>> - that you are using could also be a consideration. The most obvious
>>> - (and easiest to update a running system) is to remove things like
>>> atime,
>>> - whih with most linux distros is on by default. This causes a write
>>> - operation for the read of a file to update the last time accessed. A
>>> - couple little things can add up to a few percent improvement and
>>> - generally make the cost go down.
>>> We may need every percent we can get, so thanks for the
>>> suggestions. As mentioned above, we're running Fedora Core 3 with
>>> the latest 2.6 SMP kernel. Our physical disks are using the ext3
>>> file system, while our RAM disk is using ext2.
>>>
>>> - Or have a seperate network set up (dual nic card for example)
>>> where the
>>> - 2nd network is used just for NFS traffic. Although NFS generally is
>>> - ugly network wise, it is standard and makes things easier. Just
>>> gotta
>>> - watch the IO on the system given that the network card itself will
>>> cause
>>> - cpu cycles to be used, but lets face it cpu is cheap now. Different
>>> - drivers also work differently, and then with the 2.6 series
>>> kernels you
>>> - can use device polling instead of interupts which can help a little.
>>> This seems like something we may pursue in the short term. We are
>>> using NFS (which will be configuring for asynchronous transfers),
>>> but not with a dedicated NIC. Our system has four 3.16 GHz Intel
>>> Xeon processors. They are hyperthreaded, so the OS sees them as
>>> eight separate processors. During our 256 call testing they hit 70%
>>> idle, but during our 512 call testing they dropped to 10% idle.
>>> Obviously, 512 calls is pushing the limits but we only really need
>>> to hit around 300 for our current project. Nonetheless, the
>>> additional CPU cycles consumed by another NIC are something to watch
>>> for. Luckily, Intel will be releasing their dual-core processors soon.
>>>
>>> - When you say ramdisk here I assume you mean using conventional
>>> ram, its
>>> - cheap yes but its volatile, do you have any plans for failure of the
>>> - system or ram? Or is the data integrity itself not as critical? The
>>> - reason that people like hard drives is because most of the time if
>>> the
>>> - system goes down for any reason the data is still intact.
>>> Please see my responses to Matt regarding our RAM disk configuration.
>>>
>>> - isnt vomit free? It was a voip sniffer that worked with some codecs
>>> - many years ago (I wanna say mid-late 90s but I may be thinking of
>>> - another back then). http://vomit.xtdnet.nl/ does G.711 only.
>>> -
>>> - The bigger prIoblem that I see is that sniffers dont always get
>>> all the
>>> - traffic that is on a network particularly when the network has more
>>> - traffic on it. While this generally isnt a concern and I would
>>> like to
>>> - think that even a poorly configured network could allow for 512
>>> calls,
>>> - it is a factor to implement this type of a solution.
>>> Hopefully, it won't come to this. With all of your help, I doubt it
>>> will.
>>>
>>> ============================================================
>>>
>>> Richard,
>>>
>>> - Another project snatched from the Toilet bowl ahead of the flush.
>>> With any luck, we'll be able to shine this turd.
>>>
>>> ============================================================
>>>
>>> I apologize for the length of this post, but after consideration it
>>> seemed to be the most efficient way to address each of you.
>>>
>>> Thank you all,
>>>
>>> Matthew Roth
>>> InterMedia Marketing Solutions
>>> Software Engineer and Systems Developer
>>> _______________________________________________
>>> --Bandwidth and Colocation sponsored by Easynews.com --
>>>
>>> Asterisk-Users mailing list
>>> Asterisk-Users at lists.digium.com
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>> To UNSUBSCRIBE or update options visit:
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> --Bandwidth and Colocation sponsored by Easynews.com --
>>
>> Asterisk-Users mailing list
>> Asterisk-Users at lists.digium.com
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>> To UNSUBSCRIBE or update options visit:
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>
> _______________________________________________
> --Bandwidth and Colocation sponsored by Easynews.com --
>
> Asterisk-Users mailing list
> Asterisk-Users at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>
More information about the asterisk-users
mailing list