[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital
Recording
Matt Roth
mroth at imminc.com
Wed Sep 21 09:45:35 MST 2005
All,
This message has generated a lot of responses, so I'm going to address
each of them here in an attempt to consolidate the thread.
============================================================
Matt,
- I'm very interested in the specifics of your setup.
- How much space is on the RAM disk?
Currently it is 10 GB. We are upgrading it to 16 GB.
- What kind of RAM drive is it?
The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs.
The details for each 1 GB DIMM can be seen here:
http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm
The upgrade will involve adding 2 GB DIMMs to the system, but I don't
have the details on these yet.
The RAM disk is setup by adding the following kernel command line option
to grub.conf:
ramdisk_size=10485760
We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
By default the RAM disk's block size is 1024 bytes, so we are formatting
it as an ext2 file system with a block size of 1024 bytes using the
following command:
mke2fs -b 1024 -m 0 /dev/ram0
The block size can easily be changed from the kernel's view (using the
kernel command line option ramdisk_blocksize=XXXX) or from mke2fs's view
(using the -b XXXX argument), so please let me know if I can make an
easy optimization here.
Finally, the RAM disk is mounted using the command:
mount /dev/ram0 /digrec
A good RAMDISK howto exists at:
http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
- What format are you recording to?
- What codec are the SIP calls being placed over?
We are recording to the PCM format and using the G711 uLaw codec. High
voice quality is essential to our application (we are a call center) so
we partnered with MCI to configure our network for the required
bandwidth and chose the highest quality, zero compression codec. We
noload all other codecs in order to avoid transcoding on the switch, so
we must record to PCM. Later (on a separate server) the recordings are
mixed to GSM which provides a 5 to 1 compression ratio with very little
artifacts.
- We've run into the "Avoided deadlock" recording issues several times
when trying to do
- more than 50 concurrent recordings. Changing the ast_channel_lock loop
from 10 to 20 has
- helped somewhat reduce the warnings and reduce audio gaps on the
recordings, but what is
- really needed for more robust recording is a configurable recording
buffer that wouldn't
- freak out if a 10ms delay occurs.
Are you saying that these messages indicate a gap in a digital
recording? If so, what is the duration of the gap? If it's comparable
to a CD skip, I think we can deal with it until a buffer or another
solution is implemented.
- Good luck and please keep us updated on your progress
Thank you. I'll be keeping the list informed of our progress.
============================================================
Zoa,
- I suppose you are the person from the digium forum
That was actually my boss's boss. We thank you all the way up and down
the line for your suggestion.
- The reason i recommended you to use a ramdisk is because i think the
- problem with recording to disk is saving 20ms of stream 1, then 20 ms of
- stream 2, then 20ms of stream 3 etc etc.... meaning you write everytime
- very small things. (with a lot of seeking).
Agreed. This is why we hope that decoupling the copy (memory to disk)
from Asterisk itself and, more importantly, Asterisk's real-time
handling of the call being recorded will be sufficient.
For the record, when recording 512 simultaneous calls to the local disk
we saw a peek of about 13,000 blocks written per second.
- Our best test results were with:
-
- - buffering the recordings to a ramdisk, then
We're doing that, as per your suggestion.
- - on low load (at night) copy the files over the network (easy to shape
- the pipe, so that you dont overload anything), This way, the memory
- buffer will take care of the 'fragmentation' and not your harddisk.
If you'll note the format of the recordings and that we'll be recording
up to 200,000 minutes of calls a day, with a little quick math you'll
realize that it would take 80 to 100 GBs of memory for us to buffer a
full day's recordings. Combined with the fact that a server failure
late in the day would cause us to lose them all, this is not a desirable
solution.
Instead, we plan to write an application to call from the MONITOR_EXEC
hook that will be executed at the end of each call. This application
will be niced down to the lowest priority, and simply copy the leg files
from memory to disk. Under normal conditions (ie. our NFS server is up)
this will actually be a copy to a remote disk using an asynchronous NFS
transfer. All actual disk I/O will occur on our digital recording
server and any handling of the digital recordings will occur only after
the call they are bound to is completed.
Do you have any suggestions regarding this scenario?
- - on the remove server, do all the mixing / indexing etc. (i really
- don't mixing or converting between audio formats on the same server as
- asterisk).
That exactly describes the functionality and purpose of our digital
recording server. We do no transcoding on the Asterisk server and
perform no DSP on it. It handles only SIP traffic, as our Ts are
terminated by a Cisco AS5400HPX Universal Gateway.
- If you want to go even freakier, run asterisk (or you complete distro)
- from a ramdisk.
-
- Oh, another thing, for the people trying this the performance of hdparm
- is not linear with the quality of your calls, tweaking your disks to be
- faster will not help for asterisk when you do a copy. (in general).
Thanks. I will keep this information in mind.
- I thought over your suggestion to use a sniffer to do the recordings,
- you might pull it off, but will have to write your own to do so. (or go
- to the expensive version of commercial sniffer applications).
Trust me, using a sniffer is not our ideal solution. But if push comes
to shove...
- Also when you do things over the network, disable your onboard network
- card, and go for some more expensive network card.
- In our tests with small packets, we could increase the throughput with a
- factor 2. (related to cpu load).
-
-...the dual nic gigabit intel card was a lot better than all the other
- combinations i tried.
Done. We were going to use a Level 5 Networks EF1-21022T NIC, but it
didn't like performing DMA with anything greater than 4 GB of system
RAM. Level 5 is working on resolving this issue with their drivers, but
until then we are using an Intel Pro 1000 MT NIC installed in a 64-bit,
133 MHz PCI-X expansion slot. This Intel NIC is actually the onboard
NIC on our digital recording server (a Dell PowerEdge 1850).
============================================================
Trixter,
- filesystems are also a consideration with larger scale projects.
- Different filesystems add different amounts of overheads on different
- types of operations. Some are faster at moving small files around
- others faster with large files. This adds to the disk latency.
- Removing the disk latency itself is a good thing, since that is
- typically slower, but to crank out that last little bit of performance
- some research into the different filesystems under the specific kernel
- that you are using could also be a consideration. The most obvious
- (and easiest to update a running system) is to remove things like atime,
- whih with most linux distros is on by default. This causes a write
- operation for the read of a file to update the last time accessed. A
- couple little things can add up to a few percent improvement and
- generally make the cost go down.
We may need every percent we can get, so thanks for the suggestions. As
mentioned above, we're running Fedora Core 3 with the latest 2.6 SMP
kernel. Our physical disks are using the ext3 file system, while our
RAM disk is using ext2.
- Or have a seperate network set up (dual nic card for example) where the
- 2nd network is used just for NFS traffic. Although NFS generally is
- ugly network wise, it is standard and makes things easier. Just gotta
- watch the IO on the system given that the network card itself will cause
- cpu cycles to be used, but lets face it cpu is cheap now. Different
- drivers also work differently, and then with the 2.6 series kernels you
- can use device polling instead of interupts which can help a little.
This seems like something we may pursue in the short term. We are using
NFS (which will be configuring for asynchronous transfers), but not with
a dedicated NIC. Our system has four 3.16 GHz Intel Xeon processors.
They are hyperthreaded, so the OS sees them as eight separate
processors. During our 256 call testing they hit 70% idle, but during
our 512 call testing they dropped to 10% idle. Obviously, 512 calls is
pushing the limits but we only really need to hit around 300 for our
current project. Nonetheless, the additional CPU cycles consumed by
another NIC are something to watch for. Luckily, Intel will be
releasing their dual-core processors soon.
- When you say ramdisk here I assume you mean using conventional ram, its
- cheap yes but its volatile, do you have any plans for failure of the
- system or ram? Or is the data integrity itself not as critical? The
- reason that people like hard drives is because most of the time if the
- system goes down for any reason the data is still intact.
Please see my responses to Matt regarding our RAM disk configuration.
- isnt vomit free? It was a voip sniffer that worked with some codecs
- many years ago (I wanna say mid-late 90s but I may be thinking of
- another back then). http://vomit.xtdnet.nl/ does G.711 only.
-
- The bigger prIoblem that I see is that sniffers dont always get all the
- traffic that is on a network particularly when the network has more
- traffic on it. While this generally isnt a concern and I would like to
- think that even a poorly configured network could allow for 512 calls,
- it is a factor to implement this type of a solution.
Hopefully, it won't come to this. With all of your help, I doubt it will.
============================================================
Richard,
- Another project snatched from the Toilet bowl ahead of the flush.
With any luck, we'll be able to shine this turd.
============================================================
I apologize for the length of this post, but after consideration it
seemed to be the most efficient way to address each of you.
Thank you all,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
More information about the asterisk-users
mailing list