<br><div class="gmail_quote">On 24 July 2012 15:18, Hans Witvliet <span dir="ltr"><<a href="mailto:asterisk@a-domani.nl" target="_blank">asterisk@a-domani.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Sun, 2012-06-03 at 23:23 -0400, Tom Browning wrote:<br>
> Any tips on solving the following performance conundrum:<br>
><br>
> Asterisk 1.8.12.2 running on HP DL360 G5 and G7s<br>
><br>
> tcpdump running to capture UDP 5060/SIP signaling to .pcap files<br>
><br>
> All calls are ultimately B2BUA client -> asterisk -> PSTN<br>
><br>
> Media stays on Asterisk at all times<br>
><br>
> AGI script has exit handler that connects and updates an external<br>
> database upon BYE from either side.<br>
><br>
> I know that if exit handler hangs around too long, Bad Things (tm) will happen.<br>
><br>
> Oddly, under load (60-100 B2BUA calls), the G7s start complaining:<br>
><br>
> Autodestruct on dialog '<CALLID>' with owner in place (Method: BYE)<br>
><br>
> I/O wait is actually higher on the G5s, the G7s have fancy disk cache<br>
> cards and never get above 1% i/o wait<br>
><br>
> turn off the tcpdump process on the G7s and Autodestruct warnings go<br>
> away. The G7s should have<br>
> much more capacity than the G5s but we never, ever get Autodestruct<br>
> ... Method: BYE on the G5s.<br>
><br>
> OS is identical CentOS in both cases. Every other environmental<br>
> config is the same (network, subnet, DNS etc).<br>
><br>
> Architecture/bus/network card difference? tcpdump starving some other<br>
> resource to cause stuff to slow down?<br>
<br>
</div>Hi Tom,<br>
<br>
Regarding G5's and G7's....<br>
We have them both (all though not for asterisk) at work. and found a<br>
little snag.<br>
The purchase department interferred with our initial ordering of the<br>
hardware. As a concequence, they did not order the battery for the raid<br>
controller. (seems you have to order that explicitly)<br>
After it arrived, we were able to install linux on it and use it, but<br>
disk-io was way slower that the original G5.<br>
<br>
You might want to check/compare disk-io & throughput on your G5 vs G7.<br>
Just a thought....<br>
<span class="HOEnZb"><font color="#888888"><br></font></span></blockquote><div><br></div><div>For reference... In my opinion HP servers should never be bought without the battery or alternative, they shouldn't even be offered for sale without it...</div>
<div><br></div><div>Without a battery or non-volatile, flash based, cache the controllers in HP servers disable write caching allocating 100% of the built in cache to be a read cache. The cache built into the actual disks is also disabled in all cases, even with a battery. Both of these things are done to protect data. In the event that a write was in any of the caches during a loss of power, that data would be lost, a problem made worse by the fact that when data is written to cache the OS is told it's been safely written. HP refuse to let you put your data at risk, refusing to activate write caching without a charged battery attached or NV cache.</div>
<div><br></div><div>I personally would like to have the option to override things like this at my own risk, but, HP don't give you that option.</div><div><br></div><div>With all write caching disabled writes become very slow. The disks can't do their own reordering of writes to maximise performance. The controller can't do it's own write optimisations. If you use RAID 5/6, you'll get even more dire performance thanks to a combination of an underpowered controller in most cases, plus the fact that writes will often need to read data to complete the parity write, something that the controller can do in a smarter, not necessarily serial, way if it's got a cache of things that need to be written.</div>
<div><br></div><div>So, in summary, HP servers with no battery or nv cache typically have horrendous disk performance that can't be fixed without adding a battery or nv cache.</div><div><br></div><div>In this case, the fact that the OP mentioned that the io-wait is a lot lower on the G7s would suggest that they potentially do have a battery or nv cache which has triggered the controller to enable the write cache leading to a much improved write performance over what they would have without, and potentially, the G5s don't have this.</div>
<div><br></div><div>I'd be interested to see if other heavy writes caused the same problem and also would be interested in more detailed server specs...</div><div><br></div><div>d</div></div>