<br><br><div class="gmail_quote">2009/3/18 John Todd <span dir="ltr"><<a href="mailto:jtodd@digium.com">jtodd@digium.com</a>></span></div><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">3) A regression test does seem like an idea whose time has come. We</div>
have had soul-searching decisions like this before - the conversion to<br>
SVN, the installation of "configure" files (wow, remember the tearing<br>
of hair over that one?), the shift to features_, and others. But<br>
perhaps now is the time for a "make test" concept.<br>
The difficulty with this is not to be understated. Both Olle and<br>
Murf weighed in on this thread with the overwhelming complexity of<br>
testing something like Asterisk. This is not something that will be<br>
done overnight, and on some parts of the software I expect it will<br>
NEVER be done. However, clearly there is a need for some sort of more<br>
formal tests. Each app/function/channel/res will probably have to<br>
have its own test plan.<br>
It seems the easiest way to create a regression test of Asterisk is<br>
to create a test plan with another Asterisk system. To mimic user<br>
behavior is not, in my opinion, an impossible task. While I believe<br>
analog/digital interfaces are important, I would suspect that most of<br>
the behaviors that would fall into a regression test would be testable<br>
entirely via IP channels, and in fact it may be possible for two<br>
instances of Asterisk to operate on the same machine, one testing the<br>
other.</blockquote><div><br></div><div><br></div><div>Hi John,</div><div><br></div><div>A way back I knowlingly sacrificed reliabillity of a production server in order to try to help get 1.4 ready for prime time back when it was really not being taken up. I think my effort did help. Every time that box fell over I took time to find out why and where possible I sent in a patch. But I'm my own boss and my service is redundant so the impact wasn't so bad. Others risk losing their jobs if they don't deliver reliable phone service!</div>
<div><br></div><div>Regression testing a test-first programming and all that is definitely worthwhile. But very hard for Asterisk I think.</div><div><br></div><div>But something else that would help is just a big fat test rig where "workouts" could be run. For example, Vicidial working hard has an uncanny ability to find problems. Its also fairly easy to simulate a large inbound queue system. Mix in a nice complicated set of Local channels, Mixmonitors, different channel types etc and you can end up with a "torture tunnel" test. Production load without anybody risking losing their jobs!</div>
<div><br></div><div>New release need to survive a weekend in the torture tunnel with no crash or deadlock to be shipped?</div><div><br></div><div>(Most bugs I hit in Asterisk these days are deadlocks. Actually nearly all. So that's my first interest!)</div>
<div><br></div><div>Regards,</div><div>Steve</div><div><br></div></div>