[asterisk-dev] About asterisk development plans

[Stephen Davies]

2009/3/18 John Todd <jtodd at digium.com>

3) A regression test does seem like an idea whose time has come.  We
> have had soul-searching decisions like this before - the conversion to
> SVN, the installation of "configure" files (wow, remember the tearing
> of hair over that one?), the shift to features_, and others.  But
> perhaps now is the time for a "make test" concept.
>   The difficulty with this is not to be understated.  Both Olle and
> Murf weighed in on this thread with the overwhelming complexity of
> testing something like Asterisk.  This is not something that will be
> done overnight, and on some parts of the software I expect it will
> NEVER be done.  However, clearly there is a need for some sort of more
> formal tests.  Each app/function/channel/res will probably have to
> have its own test plan.
>   It seems the easiest way to create a regression test of Asterisk is
> to create a test plan with another Asterisk system.  To mimic user
> behavior is not, in my opinion, an impossible task.  While I believe
> analog/digital interfaces are important, I would suspect that most of
> the behaviors that would fall into a regression test would be testable
> entirely via IP channels, and in fact it may be possible for two
> instances of Asterisk to operate on the same machine, one testing the
> other.



Hi John,

A way back I knowlingly sacrificed reliabillity of a production server in
order to try to help get 1.4 ready for prime time back when it was really
not being taken up.  I think my effort did help.   Every time that box fell
over I took time to find out why and where possible I sent in a patch.  But
I'm my own boss and my service is redundant so the impact wasn't so bad.
 Others risk losing their jobs if they don't deliver reliable phone service!

Regression testing a test-first programming and all that is definitely
worthwhile.  But very hard for Asterisk I think.

But something else that would help is just a big fat test rig where
"workouts" could be run.  For example, Vicidial working hard has an uncanny
ability to find problems.  Its also fairly easy to simulate a large inbound
queue system.  Mix in a nice complicated set of Local channels, Mixmonitors,
different channel types etc and you can end up with a "torture tunnel" test.
 Production load without anybody risking losing their jobs!

New release need to survive a weekend in the torture tunnel with no crash or
deadlock to be shipped?

(Most bugs I hit in Asterisk these days are deadlocks.  Actually nearly all.
 So that's my first interest!)

Regards,
Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.digium.com/pipermail/asterisk-dev/attachments/20090319/ca4db4e5/attachment-0001.htm