[asterisk-dev] About asterisk development plans

[Steve Murphy]

On Wed, Mar 18, 2009 at 11:06 AM, Olle E. Johansson <oej at edvina.net> wrote:

>
> 18 mar 2009 kl. 17.57 skrev Gavin Henry:
>
> > 2009/3/18 John Lange <john at johnlange.ca>:
> >> On Wed, 2009-03-18 at 10:44 -0400, Mihai Balea wrote:
> >>> As far as I know, there is no test harness, so it's not surprising
> >>> that regressions pop up as often as they do.
> >>
> >> I think this is the most important point. There should be a test
> >> bench
> >> that automatically confirms every (supported) feature is working
> >> before
> >> it's released. If the process was automated then it shouldn't take
> >> that
> >> long to run the tests.
> >
> > Hi,
> >
> > I think this is key and just about every open source project has a
> > "make test".
> >
> > Take Perl and the CPAN. If your distro doesn't have a "make test",
> > many people
> > won't use your code, as they have no way of know if your libs work.
> >
> > I'm part of the OpenLDAP project and our Release & QA Engineer does
> > all this.
> >
> > Does Asterisk have one/somebody?
> >
> > Our Release & QA Engineer makes sure all the right patches are
> > applied to the
> > CVS branch and calls for testing. Our "make test" runs around 55
> > tests, but as usual
> > could do with many, many more. But this means we don't release if
> > there is a failure
> > and don't have any regressions.
> >
> > I understand * will be hard to develop a test suite for, but we should
> > at least try to have a
> > test for every feature listed here:
> >
> > http://www.asterisk.org/support/features
> >
> > We could do virtualisation different architures etc. dependant on the
> > feature we are testing or
> > even do smoke testing etc. I don't know, but it would be good to know
> > it works before it is
> > released. We may even need virtual devices etc.
> >
> > Another great example is the Samba project:
> >
> > http://build.samba.org/
> >
> > People or companies interested in seeing their kit and/or platform
> > supported add a node to the build
> > farm and send the results in.
> >
> > If the test suite is deployed like this, people desperate for testing
> > will add their servers.
> >
> If you search the mailing lists, you will see that this has been
> brought forward
> a large number of times. There has been efforts by mutliple developers
> to start
> with a test framework. The work that is in the code today is mostly
> murf's
> test suite for the AEL parse and a few others.
>
> Digium also runs a few test servers that run build tests, but not
> function tests
> as far as I know.


I'll throw in a few opinions on this issue-- Some of the software mentioned
with nice test suites are pretty easy to run tests against. In all the
cases, it's pure software involved, and you define nice input-output-compare
tests to insure no regressions arise.

But asterisk is quite challenging. To do it right, I picture
electro-mechanical
devices that will pick up a handset and dial physical analog phones,
supplying
standard sound sequences, and recording sound into files. Maybe
such a robotic phone is available for analog lines.  For sip phones, a
client
that can be driven 'robotically', under the control of some sort of script
language,
or via a command driven network interface, like the manager interface, so
you
can script up dozens of standard calling squences involving parking, xfers,
call files,
with all their permutations of eg. hookflash, call features, sip phone
buttons, etc.

You'd need sound file comparators that would judge sent-vs-received quality
loss,
echo level, dialplan scripts to run the tests and look for problems in real
time, etc. It's a
mind-boggling task! Any part of it is do-able, but wow! Testing a program
like Asterisk
with so many possible ins and outs will be a real challenge!

I've built dozens of large complex commercial products that I had to
maintain for years, with
highly critical customers in the thousands and testing was an absolute
requirement...

murf

A great project for any PBX.

> If you can step forward and help us build a foundation for a test
> suite, it will
> be extremely appreciated!
>
> /Olle
>
> _______________________________________________
> --Bandwidth and Colocation Provided by http://www.api-digital.com--
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-dev
>



-- 
Steve Murphy
ParseTree Corp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.digium.com/pipermail/asterisk-dev/attachments/20090318/13ab0c38/attachment.htm