[asterisk-dev] Asterisk Testing

[Russell Bryant]

Greetings,

I think we can do a much better job at testing within the Asterisk 
project and I suspect you do, too.  The vast install base and wide 
variety of uses gets us very good feedback.  However, it is clear that 
we can do better within the development community at improving the 
quality of the code before it makes it out in any type of release (beta, 
RC, or full release).

I would like to propose that we adopt an automated testing approach as a 
part of our development process.  There are 4 pieces to this:

1) Testing with our eyes

2) Bottom-Up testing using unit tests within Asterisk

3) Top-Down testing using an external test suite

4) Tests running constantly using a continuous integration framework


While this isn't automated testing, it is too valuable not to mention. 
One of the most effective ways of finding bugs in code is by peer code 
review.  We see this more and more as we continue to increase our usage 
of ReviewBoard for peer review before changes are merged.  It's rare 
that a change goes in as the first proposed patch that goes on 
ReviewBoard.  While it has never been a hard rule, it is strongly 
encouraged that every committer to seek a sign-off from at least one 
other committer via ReviewBoard for _every_ non-trivial change.

    https://reviewboard.asterisk.org/


For unit testing, we have a new API in Asterisk trunk for writing unit 
tests within the code.  We're up to 20 unit tests.  Writing these tests 
has already found multiple bugs and running them continuously has caught 
one regression before it made it into a release.

    See include/asterisk/test.h


The beginnings of an external test suite have been created.  There is 
still design work to do, but hopefully it will begin to be populated 
with tests soon.  In general, though, this will become a suite of tests 
that can use whatever external tools are necessary to verify Asterisk 
functionality.  There is a README.txt that documents what is there so far.

    $ svn co http://svn.digium.com/svn/testsuite/asterisk/trunk


On the continuous integration front, we have been using buildbot for a 
long time for doing compilation on various systems.  It's time that we 
extend our continuous integration approach to not only compile Asterisk, 
but also run regression tests.  After evaluating some different options, 
Bamboo has been installed and configured for our use.  So far, it is 
compiling all release branches of Asterisk.  For Asterisk trunk, it is 
running all of the unit tests after every change to the code.  The 
results of the tests are integrated into the application.

    http://bamboo.asterisk.org/

The only build slave right now is Linux.  This will be extended to other 
operating systems.  If you would be interested in providing a build 
slave and assisting with its maintenance, please talk to me.  We can 
configure up to 25 build slaves ("remote agents" in Bamboo terminology).

If you have commit access, you are strongly encouraged to sign up for an 
account on bamboo.asterisk.org and associate your account with your svn 
username.  That way, bamboo can notify you by email and/or jabber if 
there any problems associated with changes that you make.


As you work on changes to the Asterisk code, you should consider this 
approach to testing along side your development. If there is a unit test 
you could write to help prove your code works or prevent a bug from 
showing up again, then please write one!  If there is an external test 
you could write to help verify some piece of functionality you are using 
continues to work precisely as expected, then let's figure out how to 
get that into the test suite.  This kind of testing not only prevents 
regressions but it also guarantees that expected behavior will remain 
intact across releases.

I am excited about what this effort can do for the quality of Asterisk 
in the long term, and I hope you are, too.  I look forward to your 
feedback on the approach as well as participation in the development of 
our testing infrastructure.

Thanks,

-- 
Russell Bryant
Digium, Inc. | Engineering Manager, Open Source Software
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
www.digium.com -=- www.asterisk.org -=- blogs.asterisk.org