[asterisk-dev] About asterisk development plans

[Peter Beckman]

Several people on this thread have mentioned being afraid of using the
latest "stable" release.  I too have this fear, and run an older version of
Asterisk and refuse to upgrade, not knowing what changes to the code might
have been made that will break my existing system.

I regularly have a call with another friend who runs Asterisk in
production, and he constantly gripes about how upgrading broke this and how
a seemingly harmless change to the code broke a whole slew of other things.

I don't really have a lot of choice to move to another platform though, and
honestly I'm not sure changing would solve the problem.  But it is clear
that regression testing, feature testing, and automated testing is
something that would greatly improve the consistency and REAL and perceived
stability of asterisk.

On Wed, 18 Mar 2009, Olle E. Johansson wrote:

> To show the complexity of what we're discussing:
> Let's take blind transfer.
>
> Now we're up to a massive amount of tests.

  In my experience, writing the code takes 1 hour, and writing good tests
  can take 2-3 hours.

  The nice thing is that once you've written the test, it's done.  You can
  use it and run it over, and over, and over again.  You can change your
  base code all day long, then run your 15,000 tests to see if you broke
  anything else.

  And now you have a fully reproducable way to backwards test features and
  functionality.

  On top of that, you can now ask bug submitters to write you a test that
  causes a feature or some functionality to fail.  This helps the Digium
  developers focus on fixing code that someone else can reproduce
  consistently, and provide the exact code to show it.

  Everyone keeps cheering for features for Googles SoC.  Here's what I want
  -- a well documented, well thought-out automated testing suite for
  asterisk.  Once you build it, plus a few hundred tests that test the lower
  levels, libraries, and basic functionality, then push more testing out to
  developers and users.  Show us "Here's how to test your dialplan" or
  "Here's how to test your AGI" with a few examples, and we'll write tests
  to pass or fail.

> One bullet point, many, many hours of work. A lot of possibility to
> cause a bug when you fix something else.

  Sure, but if you get the test working, now you know that the code it is
  testing is working, and the test is giving you an expected pass.  When
  someone comes back and says "Hey Olle, it's passing the Call Pickup test
  #32, but when it is done like this, Call Pickup fails.  Here's a test I've
  written to demonstrate the failure."

  Then Olle can look at the test, verify if the test is valid and SHOULD
  work, then fix the code so it DOES work.  Then Olle runs the other 15,000
  tests to make sure that the fix does not break something else Olle didn't
  expect.  And if all tests pass, commit!

  Start small.  Start with simply testing the libraries.  I started testing
  my core function libraries and classes first.  Then I expanded to the next
  level of libraries which combined core functionality.  Then I expanded to
  test advanced functionality.  I didn't do it all at once, but now that I
  have it, I can test every nook and cranny of code in my systems to make
  sure things are working as expected.

Beckman
---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                 http://www.angryox.com/
---------------------------------------------------------------------------