[asterisk-dev] About asterisk development plans

[John Todd]

On Mar 17, 2009, at 11:58 PM, Olle E. Johansson wrote:

> 17 mar 2009 kl. 20.32 skrev Jon Bonilla (Manwe):
> [snip]
>> Once again I should apoligize if someone misunderstands my words. I
>> just am
>> willing to know what the goal of this project is. If it's goal is to
>> be a
>> ready-for-production-environment PBX, I think it should focus on
>> stability much
>> more.
>
> This is a very important discussion that I'm glad that you bring up.
> It is indeed
> very boring to fix bugs and test, test, test. The Digium team surely
> focuses a
> lot on bug fixing and testing, much more so today than a few years
> ago. I think
> we all agree on the goal. The big question is how to get there. Will
> more resources
> help? Is there a methodology problem in the testing done? Is the world
> too
> complex for Asterisk to live in? Help us.
>
> We do need the community to step in and work on this alongside with
> Digium.
> We need the community to take a more active role in managing Asterisk
> and not
> leave it to Digium. Digium is doing more than their share. Asterisk
> needs the
> community taking responsibility.

[snip]

I've been reading the replies to this thread, and I'll put in some  
observations and possible solutions.  The community around Asterisk  
has grown tremendously in the last several years, both from a user  
perspective and a development perspective.   A few thoughts that I  
formed while reading the thread:

1) Complexity has increased.  I believe this is the inherent nature of  
telephony software.  Asterisk started out with a few fairly crude  
features, but now people are asking (demanding!) that it perform more  
difficult tasks.  This implies creation of fairly significant new  
complexity structures to support those demands, and increases the  
difficulty of implementation.  This is not a bad thing - it is very  
_good_ that Asterisk is becoming more complex, because it is able to  
address more of the concerns that larger installations have, and also  
now fills many more "niche" solutions than previously.  But this comes  
at a cost.
   There is always a conflict between stability and features, and  
between simplicity and flexibility.  There is a relentless push to  
make Asterisk more feature-rich, and flexible for all possible  
circumstances in which it is installed.  This decreases simplicity,  
and makes a higher load of bugfixes.


2) Companies are seemingly less willing to contribute back.  There are  
dozens, if not hundreds of companies that use Asterisk as a core for  
their products.  Only a tiny handful are active participants in  
keeping Asterisk advancing, or fixing bugs.  Quite a few will send in  
their developers (under anonymous cover) to report various bugs, and  
then they are adamant about the speed of repairs being insufficient.   
The willingness of these corporate users to test anything other than  
their exact problem is also practically zero.  This is not a situation  
unique to Asterisk - most other software that becomes heavily-divested  
has the same situation until the really large companies involved in  
open source contribution start putting in resources.  I'd be  
interested in hearing how the current problems of contribution  
reluctance can be solved, as none of the few ideas I've considered  
have gained traction thus far.


3) A regression test does seem like an idea whose time has come.  We  
have had soul-searching decisions like this before - the conversion to  
SVN, the installation of "configure" files (wow, remember the tearing  
of hair over that one?), the shift to features_, and others.  But  
perhaps now is the time for a "make test" concept.
   The difficulty with this is not to be understated.  Both Olle and  
Murf weighed in on this thread with the overwhelming complexity of  
testing something like Asterisk.  This is not something that will be  
done overnight, and on some parts of the software I expect it will  
NEVER be done.  However, clearly there is a need for some sort of more  
formal tests.  Each app/function/channel/res will probably have to  
have its own test plan.
   It seems the easiest way to create a regression test of Asterisk is  
to create a test plan with another Asterisk system.  To mimic user  
behavior is not, in my opinion, an impossible task.  While I believe  
analog/digital interfaces are important, I would suspect that most of  
the behaviors that would fall into a regression test would be testable  
entirely via IP channels, and in fact it may be possible for two  
instances of Asterisk to operate on the same machine, one testing the  
other.
   These tests are not simple, and require some pretty serious  
scripting capabilities.  Many (most?) could probably even be done  
directly in the dialplan, with some external database triggering tests  
and tabulating expected results via ODBC.  AGI and AMI would be  
exceptions to this, but again would not be impossible.  As mentioned  
by Peter, it may be possible to automate the tests entirely via AGI  
and AMI, also.
   The description of 1:3 ratio of coding to test-jig creation is  
fairly accurate from a time perspective. It's 1:3 the first time the  
code is written, and then any changes after that point might be  
somthing like a 1:.05 ratio since rarely are changes to the test jig  
required.  Since Asterisk is mostly written, we're looking at a fairly  
ugly time investment to get a test jig created.  Starting small is  
certainly going to be the first step.
   I'm pleased to see this produce another thread, and any replies to  
this section of this message should be added to that threadline.


4) Some mention was made expressing concern over the priorities of  
features vs. stability.   I would suggest looking at the bug tracker  
to anyone interested in examining where Digium's priories are.  Digium  
staff spend a staggering amount of time and energy on fixing bugs and  
getting Asterisk in all forms as stable and usable as possible, and  
this comes at significant cost to Digium.  Nobody likes to do boring  
stuff like test patches and fix minor nits, but that is in fact what  
is the majority of the time spent at Digium working on Asterisk code.   
Olle and others send out requests for bug testers all the time, but  
there remains a serious shortage of people willing to test bug fixes  
or even new features.
    There are certainly new features that keep coming out - to exclude  
new features entirely and concentrate 100% on bugfixes would lead to a  
stagnant and ultimately failed project.  Digium staff and community  
members work jointly on both bugfixes and features, using priority  
systems that they alone can determine.  The vast bulk of the work  
right now is on creating an error-free system that provides the  
functionality that has been demanded by the user base.


JT

---
John Todd                       email:jtodd at digium.com
Digium, Inc. | Asterisk Open Source Community Director
445 Jan Davis Drive NW -  Huntsville AL 35806  -   USA
direct: +1-256-428-6083         http://www.digium.com/