[asterisk-users] Experience with Vicidial

[Matt Florell]

On 7/17/08, Alex Balashov <abalashov at evaristesys.com> wrote:
> Matt Florell wrote:
>
>   > No apologies necessary, I think a lot of what you said is mostly true.
>
>
> Well, thank you.  I really appreciate that you're willing to entertain
>  what I am saying without construing it as some sort of attack;  it is
>  not in the least bit intended that way.
>
>
>  > The PHP and Perl code is not the prettiest around, and a lot of it is
>  > not commented or formatted as well as it should be. However, I would
>  > disagree that there is absolutely zero regard for maintanability or
>  > readability. As with many other GPL projects out there VICIDIAL is
>  > free to use and modify, and there are many people outside of our
>  > company who have worked with the code to provide patches and added
>  > functionality.
>
>
> Indeed, and I acknowledge that this is a challenge with an open-source
>  project - a problem that is probably best solved by means that are not
>  necessarily received well politically, and is probably seen as
>  orthogonal to the spirit and philosophy of open-source.  It requires
>  some degree of centralisation of the patch management process and high
>  standards for acceptance, testing, and coding conventions.  This leads
>  to a process that gives the perception of being more "closed," a la the
>  Linux kernel, and, as some would have it, perhaps the Asterisk source
>  tree.  (I can't really say, as I've never attempted to contribute any of
>  my modifications of the Asterisk source to the project.  *hangs head in
>  shame*)

Bad programmer, no beer for you :)

But seriously, Asterisk is a better example of doing things right more
recently, a couple of years ago all sorts of stuff went into the
"stable" releases of Asterisk without  enough testing resulting in
some pretty big bugs (like in asterisk 1.2.10-12) And even further
back the code conventions of Asterisk 6 years ago were in many places
about as good as VICIDIAL is today. I think it's the evolution of a
project from being in its early years as opposed to its more mature
years like Asterisk has started to enter. If I were to start writing
VICIDIAL today I would have done many things a lot differently from
how they are, but then again if I started from scratch today it
wouldn't be a functioning product for many months.

Right now, we do basic testing of everything before it is committed to
SVN. We are planning on starting an official Q&A process when we are
able to that would test a full range of functions before every commit.


>  But, maintainability and extensibility are probably the biggest
>  challenges to the adoption of an open-source project by a commercial
>  organisation, although those challenges are even more formidable for
>  proprietary, closed-source products.  The stark, naked economic
>  realities of adopting something are still there.

There are a few large companies that have adopted VICIDIAL and
internalized the maintenance of the code for their systems, so we know
that the state of the code has not been a road-block for all
companies.


>  The integration paths, APIs, transparency, modularity and extensibility
>  are the most important central concerns.  For instance, many likely use
>  cases of ViciDIAL entail at least some degree of integration with
>  existing business systems, rules, and logic;  after all, the data that
>  goes into the hopper must come from somewhere.  :-)

Most companies will do some degree of basic lead loading and lead
exporting integration on the back-end along with integration with
web-based CRM systems on the agent front-end. And this has proven to
usually be not too difficult for many companies to accomplish.

>  ... Which leads me to my point:  contrary to what is often zealously
>  claimed by purveyors and advocates of open-source solutions, the simple
>  fact that the code is open **does not, ipso facto, offer the necessary
>  level of integration and extensibility.**   Version changes, feature
>  changes, bug fixes, and other revisions cannot be readily or easily
>  applied if an organisation chooses to essentially "fork" an internal
>  revision of stock code, so invariably the preoccupation of good
>  engineering project management becomes whether the custom code can be
>  kept "outboard" in modules entirely separate from the main code tree, so
>  that the latter can remain more or less pristine across upgrades,
>  updates, add-ons, etc.

This is a big reason that we use SVN and we started maintaining a
"stable" branch that we apply only minor changes and bug fixes to and
a "development" branch that all new development takes place on.

>  This is where the importance of data import/export, APIs, and other
>  integration paths comes in.  It's the same reason why Asterisk worked
>  out so well for you in creating ViciDIAL;  you can do pretty much
>  everything in AGI, instead of having to hack the source or even do a
>  whole lot in the configs.
>
>  And, of course, in closed-source situations, these integration paths
>  become the lifeline - the only possible path for integration and
>  customisation.  So, these access points to the "plumbing" of the
>  program, where tweaks can be made to the nature and content of its flow,
>  are extremely vital.
>
>  ViciDIAL doesn't really have these, as far as I can tell.

We have actually do now have a agent-side API that allows outside
processes to have very limited control over agent interface functions.
As it's features grow we will also work on more back-end API
functionality not directly related to the agent.

>
>  > The 50,000+ lines of code were mostly written by me over the course of
>  > 5 years, and as we are now having more people working on the code and
>  > we are going through the scripts we are working to make the code
>  > easier to understand. It is important to mention that VICIDIAL is
>  > quite complex and offers a lot of features that add to the complexity
>  > of the code. Many of these features were not even conceived when the
>  > project was started so they were added in in the most efficient manner
>  > that was available. At this point there are over 1000 database fields
>  > across 60+ tables that control how VICIDIAL works.
>
>
> Yeah, I understand.  It's just how the project evolved;  I'm sure when
>  it was started it was intended to perform a relatively narrow subset of
>  what it does now, and things just got added on.  Nothing wrong with that
>  intrinsically;  it's how many open source projects got their beginnings,
>  and hardly to their detriment.
>
>  Still, the ViciDIAL code is an example of something that appears to have
>  been written, but not really designed, engineered, or orchestrated.  As
>  the complexity increases in the manner you describe above, it is
>  critically important to be able to decompose new elements out of the
>  core and into modules, libraries, packages, etc.  And good developer
>  documentation for the interfaces to these things must be crafted.  The
>  thinking must become a lot less "linear" in order for the
>  maintainability and economic feasibility of the code to scale
>  correspondingly to the features on the functional veneer of the application.

Yes, I agree, modularization is a future goal of ours that
unfortunately we do not have the time to work on at the moment, but it
is something that we have talked about a great deal.


>  As a somewhat related note, a lot of the constructs used in the code are
>  simply naive, from a programmatic perspective.  At this level of
>  complexity, the code should feel like it belongs to an application,
>  instead of a very, very complicated script.

I must say my programming skills have grown a great deal since
starting this project, and it shows if you compare the early code to
the more recent code. Many newer major features are now starting with
a planning phase before any programming is done, this has helped the
development and documentation of new features greatly, and as we have
the resources we will go back and reengineer things in a more
cohearant manner.


>  But worst of all is the organisation and readability.  I've worked to
>  evaluate the adoption of ViciDIAL with three different organisations
>  now, suggesting it with professional impartiality - apart from the
>  advocacy inherent in simply recommending it.  Each time, I would say a
>  genuine, sincere, diligent effort was made by the developers to get to
>  understand the code and explore how to go about customising it, and in
>  each case the code was ultimately judged too spaghetti to be workable.
>  The general complaints were - (1) impossible to read (2) strangely and
>  inconsistently formatted (3) lack of modularisation, abstraction,
>  centralisation, and other vital architectural considerations.

It really depends on the degree to which you need to alter the code as
to how difficult it is. Simple integrations with customer data are
fairly easy to do through data connecitons or scripts running and
converting data between systems. On the other hand, if you want to
significantly alter the schema or layout of the agent interface, then
you are in for a lot of work.


>  > As for scalability, VICIDIAL scales to hundreds of seats across
>  > multiple Asterisk servers. It can do this because of it's reliance on
>  > the MySQL database that acts as the core of a VICIDIAL system. We
>  > chose to use MySQL instead of a dedicated communications protocol so
>  > that the data could be accessed and used by almost any programming or
>  > user interface, and still remain extremely fault-tolerant and
>  > resistant to issues on any individual system.
>
>
> This is advantageous and portable, but a database really should not be
>  used for a high-speed, high-volume IPC mechanism inherently.  That's not
>  what databases are for.

Having a centralized system with universal data access has really
helped us in scalability and integration with other systems. Initially
I did not start out with MySQL handling everything, but as I ran tests
and worked more with it, it proved to be extremely reliable and very
fast, so I built everything around it and it has proven that it can
handle the job. It might not be the best or most efficient tool for
the job, and we may look into other methods of handling IPC in the
future, but for now it has proven to be one of the most stable
elements of a VICIDIAL system, with some databases running for over a
year without restart while still handling 5000+ queries per second.


>  Secondly, I think that a lot of the need for such a mechanism arises
>  from the fact that ViciDIAL is driven by so many concurrent processes
>  and cron jobs;  it ends up being a surrogate for locking and IPC.
>
>  Of course, this is just my opinion, but I think the dialer component of
>  ViciDIAL (i.e. the AGI part) could benefit from a much more monolithic
>  approach involving multiple threads and a great deal more synchronous
>  I/O and event multiplexing.   This would eliminate a lot of the need to
>  pump so much data in and out of the database for the purpose of
>  maintaining synchronisation in the first place.

When VICIDIAL started 5 years ago there were some serious reliability
issues with Asterisk under high loads, which is why some of the things
are done the way they are in VICIDIAL. I had to spend a lot of time
and effort finding ways to validate the data coming from Asterisk and
still be tolerant of Asterisk system freezes on multi-server systems.
Most of those issues have gone away, but they still do happen and the
fault tolerance built in to VICIDIAL has made it a better system in
the end.


>  Also, you don't necessarily have to develop a custom IPC protocol for
>  communication amongst components;  there are plenty of workable
>  distributed service architectures out there.

Is there a specific IPC protocol that you would recommend I take a look at?


>
>  > As for efficiency, yes there are a lot of small inefficiencies in the
>  > code, but most of the major bottlenecks were removed after changing
>  > many AGI scripts to FastAGI and rewriting several other scripts, which
>  > resulted in a 70-80% reduction in non-Asterisk load on a VICIDIAL
>  > system.
>
>
> Ah, well, glad to hear it!

Yes, we were pretty amazed(and happy) by the difference it made, and
we are in the process of doing more consolidation in development now.


>  It's been a while now, but I saw a main issue to be a lot of redundant
>  and unnecessary database calls in dozens of different places.
>
>  In a lot of cases, what ViciDIAL's AGI scripts appear to do is:
>
>  1. Get data out of database;
>
>  2. Somehow programmatically transform the data (i.e. do something with
>  it in a Perl %hash, or sort it in an array);
>
>  3. Feed the data out of the programmatic construct back into the database.
>
>  4. Do more things with it there.
>
>  5. Get it back out of the database and do some more stuff with it in the
>  program.
>
>  6. Rinse, repeat.
>
>  Sometimes this is unavoidable, and also, MySQL's abysmal lack of decent
>  features prior to 5.x might have been another constraint.  For instance,
>  without supporting sub-queries, one doesn't have a lot of choice in many
>  cases except to get the data out, and use parts of it as identifiers for
>  other queries.
>
>  All the same, a lot of this stuff could be done with stored procedures,
>  more joins, and other good, judicious use of in-database facilities.
>  There is also a dire need for centralisation of a lot of the database
>  calls.  On more than one occasion while monkeying around with ViciDIAL I
>  discovered that to make some minor adjustments to the schema and data
>  treatment, I had to make changes in dozens of places across 15-20 files.

After the next full release we will do away with MySQL 4.0 and 4.1
support and require 5.0+ so we can do a lot of what you suggest. We
have hundreds of systems out there with 4.X MySQL and we have always
tried to stay backwards compatible, but we have started to put notes
into the documentation and in the forums that we will be dropping
support for these older versions so that we can add new functions and
increase efficiency in future versions.

As for the schema, we are cleaning up the code to better handle
modifications to the schema, but it is a big job. We are updating code
as we work on newer features or go into files for bug fixes. After the
next major release we will be doing code cleanup to finally remove any
use of fixed data structures in the programming.


>  > Javascript and XMLHTTPRequest are wonderful in some
>  > ways, but are very hard to work with in a complex application like
>  > VICIDIAL and they are not the most efficient solution despite all of
>  > their advantages of client portability and
>  > zero-installation/configuration.
>
>
> Ain't that the truth.  :-)
>
>
>  > We would love to hear more about the specific issues that you found
>  > working with VICIDIAL, if you could email me back or just post them to
>  > the VICIDIAL forum or Issue Tracker.
>
>
> I would, but a lot of my issues are philosophical and relate to broad
>  project management / business considerations rather than extremely
>  specific, well-defined lackings.
>
>
>  > Could you tell me what the last version of VICIDIAL that you worked with was?
>
>
> I do not recall.  It was whatever was current and stable in January 2008.

That would probably be the 2.0.4 release. We have done 3 updates to
this branch in SVN and released new code snapshots with all sorts of
bug fixes and enhancements in the last 7 months that do not require
any database changes to upgrade.

>
>  -- Alex

Thanks again for your well though-out feedback.

MATT---


>  --
>  Alex Balashov
>  Evariste Systems
>  Web    : http://www.evaristesys.com/
>  Tel    : (+1) (678) 954-0670
>  Direct : (+1) (678) 954-0671
>  Mobile : (+1) (706) 338-8599
>
>  _______________________________________________
>  -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
>  AstriCon 2008 - September 22 - 25 Phoenix, Arizona
>  Register Now: http://www.astricon.net
>
>  asterisk-users mailing list
>  To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>