[asterisk-users] Experience with Vicidial

[Alex Balashov]

Matt Florell wrote:

> But seriously, Asterisk is a better example of doing things right more
> recently, a couple of years ago all sorts of stuff went into the
> "stable" releases of Asterisk without  enough testing resulting in
> some pretty big bugs (like in asterisk 1.2.10-12) And even further
> back the code conventions of Asterisk 6 years ago were in many places
> about as good as VICIDIAL is today. I think it's the evolution of a
> project from being in its early years as opposed to its more mature
> years like Asterisk has started to enter. If I were to start writing
> VICIDIAL today I would have done many things a lot differently from
> how they are, but then again if I started from scratch today it
> wouldn't be a functioning product for many months.

I know this dilemma well, and sympathise.

In my experience, with some projects a gradual evolution to more 
sophisticated approaches is possible, but with others the work involved 
in making that happen would be greater - and the result less elegant and 
cohesive - than simply rewriting the thing.  It depends on a lot of 
things, some of them important - but intangible - aesthetic judgments.

> There are a few large companies that have adopted VICIDIAL and
> internalized the maintenance of the code for their systems, so we know
> that the state of the code has not been a road-block for all
> companies.

Oh, sure, and I doubt that anything is a roadblock with sufficient 
spiritual commitment.

The ViciDIAL code *is* modifiable and maintainable, logically;  it's 
code, after all.  At the end of the day, it is still Perl or PHP.  The 
issues I've run into have been with the willingness of developers and 
managers to dive into code that looks like that.  The claim is never 
that it's conceptually too complicated, just that any semblance of 
organisation, structure, readability, modularisation or abstraction is 
conspicuously absent.  For that reason, it is seen as too much work to 
deal with it.

> Most companies will do some degree of basic lead loading and lead
> exporting integration on the back-end along with integration with
> web-based CRM systems on the agent front-end. And this has proven to
> usually be not too difficult for many companies to accomplish.

A lot depends on the size and scope of the operation and the intended 
application of the dialer.

"It uses an open database, and you can put your stuff into the database" 
is a surprisingly ineffective integration path in settings with either 
high volume or dense business rules requirements.  This is actually one 
of the biggest obstacles I ran into in the previously mentioned endeavours.

For one, by requiring that leads be entered into ViciDIAL's lead table, 
you force the source to shoehorn its data exactly to your schema for 
leads, which usually results in some degree of information loss and 
redundancy.  And any derivative consequences elsewhere in the database 
arising from those importations must also be dealt with, e.g., managing 
IDs used for keying the rows, etc.

For another, unless the data used by the company conveniently resides in 
MySQL - yea, the same instance of it - as ViciDIAL's, some sort of 
synchronisation job or process will be necessary to refresh the lead 
table by pumping in new data.  This is not really feasible in situations 
with a very, very large amount of leads.  It creates yet another 
scheduled, recurring potential point of failure (the sync job).  The 
data simply may not be possible to load to keep pace with the dials.  It 
consumes bandwidth and resources and time.

If a company does most of its lead management, qualification and other 
applications of business rules in its own database (which may or may not 
be something Linux/open-source based, btw!), that database becomes a 
redundant middle storage layer for the data since the goal is simply to 
get it into ViciDIAL.  But then, some persistent information about the 
leads must be stored, as a lot of companies tend to call leads back and 
follow up... etc, etc, etc.

In my experience, this led to attempts to bifurcate the ViciDIAL lead 
table and join on some other custom tables containing needed 
supplemental information.  Then it was discovered that this will require 
more code changes (many of redundant, duplicated statements) in ViciDIAL 
than miles in the width of Heaven.  That's usually the point at which 
hats were thrown down and, "Damn it, we might as well just write 
something ourselves" was said.

Anyway, my real point here is that I think relatively easy, turn-key 
integration and customisation on a software and data interchange level 
may prove to be the most significant issue in determining the success 
and future of ViciDIAL, at least if my experience is any indication.

(And what my experience suggests is that almost nobody uses it straight 
out of the box.  I could be very wrong on that.  Perhaps some of the 
most happy adopters are companies with very different business models 
from those that use predictive dialers traditionally.)

It's the main reason why companies with existing, proprietary dialers 
are often looking to leave them and consider ViciDIAL as a replacement; 
  the proprietary dialer is a hindrance, in terms of cost and/or 
features, to the use of sophisticated custom logic.


> We have actually do now have a agent-side API that allows outside
> processes to have very limited control over agent interface functions.
> As it's features grow we will also work on more back-end API
> functionality not directly related to the agent.

As suggested above, I think this is absolutely crucial.

> Yes, I agree, modularization is a future goal of ours that
> unfortunately we do not have the time to work on at the moment, but it
> is something that we have talked about a great deal.

I'm glad you're aware of it.  :)

The problem one always runs into is that the bigger the code base grows, 
the less likely that any retroactive 
refactoring/reorganisation/rewriting is going to happen, or is even 
possible.

That's why I was bringing up the possibility of a rewrite.

> On the other hand, if you want to
> significantly alter the schema or layout of the agent interface, then
> you are in for a lot of work.

I think this is quite high on the list of importance, though.

It shouldn't be a lot of work;  to the extent that it's going to be 
work, I think most adopters are willing to do the work.  But the work 
needs to not require changing extremely obfuscated code in 50 different 
places.  There should be one place where these things are respectively 
defined and built.

> 
> Having a centralized system with universal data access has really
> helped us in scalability and integration with other systems. Initially
> I did not start out with MySQL handling everything, but as I ran tests
> and worked more with it, it proved to be extremely reliable and very
> fast, so I built everything around it and it has proven that it can
> handle the job. It might not be the best or most efficient tool for
> the job, and we may look into other methods of handling IPC in the
> future, but for now it has proven to be one of the most stable
> elements of a VICIDIAL system, with some databases running for over a
> year without restart while still handling 5000+ queries per second.

I realise it's currently doable, but that does not mean it is 
methodologically correct, formally viable, or ultimately scalable.  Much 
of why what you are saying is true is accounted for by the continuous 
and rapid evolution of computing hardware - no matter how woefully 
inefficient, it still works OK.  This is the same argument used by folks 
who mis-apply XML, or Java, or other things considered "very 
heavyweight" unnecessarily but have beefy hardware that seems to let 
them wing it without disruptive performance penalties anyway.

Just because it can be done does not mean it should be done.

I think this may be what separates programming methodologies into 
radically different classes;  there's Just Making It Work (even to 
relatively demanding standards of robustness), and then there's Doing 
Things The Right Way.

There's a reason why using the database as an IPC mechanism is 
considered an antipattern.

( http://www.bookrags.com/wiki/Database_as_an_IPC )

>>  Also, you don't necessarily have to develop a custom IPC protocol for
>>  communication amongst components;  there are plenty of workable
>>  distributed service architectures out there.
> 
> Is there a specific IPC protocol that you would recommend I take a look at?

No, I don't have any recommendations in this area per se.  There are 
lots of different approaches you could take, from the web services 
approach to the Java RMI-style approach.  The Perl runtime environment 
offers a very advanced level of symbol table introspection, resulting in 
plethoric attempts to fashion distributed object/process architectures 
for it.

If a lot of the IPC happens between the agent portal side and the dialer 
side, I would recommend considering a SOAP / XML-RPC type approach, 
although by very virtue of being XML I have often criticised them for 
being unnecessarily heavyweight.  Ah well.

> 
> After the next full release we will do away with MySQL 4.0 and 4.1
> support and require 5.0+ so we can do a lot of what you suggest. We
> have hundreds of systems out there with 4.X MySQL and we have always
> tried to stay backwards compatible, but we have started to put notes
> into the documentation and in the forums that we will be dropping
> support for these older versions so that we can add new functions and
> increase efficiency in future versions.

Ah, good deal!

> As for the schema, we are cleaning up the code to better handle
> modifications to the schema, but it is a big job. We are updating code
> as we work on newer features or go into files for bug fixes. After the
> next major release we will be doing code cleanup to finally remove any
> use of fixed data structures in the programming.

Awesome.  Glad to hear.  I'm glad you've got this stuff in mind.

-- Alex


-- 
Alex Balashov
Evariste Systems
Web    : http://www.evaristesys.com/
Tel    : (+1) (678) 954-0670
Direct : (+1) (678) 954-0671
Mobile : (+1) (706) 338-8599