[asterisk-users] Experience with Vicidial

[Matt Florell]

On 7/17/08, Alex Balashov <abalashov at evaristesys.com> wrote:
> Matt Florell wrote:
>
>  > But seriously, Asterisk is a better example of doing things right more
>  > recently, a couple of years ago all sorts of stuff went into the
>  > "stable" releases of Asterisk without  enough testing resulting in
>  > some pretty big bugs (like in asterisk 1.2.10-12) And even further
>  > back the code conventions of Asterisk 6 years ago were in many places
>  > about as good as VICIDIAL is today. I think it's the evolution of a
>  > project from being in its early years as opposed to its more mature
>  > years like Asterisk has started to enter. If I were to start writing
>  > VICIDIAL today I would have done many things a lot differently from
>  > how they are, but then again if I started from scratch today it
>  > wouldn't be a functioning product for many months.
>
>
> I know this dilemma well, and sympathise.
>
>  In my experience, with some projects a gradual evolution to more
>  sophisticated approaches is possible, but with others the work involved
>  in making that happen would be greater - and the result less elegant and
>  cohesive - than simply rewriting the thing.  It depends on a lot of
>  things, some of them important - but intangible - aesthetic judgments.
>
>
>  > There are a few large companies that have adopted VICIDIAL and
>  > internalized the maintenance of the code for their systems, so we know
>  > that the state of the code has not been a road-block for all
>  > companies.
>
>
> Oh, sure, and I doubt that anything is a roadblock with sufficient
>  spiritual commitment.
>
>  The ViciDIAL code *is* modifiable and maintainable, logically;  it's
>  code, after all.  At the end of the day, it is still Perl or PHP.  The
>  issues I've run into have been with the willingness of developers and
>  managers to dive into code that looks like that.  The claim is never
>  that it's conceptually too complicated, just that any semblance of
>  organisation, structure, readability, modularisation or abstraction is
>  conspicuously absent.  For that reason, it is seen as too much work to
>  deal with it.
>
>
>  > Most companies will do some degree of basic lead loading and lead
>  > exporting integration on the back-end along with integration with
>  > web-based CRM systems on the agent front-end. And this has proven to
>  > usually be not too difficult for many companies to accomplish.
>
>
> A lot depends on the size and scope of the operation and the intended
>  application of the dialer.
>
>  "It uses an open database, and you can put your stuff into the database"
>  is a surprisingly ineffective integration path in settings with either
>  high volume or dense business rules requirements.  This is actually one
>  of the biggest obstacles I ran into in the previously mentioned endeavours.
>
>  For one, by requiring that leads be entered into ViciDIAL's lead table,
>  you force the source to shoehorn its data exactly to your schema for
>  leads, which usually results in some degree of information loss and
>  redundancy.  And any derivative consequences elsewhere in the database
>  arising from those importations must also be dealt with, e.g., managing
>  IDs used for keying the rows, etc.
>
>  For another, unless the data used by the company conveniently resides in
>  MySQL - yea, the same instance of it - as ViciDIAL's, some sort of
>  synchronisation job or process will be necessary to refresh the lead
>  table by pumping in new data.  This is not really feasible in situations
>  with a very, very large amount of leads.  It creates yet another
>  scheduled, recurring potential point of failure (the sync job).  The
>  data simply may not be possible to load to keep pace with the dials.  It
>  consumes bandwidth and resources and time.
>
>  If a company does most of its lead management, qualification and other
>  applications of business rules in its own database (which may or may not
>  be something Linux/open-source based, btw!), that database becomes a
>  redundant middle storage layer for the data since the goal is simply to
>  get it into ViciDIAL.  But then, some persistent information about the
>  leads must be stored, as a lot of companies tend to call leads back and
>  follow up... etc, etc, etc.
>
>  In my experience, this led to attempts to bifurcate the ViciDIAL lead
>  table and join on some other custom tables containing needed
>  supplemental information.  Then it was discovered that this will require
>  more code changes (many of redundant, duplicated statements) in ViciDIAL
>  than miles in the width of Heaven.  That's usually the point at which
>  hats were thrown down and, "Damn it, we might as well just write
>  something ourselves" was said.
>
>  Anyway, my real point here is that I think relatively easy, turn-key
>  integration and customisation on a software and data interchange level
>  may prove to be the most significant issue in determining the success
>  and future of ViciDIAL, at least if my experience is any indication.
>
>  (And what my experience suggests is that almost nobody uses it straight
>  out of the box.  I could be very wrong on that.  Perhaps some of the
>  most happy adopters are companies with very different business models
>  from those that use predictive dialers traditionally.)
>
>  It's the main reason why companies with existing, proprietary dialers
>  are often looking to leave them and consider ViciDIAL as a replacement;
>   the proprietary dialer is a hindrance, in terms of cost and/or
>  features, to the use of sophisticated custom logic.
>
>
>
>  > We have actually do now have a agent-side API that allows outside
>  > processes to have very limited control over agent interface functions.
>  > As it's features grow we will also work on more back-end API
>  > functionality not directly related to the agent.
>
>
> As suggested above, I think this is absolutely crucial.
>
>
>  > Yes, I agree, modularization is a future goal of ours that
>  > unfortunately we do not have the time to work on at the moment, but it
>  > is something that we have talked about a great deal.
>
>
> I'm glad you're aware of it.  :)
>
>  The problem one always runs into is that the bigger the code base grows,
>  the less likely that any retroactive
>  refactoring/reorganisation/rewriting is going to happen, or is even
>  possible.
>
>  That's why I was bringing up the possibility of a rewrite.
>
>
>  > On the other hand, if you want to
>  > significantly alter the schema or layout of the agent interface, then
>  > you are in for a lot of work.
>
>
> I think this is quite high on the list of importance, though.
>
>  It shouldn't be a lot of work;  to the extent that it's going to be
>  work, I think most adopters are willing to do the work.  But the work
>  needs to not require changing extremely obfuscated code in 50 different
>  places.  There should be one place where these things are respectively
>  defined and built.
>
>
>  >
>  > Having a centralized system with universal data access has really
>  > helped us in scalability and integration with other systems. Initially
>  > I did not start out with MySQL handling everything, but as I ran tests
>  > and worked more with it, it proved to be extremely reliable and very
>  > fast, so I built everything around it and it has proven that it can
>  > handle the job. It might not be the best or most efficient tool for
>  > the job, and we may look into other methods of handling IPC in the
>  > future, but for now it has proven to be one of the most stable
>  > elements of a VICIDIAL system, with some databases running for over a
>  > year without restart while still handling 5000+ queries per second.
>
>
> I realise it's currently doable, but that does not mean it is
>  methodologically correct, formally viable, or ultimately scalable.  Much
>  of why what you are saying is true is accounted for by the continuous
>  and rapid evolution of computing hardware - no matter how woefully
>  inefficient, it still works OK.  This is the same argument used by folks
>  who mis-apply XML, or Java, or other things considered "very
>  heavyweight" unnecessarily but have beefy hardware that seems to let
>  them wing it without disruptive performance penalties anyway.
>
>  Just because it can be done does not mean it should be done.
>
>  I think this may be what separates programming methodologies into
>  radically different classes;  there's Just Making It Work (even to
>  relatively demanding standards of robustness), and then there's Doing
>  Things The Right Way.
>
>  There's a reason why using the database as an IPC mechanism is
>  considered an antipattern.
>
>  ( http://www.bookrags.com/wiki/Database_as_an_IPC )
>
>
>  >>  Also, you don't necessarily have to develop a custom IPC protocol for
>  >>  communication amongst components;  there are plenty of workable
>  >>  distributed service architectures out there.
>  >
>  > Is there a specific IPC protocol that you would recommend I take a look at?
>
>
> No, I don't have any recommendations in this area per se.  There are
>  lots of different approaches you could take, from the web services
>  approach to the Java RMI-style approach.  The Perl runtime environment
>  offers a very advanced level of symbol table introspection, resulting in
>  plethoric attempts to fashion distributed object/process architectures
>  for it.
>
>  If a lot of the IPC happens between the agent portal side and the dialer
>  side, I would recommend considering a SOAP / XML-RPC type approach,
>  although by very virtue of being XML I have often criticised them for
>  being unnecessarily heavyweight.  Ah well.
>
>
>  >
>  > After the next full release we will do away with MySQL 4.0 and 4.1
>  > support and require 5.0+ so we can do a lot of what you suggest. We
>  > have hundreds of systems out there with 4.X MySQL and we have always
>  > tried to stay backwards compatible, but we have started to put notes
>  > into the documentation and in the forums that we will be dropping
>  > support for these older versions so that we can add new functions and
>  > increase efficiency in future versions.
>
>
> Ah, good deal!
>
>
>  > As for the schema, we are cleaning up the code to better handle
>  > modifications to the schema, but it is a big job. We are updating code
>  > as we work on newer features or go into files for bug fixes. After the
>  > next major release we will be doing code cleanup to finally remove any
>  > use of fixed data structures in the programming.
>
>
> Awesome.  Glad to hear.  I'm glad you've got this stuff in mind.
>
>  -- Alex

Thank you very much for your suggestions and points you have made. I
will be putting the text of this email thread in my development notes
directory so that I can reference back to it in the future as we
address these elements that we have discussed.

MATT---