[asterisk-dev] Parsing in Asterisk

[Tilghman Lesher]

We need a revamp of parsing in Asterisk.  What we have now creates
serious problems when writing complex extensions.conf logic.  For 
example, I recently had to coach someone to write the following in his
extensions.conf to make it work correctly:

exten => s,n,Set(ODBC_TEST3(foo)=123\,\\"456\,789\\"\,012)

Dissecting this, we're passing 3 arguments (as VAL1, VAL2, VAL3) to a
dialplan function named ODBC_TEST3.  The second argument contains a
comma, which needs 4 different escaping sequences to be passed correctly
into the function.  First, the comma itself needs to be escaped, to
avoid it from being translated into a '|' by the extensions.conf parser.
Second, the entire second argument needs to be quoted so that it is seen
as a complete entity (as opposed to two different arguments).  Third,
the quotes need to be escaped (with a '\') to prevent them from being
eaten by the argument parser called by Set (because Set takes multiple
variable pairs).  And last, the backslashes themselves need to be
escaped, to protect them from the extensions.conf parser, which itself
sees as escape characters.

If you can't tell already, this is madness, and we need to do something
about it.

To understand where all of these layers came from, we have to go back
a bit into Asterisk dialplan history.  The first extensions.conf syntax
looked a bit different than it does today (ignoring changes in the app
names):

exten => s,3,SetVar,foo=1

Back then, we didn't have the nice syntax of the arguments passed to
the application enclosed in nice parentheses; rather, we had a list of
values to the exten keyword.  The application arguments just happened to
be the fourth argument.  Today, we rely on a transparent translation
from the new easier-to-read syntax to the old syntax.  This maintains
reverse compatibility, although at a cost in future complexity.

This leads me to my first conclusion:  we need to remove the transparent
translation, preferring the comma as our argument delimiter.  I am aware
that this is a significant change and it will require a flag day,
because all applications currently parse on the pipe character.
However, I believe that this is necessary to allow Asterisk dialplans to
become more complex without sacrificing readability.

The second problem is that at some point, the Set app was made to accept
multiple variable name/value pairs.  While this may seem like a good
idea, it really complicates things, because it means that we cannot
easily set values with pipes.  Worse, because it uses the standard
parser, it strips a level of escape characters, which means that in
order to pass such an argument as a value to a dialplan function, you
need to escape it twice.

At the end of the day, what I'd like to happen is for dialplan parsing
to become far more sensible than it is currently.  This will allow
dialplan developers to create more complexity, without continually
running into the wall of needing endless amounts of escape characters,
to get Asterisk to perform as expected.

I have gotten limited approval, at least in passing, for the change to
Set, but I wanted to put the issue in front of all the developers and
ask for feedback before we go and make any of these changes.  Obviously,
I think this is important, so at a meta-level, I need to ensure that
others are on board (or if I've missed something, I need feedback on
that, as well).

-- 
Tilghman