[asterisk-dev] PJSIP Device Feature Key Synchronization architecture

[asterisk at phreaknet.org]

On 10/21/2022 5:34 AM, Joshua C. Colp wrote:
> On Thu, Oct 20, 2022 at 8:23 PM <asterisk at phreaknet.org 
> <mailto:asterisk at phreaknet.org>> wrote:
>
>     Hi, all,
>
>          Something I have been working on recently is adding support to
>     PJSIP for device feature key synchronization (the as-feature-event
>     Broadworks spec that many common IP phones, e.g. Polycom support) to
>     control server-side features from endpoints. It's using the PJSIP
>     pub/sub capabilities; I had to add the ability to execute a custom
>     module callback when a SUBSCRIBE is refreshed, but with that
>     addition,
>     it works as it's supposed to.
>
>
> You should further elaborate on all of the server-side features you 
> expect to implement, if it extends beyond DND.

The other one is Call Forwarding (Always/Busy/No Answer, with number of 
rings)

>
>     I wanted to solicit some input on what an ideal way of triggering
>     updates should be. Currently we have hints, which actually work
>     reasonably all right for Do Not Disturb, which is simply a boolean
>     on/off, easily represented with a hint and custom device state for
>     DND.
>     The PJSIP module emits an AMI event, the user can process it and
>     change
>     the device state if needed, which will trigger a NOTIFY to go out
>     to the
>     endpoint.
>
>
> What user? An outside AMI application? An internal consumer in Asterisk?

The administrator of the Asterisk system, who can add AMI logic to 
receive the event and then do something with it.
The reason this is needed is the phone isn't turning DND on directly, 
for example. It's merely a request. The server can decide to not allow 
it, for example if that phone isn't allowed to toggle DND. The server 
will process it, and send it the updated status. This usually reflects 
what the phone wanted, but not necessarily.
(I've elaborated on this more below)

>
>     This sort of came up about 12 years ago[1]. The actual SIP stuff
>     is not
>     complicated; it's the user interface to it that requires more
>     thought.
>     For call forwarding, there are more moving pieces and abusing
>     hints/custom device state for that is super clunky. You can't
>     communicate the call forwarding target, # of rings, etc. in a device
>     state, so additional hints are then needed for that. It works but
>     it's
>     super clunky and I don't think this is a great pipeline.
>
>
> Okay, so this covers call forwarding as well.
Yes.
>
>
>     I'm wondering if people have thoughts on what an ideal mechanism
>     would
>     be for users, once they process a request to enable/disable a feature
>     from the phone, to communicate that to the PJSIP module. The problem
>     with abusing hints, especially for call forwarding, is it's not a
>     good
>     way to communicate details into the module. One option perhaps is to
>     have dialplan extensions, setup in a manner similar to use with
>     EVAL_EXTEN, where it returns the current value needed, as any
>     relevant
>     function, DB, ODBC, CURL, custom function, etc. could be used to
>     retrieve the current feature value. The clunky part is more
>     signaling to
>     the PJSIP module that it needs to send the phone the updated
>     status (by
>     checking those extensions, for example). The device state callback
>     happens to be convenient for this kind of signaling but not really
>     appropriate here. It would be better to push the info into the module
>     directly rather than the signaling it and making it retrieve the
>     updated
>     data in some arbitrary way.
>
>     So with this in mind, I'm currently leaning in the direction of a
>     dialplan function/AMI action that could be used to set the
>     appropriate
>     info for a subscription, which would trigger the NOTIFY, and then
>     nothing would actually need to be added to the dialplan at all
>     (inasmuch
>     as hints and things of that nature). One d I think starting purely
>     from that perspective
>
>     isadvantage of this is that
>     for every single update, unlike callbacks, we have to traverse the
>     entire list of subscriptions (though maybe that's not a big deal).
>     The
>     bigger problem is this is push only and the PJSIP module still
>     needs to
>     be able to "pull" feature statuses on demand, which is where the
>     hint/lookup model is useful. A potential middle ground solution is
>     use
>     the dialplan function/AMI action to push only, but cache all of
>     this in
>     AstDB (as subscriptions themselves are), so that we can retrieve the
>     latest/most current value at any point if needed. Then we don't
>     need to
>     be concerned at all with how the user is managing state as that is
>     fully
>     decoupled, although obviously this would lead to a little
>     duplication/redundancy. Any thoughts?
>
>
> You've thrown a lot of lower level implementation details at us and to 
> be quite honest it's overwhelming. There's no full user facing 
> examples of how it would all work, beyond the bits and pieces in your 
> text that we'd then have to deduce and after reading a few times it 
> doesn't feel very friendly. To start off with: Is this a developer API 
> and interface, or is this also meant for the common everyday user? I 
> would hope it's for the common everyday user, in which case it should 
> be approached from that perspective first with implementation details 
> following.

Yes, it's for the common every day end user. A callback mechanism if 
used would be more of a development one but that would be more a means 
to an end.

Here's an example of what I have in my dialplan right now, in the 
subscribe_context for the endpoint:
exten => dnd_Polycom5,hint,Custom:${EXTEN}
exten => callfwd_Polycom5,hint,Custom:${EXTEN}
exten => callfwdalways_Polycom5,1,${FOOBAR(callforward,2135)}
exten => callfwdbusy_Polycom5,1,${FOOBAR(callforwardbusy,2135)}
exten => callfwdnoanswer_Polycom5,1,${FOOBAR(callforwardnoanswer,2135)}

The user gets the AMI event, processes it with whatever processing is 
needed (e.g. checking that the DND feature is available for that line, 
setting it in AstDB, ODBC, or whatever is the source of truth for 
feature statuses), and then updates the relevant hints above.

The module is currently hardcoded to use these extensions in the 
subscribe context: the prefix + the endpoint name. Obviously that's also 
inflexible.
Right now, the first 2 extensions the user will set to signal the module 
to send an updated NOTIFY. The first hint by virtue of being binary 
contains the DND status itself, and the 3 bottom extensions are needed 
to retrieve the call forwarding numbers from the source of truth for 
these features. (Here, FOOBAR is a custom - but any arbitrary - dialplan 
function I have that retrieves the status).

I bring this up only to show the current implementation and how hacky it 
is; I don't like this at all or think it's appropriate (except for DND, 
possibly). It was more a proof of concept of testing the underlying SIP 
technology.

A better implementation might look like:

User gets the AMI event and processes it as usual, and then simply does 
Set(PJSIP_DEVICE_FEATURE_STATE(PJSIP/Polycom1,donotdisturb)=enabled) or 
Set(PJSIP_DEVICE_FEATURE_STATE(PJSIP/Polycom1,callforwardingnoanswer)=8675309,4) 
(forward on no answer to 8675309 after 4 rings).

Internally, PJSIP_DEVICE_FEATURE_STATE could also persist input to AstDB 
so it's available to the module.

> As an everyday user I'd expect not to have to deal with AMI or complex 
> dialplan. I'd expect to be able to set and get the information from 
> the dialplan using dialplan functions (or a single function) so I 
> could use that in the dialplan, and have it "just work" with my phones 
> that support the feature. I'd expect it to persist across Asterisk 
> restarts. For example ${EXTENSION_DND(alice)} for retrieving DND 
> status of Alice, and if Asterisk restarted then that should stay the same.

That's a good point. The problem is that means that Asterisk internally 
is the source of truth of DND, and that may not necessarily be what 
people want. For instance, that wouldn't meet my own requirements. The 
way that Broadworks works is the phone is merely requesting a certain 
disposition, but the server isn't under any obligation to carry it out. 
So I think there needs to be some mechanism for the user to be involved 
in that pipeline, to be able to deny something that a phone wants. Maybe 
the user doesn't have DND, maybe certain phones aren't allowed to 
toggle, whatever. Lots of people store their feature states in MySQL 
databases and use them for Asterisk clusters. Some systems might have 
specific requirements for that. So users should have flexibility to 
reject it. On some systems, maybe the same DND status is used for 
several lines and Asterisk internally would not have any idea about this.
There are lots of different scenarios that are beyond what I think 
Asterisk itself should handle, hence the "two-part" process described 
above: the user (system admin) can do whatever needs to be done, and 
then just tell PJSIP what the new state is. PJSIP doesn't need to know 
or care about where feature states are actually stored or what the logic 
is or how they are mapped to endpoints.

I do see your point though and I think it would be nice to have a 
"simple, default mode" where Asterisk will internally just "approve 
everything" that the phone wants, and users can use that if that suits 
their needs, but fundamentally I think users should be able to be 
involved in the decision pipeline if they want/need to. I'm not yet sure 
what that would look like: maybe a pjsip.conf option to emit AMI events 
rather than auto-handling them? And then the PJSIP_DEVICE_FEATURE_STATE 
function would have to be used to tell PJSIP what to do, and Asterisk 
itself would not be the source of truth for feature statuses in this 
case (but it would cache them as described above) (though in this case 
reading the PJSIP_DEVICE_FEATURE_STATE could still return the cached 
disposition)

> For developers the same thing could apply except having AMI actions to 
> set and get.
>
> Functionally this is a simple storage mechanism of data. The AstDB is 
> used to persist such information. It could use sorcery to then be 
> backed by a real database. The manipulation could be like I said 
> previously, using dialplan functions with also AMI actions. To get the 
> information to consumers it would be a defined API to allow 
> setting/getting/subscribing/anything else, and for pushing to 
> consumers it would use stasis which would also give AMI events when 
> things changed. It's sorta like device state, but with persistent data 
> and simpler.
>
> This would also need explicit configuration to ensure that stale data 
> doesn't persist in the persistent storage.
Can you elaborate on what you mean by this? If something like 
PJSIP_DEVICE_FEATURE_STATE was the only way of telling the PJSIP module 
that a device state changed, then as long as the system administrator 
maintains the invariant of always telling PJSIP about feature state 
changes, this means it should have the latest feature data at all times, 
and it shouldn't get stale.

For example, this all needs to work with feature codes too - assuming 
*78 to enable DND, *79 to enable, and *72 for call forwarding variable, e.g:
exten => *72,1,Read(callfwdnum)
    same => n,Set(ODBC_CALLFWD(${CALLERID(num)})=${callfwdnum}) ; not 
sure if the syntax is right... it's been a hot second since I've used 
ODBC stuff...
    same => 
n,Set(PJSIP_DEVICE_FEATURE_STATE(${HINT(${CALLERID(num)})=${callfwdnum})
    same => n,Return(0)
exten => *78,1,SetODBC_DND(${CALLERID(num)})=1)
    same => 
n,Set(PJSIP_DEVICE_FEATURE_STATE(${HINT(${CALLERID(num)})=enabled)
    same => n,Return(0)
exten => *78,1,SetODBC_DND(${CALLERID(num)})=0)
    same => 
n,Set(PJSIP_DEVICE_FEATURE_STATE(${HINT(${CALLERID(num)})=disabled)
    same => n,Return(0)