[asterisk-dev] AstriDevCon Follow Up - Asterisk and Kamailio - smoother integration

[Matthew Jordan]

On Mon, Jan 12, 2015 at 10:53 AM, Daniel-Constantin Mierla <
miconda at gmail.com> wrote:

> Hey guys,
>
> I hope everyone here enjoyed nice time during the winter (or summer for
> lucky ones at this time of the year) holidays break!
>
> Being rather busy after AstriDevCon, but I also postponed starting the
> discussion, given that I expected low activity ahead and didn't want to
> get it lost in the archive ... so back to the topic ... and expect large
> message below ...
>

No worries at all! I'm still digging through the list of items from
AstriDevCon as well - the holidays always slows down progress on these
kinds of things.


> During AstriDevCon sessions in Las Vegas, we discussed briefly of what
> can be done for a better integration between Asterisk (as media server)
> and Kamailio (as a front end SIP singling server/proxy). I said I would
> follow up on mailing list, so here we are. Obviously, many are using the
> two together, therefore any improvement to make it simpler could make a
> very good impact on both sides.
>
> To be honest, I lost track of many new 'cool' things happening lately in
> Asterisk with the newer APIs, I will have to catch up very soon,
> meanwhile, if any of my ideas are already there, then even better.
>
> The typical integration of Asterisk and Kamailio was to share somehow
> user profiles via database -- relying on Asterisk realtime, either using
> Asterisk database structure with kamailio connecting to it, or making
> SQL views to suit the two applications. I wrote some tutorials over the
> time -- for reference, just in case anyone wants to get the idea better:
>
>   - http://kb.asipto.com/asterisk:index
>
> The approach works very well when dealing in a more or less static
> deployment. However, there is a lot to benefit on top of elasticity of
> the cloud networks, where the nodes could be fired on demand, in
> different locations of the world. In such situations, sharing the
> database can become really slow, adds issues with replications, etc.
>
> To set the specifics of a deployment:
>   - kamailio takes care of signaling 'heavy' processing (authentication,
> user location, filtering DoS attacks, etc.)
>   - asterisk takes care of 'heavy' pbx features and media processing
>
> When I fire a new asterisk, I want to give it (e.g., in config file) one
> or many IP addresses that belong to kamailios and asterisk should
> consider them trusted, no other authentication should be done for
> traffic originating from those IPs. The rest of integration would
> hopefully automagically just work.
>
> One of my ideas was to be able to define subscriber (phone) profile
> templates and the proxy will indicate which template to use for caller
> and callee (when they are local subscribers). Attributes of the template
> can be taken from SIP message or be statically defined. I know about the
> config templates with [name](!), but that is still quite static,
> extension have to be defined and templates associated with them. It
> could be on same direction, but more dynamic.
>

This gets into the notion of having a "push" configuration model instead of
a "pull" configuration model that was discussed at AstriDevCon. Note that
this is just one method of solving this problem - as you mention later,
there are others which may be better - but it's probably worth discussing
push configuration here.

A fuller explanation follows:

In Asterisk 12, the notion of a generic data access layer was introduced
that wraps up the traditional configuration approaches (static conf files,
static Asterisk Realtime, and dynamic Asterisk Realtime) as well as some
internal data providers (memory, AstDB). This was done for quite a few
reasons:
(1) Using the ARA had quite a few deficiencies, as evidenced by the SQL
statements littering chan_sip and other portions of the code base. It was
quite common for logic to diverge based on what provided the configuration,
which led to all sorts of difficult bugs to test and fix. That being said,
the actual ARA drivers themselves could fit behind a new API, and so they
were mostly left alone. This lets older modules continue to use ARA as they
previously did, and newer modules to take advantage of the existing
backends.
(2) A new generic data access layer allowed us to take advantage of work
done in Asterisk 11's configuration framework. The configuration framework
in Asterisk 11 provides type conversion from configuration data to in
memory objects, default value application and object construction,
documentation checking, object lifetime management, as well as thread
safety during reloads. All of this makes modules that use the configuration
data a lot safer and more consistent.
(3) Because data can also be provided by an in memory cache or the AstDB,
we can now provide caching for ARA backends. A user can also modify via
configuration where data is stored, pushing data out to a more permanent
storage if the AstDB isn't desired.

On top of it, this is a pluggable architecture, where data providers are
simply dynamically loadable modules. The method of providing configuration
is very 'ad hoc', wherein a user of the new data access layer does not care
where the data comes from, so long as something can provide it.

In the end, Josh called this "sorcery", presumably because there is an
awful lot of magic that happens under the hood in that API.

Today, all of the providers - called "wizards" - use a pull model of
providing data. Taking the PJSIP stack as an example (this is a very high
level view, there's a lot of details removed):

* An inbound request is received and passed up the stack into res_pjsip
* res_pjsip, which has a pluggable architecture of its own, looks for an
endpoint identifier module
* res_pjip_endpoint_identifier_user picks up the request, and look at the
user portion of the SIP URI in the From header
* res_pjsip_endpoint_identifier_user queries sorcery for an endpoint that
matches the user
* sorcery (via configuration) maps this to some configuration provider.
This could be a static file (such as pjsip.conf), a database, or something
else. It converts the key/value pairs into an ast_sip_endpoint ao2 ref
counted object.
* res_pjsip_endpoint_identifier_user gets back the ast_sip_endpoint object,
passing it back to res_pjsip - which now has an endpoint to use for the
request.

Now, we could just as easily *not* pull for the configuration data.
Instead, we could do something like the following:

* An API call is made which pushes a JSON blob into Asterisk containing the
necessary information for a PJSIP endpoint. A sorcery wizard picks this up
and stores it in the in-memory sorcery wizard.
* An inbound request is received and passed up the stack into res_pjsip
... and all the rest is actually the same - as sorcery simply uses the
in-memory cache of the received information from the API.

Since caching is tricky, the lifetime of the PJSIP endpoint in this
Asterisk instance could just as easily be managed by the external
application. A different API call could tell the Asterisk instance to purge
the endpoint from memory.

>From Kamailio's perspective, you could:

* Receive an inbound request
* Pick some Asterisk resource in your pool of available servers
* (Assuming this is done using a REST API) PUT the configuration of the
endpoint on the specific destination server
* Route the call to that server
* Later, DELETE the configuration off the server


>
> For better exemplification, in pseudo code:
>
> [template-match]
> template-as-caller=${header(X-Caller-Template)
> template-as-callee=${header(X-Callee-Template)
>
> [template-user-x]
> userid-as-caller=${From-Header-URI-Username}
> userid-as-caller=${Request-URI-Username}
> codecs=g711,g729
> mwi=yes
> maiboxid=${header(X-MailBoxId)}
> maiboxpin=${header(X-MailBoxPin)}
> location =
> "sip:${From-Header-URI-Username}@
> ${Source-IP}:${Source-Port};transport={Transport-Protocol}"
> publish-dialog-info=yes
> ...
>
> The proxy has to set the header X-Caller-Template if the caller is local
> and X-Callee-Template if the callee is local, plus other headers than
> are necessary/used in templates.
>

This is another way of identifying inbound requests without requiring any
API interaction whatsoever. You'd certainly have the advantage of being
even more performant - no push configuration necessary whatsoever - but at
the cost of some of the flexibility that the push configuration affords.

As mentioned earlier, the PJSIP stack has a very flexible architecture that
lets you "plug-in" how endpoints are identified. In the existing stack,
there are three different endpoint identifiers:
* res_pjsip_endpoint_identifier_user - associate inbound requests based on
the user portion of the SIP URI in the From header
* res_pjsip_endpoint_identifier_ip - associate inbound requests based on a
static IP address
* res_pjsip_endpoint_identifier_anonymous - associate inbound requests with
one particular endpoint. If present, all unmatched requests are matched to
this endpoint.

You could just as easily have res_pjsip_endpoint_identifier_header. It
could do what you said: look for a custom header (which, itself, could be
configurable) and, if the header is present, match it to a single specific
endpoint in the same fashion as res_pjsip_endpoint_identifier_anonymous.

The downside of this approach is that all devices must support all
configuration parameters of a single configured endpoint. That is, if you
have something like:

[header-identifier]
type=header-identifier
endpoint=caller-endpoint
X-Caller-Template=foo

[header-identifier]
type=header-identifier
endpoint=callee-endpoint
X-Callee-Template=bar

[endpoint-template](!)
type=endpoint
allow=!all.g722,ulaw

[caller-endpoint](endpoint-template)
context=from-kamailio-local
identify_by=header

[callee-endpoint](endpoint-template)
context=from-kamailio-remote
identify_by=header

Then you could have your endpoint identifier match all requests with a
header of X-Caller-Template to caller-endpoint and all requests with a
header of X-Callee-Template to callee-endpoint. However, all devices would
have to support g722 and ulaw, and - if NAT settings were necessary - then
all devices would have to share the same NAT settings. This may not be must
of a concern in a well defined system front-ended by Kamailio.

Writing this module would not be terribly difficult - it would look a lot
like a combination of res_pjsip_endpoint_identifier_ip and
res_pjsip_endpoint_identifier_anonymous. (Not that SLOC is a good
measurement of difficulty, but res_pjsip_endpoint_identifier_ip tops out at
540 lines of code, and it is by far the largest of the endpoint
identifiers).

Note that this approach is not mutually exclusive with a push
configuration, or with using a database. With sorcery, the configuration
shown above could come from a database - it's just that Asterisk won't need
to obtain it often.


>
> Practically, Asterisk would have to build the user profile on demand,
> without needing to interogate any database or other resource over the
> network (I know APIs can be used everywhere :-), but they fail also, or
> in other words, SIP is an API as well). It can reuse the variable
> already available on the SIP channel, I used pseudo-code to make it
> easier to understand.
>

Yup! And as I mentioned, the idea of having an endpoint identifier that
looks at custom SIP headers is not mutually exclusive with an API. One
could foresee having a small set of SIP headers that Kamailio uses to 'tag'
allowed requests, particularly if there were a variety of devices that it
was passing to Asterisk (different codec sets, etc.) It could have all of
these pre-configured on its pool of servers; have the templates defined in
a database; or it could use a push model to provide the configuration on
the fly to Asterisk servers as they are spun up.


>
> For the moment I would stop here, already long message. And might be
> already possible ... if not, before going on other directions, lets see
> the interest and maybe other proposals for a better media
> server/signaling server integration.
>

I think it's very possible to add a new endpoint identifier.

The push configuration model is a bit more work, but also very doable.

I'd be happy to provide more details on either approach if someone were
interested in contributing it!


>
> Before the real end, some questions for the larger devel community that
> should help deciding on how to get the two projects working smoother
> together:
>   - what would you like from the proxy (signaling server) to be exposed
> easier?
>   - if already using the two, what parts you would like to simplify?
>   - how would you split the workload? In other words, what services you
> would prefer to be done better in one or the other, given that some
> features overlap?
>

I'm going to defer to Olle on those questions for now - as he uniquely has
a lot of experience with both Asterisk and Kamailio - but I'll think some
more on those questions this week.


> Have a great 2015 coding Asterisk and Kamailio! Looking forward to
> meeting some of you around the globe this year!
> Daniel
>

Thanks for the e-mail and suggestions! I'm looking forward to what else we
can come up with.

Matt

-- 
Matthew Jordan
Digium, Inc. | Engineering Manager
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150112/2c4790f4/attachment-0001.html>