[asterisk-dev] PJSIP realtime scalability problem

[Michael Ulitskiy]

On Sunday, October 18, 2015 02:20:37 PM Matthew Jordan wrote:

> No worries - just want to make sure we get to the bottom of the issues
> you're experiencing!

Thanks. I appreciate that.

> > The problem here isn't actually related to caching implementation, but to
> > the way pjsip matches endpoints.
> >
> > Whenever sip request arrives pjsip initially performs lookup for
> > 'username at domain' and if it fails it falls
> >
> > back to lookup by username only.
> >
> > It results in 2 queries:
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'ep1 at domain';
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'ep1';
> >
> > Now in my environment only the 2nd one will succeed and will be cached. Now
> > for every sip request my asterisk
> >
> > will be issuing
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'ep1 at domain';
> >
> > that will never succeed followed by retrieving 'ep1' from cache.
> >
> > Basically I'd like to have a way to suppress lookup for 'username at domain' or
> > at least to cache the negative results.
> >
> 
> I think that makes sense. The code from
> res_pjsip_endpoint_identifier_user that does the endpoint
> identification by user name portion is shown below:
> 
>     /* Attempt to find the endpoint given the name and domain provided */
>     snprintf(id, sizeof(id), "%s@%s", endpoint_name, domain_name);
>     if ((endpoint = ast_sorcery_retrieve_by_id(ast_sip_get_sorcery(),
> "endpoint", id))) {
>         goto done;
>     }
> 
>     /* See if an alias exists for the domain provided */
>     if ((alias = ast_sorcery_retrieve_by_id(ast_sip_get_sorcery(),
> "domain_alias", domain_name))) {
>         snprintf(id, sizeof(id), "%s@%s", endpoint_name, alias->domain);
>         if ((endpoint =
> ast_sorcery_retrieve_by_id(ast_sip_get_sorcery(), "endpoint", id))) {
>             goto done;
>         }
>     }
> 
>     /* See if the transport this came in on has a provided domain */
>     if ((transports =
> ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "transport",
> AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL)) &&
>         (transport = ao2_callback(transports, 0,
> find_transport_in_use, rdata)) &&
>         !ast_strlen_zero(transport->domain)) {
>         snprintf(id, sizeof(id), "%s@%s", endpoint_name, transport->domain);
>         if ((endpoint =
> ast_sorcery_retrieve_by_id(ast_sip_get_sorcery(), "endpoint", id))) {
>             goto done;
>         }
>     }
> 
>     /* Fall back to no domain */
>     endpoint = ast_sorcery_retrieve_by_id(ast_sip_get_sorcery(),
> "endpoint", endpoint_name);
> 
> As you can see, we have a number of lookups that can occur:
>  (1) First, we attempt to look up the endpoint by user at domain
>  (2) If that fails, we grab any domain aliases that may exist, and do
> a lookup by user at domainalias
>  (3) If that fails, we look at the transports to see if they have any
> domains. If so, we do a lookup by user at transportdomain
>  (4) Finally, if that fails, we do a lookup by user name portion only
> 
> A simple solution would be to have a configuration option that skips
> some of those checks.
> 
> It's probably worth opening an issue for that as well. While an
> improvement, it is one that would skip one to three sorcery lookups on
> every endpoint request, which would reduce contention both on the
> cache and - when the cache is empty - on the database. For larger
> systems, as you've pointed out, that's practically a bug.

Yes. I found that code and in my local copy I just commented out unneccessary lookups.
Having a configuration options to control them in my opinion would be the best.
I'll sure open an issue for that too.

> > sorcery.conf:
> >
> > [res_pjsip]
> >
> > endpoint=config,pjsip.conf,criteria=type=endpoint
> >
> > endpoint/cache=memory_cache,expire_on_reload=yes,object_lifetime_maximum=600,object_lifetime_stale=300
> >
> > endpoint=realtime,ps_endpoints
> >
> > aor=config,pjsip.conf,criteria=type=aor
> >
> > aor/cache=memory_cache,expire_on_reload=yes,object_lifetime_maximum=600,object_lifetime_stale=300
> >
> > aor=realtime,ps_aors
> >
> > extconfig.conf:
> >
> > ps_endpoints => pgsql,users,pjsip_endpoints_v
> >
> > ps_aors => pgsql,users,pjsip_aors_v
> >
> > When asterisk starts up and loads pjsip it does the following:
> >
> > SELECT * FROM pjsip_aors_v WHERE id LIKE '%' ORDER BY id
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id LIKE '%' ORDER BY id
> >
> > thus loading all endpoints and AORs in memory. Then the worst part, it
> > follows on with loading all
> >
> > endpoints and AORs individually with queries like this:
> >
> > SELECT * FROM pjsip_aors_v WHERE id = 'ep1'
> >
> > SELECT * FROM pjsip_aors_v WHERE id = 'ep2'
> >
> > ...
> >
> > SELECT * FROM pjsip_aors_v WHERE id = 'epN'
> >
> > then
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'ep1'
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'ep2'
> >
> > ...
> >
> > SELECT * FROM pjsip_endpoints_v WHERE id = 'epN'
> >
> > With 10K endpoints it results in 20K queries to db at asterisk startup. Now
> > imagine multiple asterisk
> >
> > servers. This is the biggest problem.
> >
> > Also, to my surprise, this initial loading doesn't populate cache.
> >
> > Right after asterisk startup I do "sorcery memory cache dump
> > res_pjsip/endpoint" and it's empty therefore causing
> >
> > additional db lookups as asterisk starts to serve sip requests.
> >
> 
> One question I have on this point (as your later e-mail noted that the
> cache was getting populated on initial load) - what ARA backend are
> you using for realtime? While nothing has moved forward yet with it,
> we are aware of some improvements that could be made with the ODBC
> backend specifically around performance.

I'm using res_config_pgsql. I've tried to use odbc once, couldn't make it work and abandoned it 
(no blame to anyone, likely I just wasn't persistent enough). Didn't think that introducing
yet another abstraction layer is a good idea anyway since the native driver exists.

I actually doubt it has anything to do with backend driver as I'd imagine that lookup requests are created
by upper layers and the biggest problem of course is a separate query per-endpoint. That doesn't seem
to be needed because asterisk performs bulk loading of all endpoints as the first step anyway (LIKE '%' query)
and should already have all that information.

Please don't forget that this all applies not just to endpoints, but to AoRs (and probably to other objects) too.

I've just started to look at pjsip codebase, but here's what I think. It looks like bulk loading is done with 

ast_sorcery_retrieve_by_fields(sip_sorcery, "endpoint",AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL);

API call. Correct me if I'm wrong, but I think this call does not populate cache.

Individual endpoints lookup is done with ast_sorcery_retrieve_by_id() and this call does seem to work with cache
(check if object is in the cache and populate cache if needed).

If you could make bulk call to populate cache then all subsequent individual lookups should be satisfied by cache, thus
moving contention from database to local cache. While not perfect it would make me MUCH happier.

> Initial loading is probably always going to be something of a problem,
> as endpoints often will get populated during qualify/registration. The
> only way I can think to address that is to either not qualify the
> endpoints, qualify the endpoints with a larger range of qualification
> times, or to address any bottlenecks in the realtime drivers
> themselves.

It's totally fine to retrieve endpoints that need to be retrieved. But I guess the main idea of on-demand loading
is to retrieve only those that needed and nothing else. 
As far as I understand chan_sip offered the following tradeoff for endpoints with "qualify": qualify starts after endpoint
gets loaded in memory, i.e. is used for the first time (registered/dialed something/received a call/etc). That was totally
fine with me.
On the hand if you could qualify initial load with something like: 
WHERE qualify_frequency is not null or exists(registration) or ...
that's perfect.
I'm not familiar with pjsip architecture well yet, but if that's out of the question, then, please just restrict the initial
loading to a single bulk query

> >> > Caching also doesn't help at all with CLI commands like "pjsip show
> >
> >> > endpoints" in which case asterisk
> >
> >> >
> >
> >> > reloads the whole list from db instead of showing what it has in-memory.
> >
> >>
> >
> >> That actually is by design.
> >
> >>
> >
> >> Say we are caching endpoints. The cache only contains the n most
> >
> >> recently requested endpoints, *not* every endpoint that you may have
> >
> >> in your system. Hence, if you ask for all endpoints, we have to bypass
> >
> >> the cache and get all endpoints in order to accurately fulfill the
> >
> >> request.
> >
> >>
> >
> >> Given that this is a human interaction and not a run-time machine
> >
> >> interaction, the fact that you're requesting all endpoints results in
> >
> >> going out to the database is not unreasonable.
> >
> >
> >
> > Well I see your point. The thing is that in a system where endpoints are
> >
> > dynamically spread over multiple asterisk systems I never want to see
> >
> > all the endpoints. Only those that's been served by this asterisk and
> > cached.
> >
> > May be it's worth having a command that shows only cached endpoints?
> >
> > Basically I was happy with how chan_sip worked in that regard - only loading
> >
> > endpoints on-demand and only showing those endpoints that are loaded in
> > memory.
> >
> 
> There actually was an e-mail on the users list about a similar notion
> [1] - namely, a 'pjsip show endpoints like' command. Are the endpoints
> on a particular Asterisk system named in such a fashion that you know
> - from the name - which ones you generally want to see?

No, not at all. Endpoints are assigned to asterisk systems dynamically by frontend proxy
and can be dynamically moved between systems at any time. That's why it was really
convenient to work with chan_sip realtime that only loaded endpoints on demand (as sip request arrived)
and always could show me which endpoints have recently been served on this system.

> [1] http://lists.digium.com/pipermail/asterisk-users/2015-October/287781.html
> 
> It is possible as well to 'dump the cache', as you've noticed from the
> CLI command. A PJSIP variant of that could probably be written as
> well, although I haven't investigated exactly what level of access
> PJSIP has to what is in the cache and what is not. (Generally, the
> fact that something is coming from a cache should be transparent to
> the consumer.)

CLI commands is actually lesser of a problem, especially if name completion issue we discussed before
gets optimized. Yes, it would be nice to have a pjsip variant of commands that looks in the cache,
but I guess at this point I can survive with cache dump. I could probably just alias "pjsip show cache" to
"sorcery cache dump" command

> 
> If I boil your issues down, it sounds like the problems are mostly around:
> (1) CLI commands (or AMI actions, for that matter) that attempt to
> operate on all endpoints. Generally, options that accessed only the
> subset of the endpoints that were present in the cache would be
> desirable.
> (2) Initial loading of endpoints can take awhile and present a lot of
> contention on the database.
> 
> Does that sound right?

Absolutely.
(1) It's desirable, but not as much of a problem as (2). As you noted I could look at the cache. An ability to look at
individual cache entries in addition to wholesale dump would also be desirable I guess.
(2) A separate query per-endpoint is the biggest problem. Other optimizations would be welcome too.

I've just started to look at pjsip, so there may be more stuff I'll be bothering you with, but this is what I've got so far ;-)
 
> Matt

Thanks a lot,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20151019/641b954f/attachment-0001.html>