[asterisk-dev] scalability issue with realtime lastms update on qualify state change

Tilghman Lesher tilghman at meg.abyt.es
Sun Nov 17 01:58:12 CST 2013

On Sat, Nov 16, 2013 at 6:36 PM, Damon Estep <damon at soho-systems.com> wrote:
>> On Sat, Nov 16, 2013 at 2:02 AM, Olle E. Johansson <oej at edvina.net> wrote:
>> > Yes, there's one single thread in chan_sip handling all the Qualify
>> > and inbound sip messages. If we moved the database updates to a
>> > background thread the situation would improve. Then the monitor thread
>> > would just move on to the next message, letting another thread wait for the
>> database.
>> Secondarily, the pokes (qualifies) are sent in each peer thread.  If instead, a
>> single background thread were sending the qualifies, (possibly even rate limiting
>> the sending, in order to ensure the responses can be received on time), that
>> would additionally eliminate the flapping.  The issue is that the return packets
>> cannot be acted upon before the expiration time, because the initial poke
>> packets are sent in parallel, but the responses have to be received serially, within
>> a single thread.
>> I disagree with the alternate coding proposal, because it attempts to code
>> around the problem by turning something off, rather than attacking the core of
>> the problem, which solves it for everybody.
> Well, out of desperation I have written a patch that allows the user to disable realtime updates of lastms with a configuration setting of rtlastms=no. This at least lets me turn off database updates on qualify state change until there is a better solution, and I agree that the solution needs to be a better one. I will be testing in a lab and then in production over the next week.
> Even with a separate thread for qualify I would still argue that lastms realtime database updates should be configurable.
> rtupdate was introduced to toggle registration/expiration updates to the realtime database. Sometime after rtupdate was introduced qualify state changes were added to the list of realtime database updates. Qualify state changes are not registration information, so at that time it would have been a good idea to make them optional, since the amount of new database traffic can be substantial on heavily loaded systems.
> In summary, I believe being able to toggle lastms realtime updates on and off independently of realtime registration updates makes a lot of sense, and provides an immediate solution to a real problem, while not impacting users that rely on it, particularly if the default setting is on.
> In terms of the better solution, I am not familiar enough with all of the code and not competent enough to tackle the project of moving the qualify and subsequent database updates to a separate thread. My company on the other hand might be willing to contribute a few $ to the cause if there is someone with the time and skills to tackle it.  Any idea what kind of project that would be in terms of time?

If you really are averse to the effect that saving lastms in the
database causes, why don't you just remove that column from your
realtime tables?  We've had this ability to remove columns that you
prefer not to save in dynamic realtime since 1.6.2, and it sounds like
this is, in effect, precisely what you'd prefer.


More information about the asterisk-dev mailing list