[asterisk-dev] Clustering the Asterisk Database (for fun and profit)

[Matthew Jordan]

Hey all -

So, this last week I was travelling, and since flights are always a
good time to write some code, I decided to play around with an idea
that had been sitting in the back of my head for a little bit:
clustering the Asterisk Database. I got the notion after looking at
Kamailio's htable implementation, and the clustering that it can do
using SIP DMQ messages. A similar mechanism in Asterisk felt rather
doable, since we have:
(1) the ability to pub/sub internally the state of some object
(2) various clustering engines that make use of that internal pub/sub bus
(3) the ability to PUBLISH state between Asterisk instances

I initially debated whether it was appropriate to do in the AstDB or
in Sorcery (which would be a more generic way of clustering
information). In the end, I decided on the AstDB, for a few reasons:
(1) Local information is already stored in there from a variety of
places, while not everything uses Sorcery
(2) Putting it into Sorcery would make an already complex framework more complex
(3) I suspected it would be easier to 'control' the sharing of
information in the AstDB - which is a specific implementation - than
in a generic DAL like Sorcery

The actual implementation I went with works as follows:

* Any family in the AstDB can be marked as "shared" using a new
function - DB_SHARED().
  - Sharing family 'foo': DB_SHARED(put,global)=foo
  - Deleting the shared status of 'foo': DB_SHARED(delete)=foo

* The 'shared' status of a database family is independent of the
actual data stored in the AstDB. Thus you can change whether or not a
family is shared without impacting the underlying data.

* Updating/deleting a key in a shared family causes a Stasis message
to be fired off to consumers letting them know that the key was
updated/deleted. The consumers choose how to send that information to
other Asterisk instances in a cluster. Non-shared families are not
shared (which seems obvious...) - this minimizes the traffic on Stasis
to only the entries that should be propagated.

* Families in the AstDB can be shared in one of two ways - 'global' or 'unique'.
  - A 'global' shared family shares the family/key space with all
other Asterisk instances. For example, if in a local Asterisk instance
family 'foo' with key 'bar' has value '1', then a separate Asterisk
instance that publishes state for 'foo/bar' with value '2' will
overwrite the local copy of 'foo/bar'. This is useful when you want
the same data replicated across all Asterisk instances.
  - A 'unique' shared family replicates its state across other
Asterisk instances, but does *not* overwrite the local copy of the
families. Instead, remote versions of a particular family/key are
placed into a new family on the local Asterisk instance, determined by
the EID of the remote Asterisk instance. Using the same example, if in
a local Asterisk instance (with EID 11:11) the family 'foo' is shared
with key 'bar' and value '1', and a remote Asterisk instance (with EID
ff:ff) publishes state for 'foo/bar' with value '2', then in the local
Asterisk instance this does not overwrite the local copy. Instead, the
remote key/value pair is stored in 'ff:ff/foo/bar' with value '2'.
Likewise, since the local family 'foo' is shared 'uniquely, on the
remote instance, the local copy is stored in '11:11/foo/bar' with
value '1'. This is useful for families that may have conflicting data
keys that need additional context dependent processing to aggregate,
such as SIP registrations.



Some possible uses:

* Global shared blacklists

[globals]

SHARED_DB(put,global)=blacklist

[default]

exten => add_to_blacklist,1,NoOp()
; bad caller!
same => Set(DB(blacklist/${CALLERID(name)})=)

* Shared SIP registries

[globals]

SHARED_DB(put,unique)=registrar


Or, really, just share any data you want between Asterisk servers. I
suspect there's a lot of interesting ways to use this.

After a bit of tweaking on the return flight, I'm fairly sure the
concept works [1]. Before I go much further on it (that is, make it
actually usable, clean up CLI commands, write a *lot* of tests, finish
up the PJSIP/Corosync/etc. implementations, etc.) - does this sound
like something that would be generally useful? Are the mechanisms for
marking a family as being shared logical? Is there a better way to
handle the different use cases for shared data?

[1] http://svn.asterisk.org/svn/asterisk/team/mjordan/trunk-astdb-cluster/

-- 
Matthew Jordan
Digium, Inc. | Director of Technology
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org