[asterisk-dev] Clustering the Asterisk Database (for fun and profit)

[Nir Simionovich]

Hey Matt,

  One of the subjects I normally tackle first, when building a federated
system is the data store.
Normally, I would use something like Redis and utilize its PUB/SUB bus to
propagate information
across nodes.

  Having a similar mechanism with Asterisk seems like a good idea, would
surely reduce much
pain with large scale systems. Having said that, here are a couple of
things that I would love to
see in such a mechanism:

1. Data Sharding - While propagating information across the bus to the
other Asterisk servers is
    one thing, it will also be useful to make the information shardable.
For example, information like
    settings I'd love to share across nodes, however, information like SIP
registrations I'd love to shard.
    Why shard SIP registrations? simple, I'd love to be able to track the
bouncing of registrations
    across the network, and maybe in the future, enable a Kamailio server
to bounce your sessions
    from one server to the other, without the need to fully re-register or
something like that (just a thought).

2. Presets - Thinking from an integrator/developer point of view, I think
it would be wise to add a
    astdb_cluster.conf file. The idea is to declare specific settings and
astdb families and automaticaly
    shared. For example, imagine the following file:

[general]
cluster_name = cluster.voip    ; must be identical across all nodes
cluster_node = 1
cluster_secret = some.super.secret.key

[family1]
sharing_policy = multicast
sharing_nodes = 2,3,4,5
sync_interval = 500 ; in ms
sync_mode = ro

[family2]
sharing_policy = broadcast
sync_interval = 5000 ; in ms
sync_mode = rw

  The idea is the following: some information requires rapid sync, for
example, state information within a
queue. Other information may require slow sync - replication of
registrations between nodes, or even propegation
of extension settings between nodes. The idea is "define it and forget
about it" - the minute the sharing is
defined, the astdb clustering tool will take care of the rest. The
"sync_mode" will set if the node is allowed to
update values in the astdb family. For example, some nodes may serve as
hot-backups, others may serve as
active-active clusters.

  One thing that I see a potential issue with is astdb locking. Mainly, how
do we maintain astdb persistence
across all nodes. AstDB is one of those things that if it fails or becomes
corrupted, your entire Asterisk becomes
fairly unusable (ala FreePBX style).

  Another issue that needs tackling is split-brain. What happens when a
cluster becomes split and then converges
back. What information is considered consistent and true? how do we go
about and make sure that whatever is
converged is the right thing?

Just my 2c on the subject at hand.

Nir S




On Mon, Mar 9, 2015 at 6:35 PM, Matthew Jordan <mjordan at digium.com> wrote:

> Hey all -
>
> So, this last week I was travelling, and since flights are always a
> good time to write some code, I decided to play around with an idea
> that had been sitting in the back of my head for a little bit:
> clustering the Asterisk Database. I got the notion after looking at
> Kamailio's htable implementation, and the clustering that it can do
> using SIP DMQ messages. A similar mechanism in Asterisk felt rather
> doable, since we have:
> (1) the ability to pub/sub internally the state of some object
> (2) various clustering engines that make use of that internal pub/sub bus
> (3) the ability to PUBLISH state between Asterisk instances
>
> I initially debated whether it was appropriate to do in the AstDB or
> in Sorcery (which would be a more generic way of clustering
> information). In the end, I decided on the AstDB, for a few reasons:
> (1) Local information is already stored in there from a variety of
> places, while not everything uses Sorcery
> (2) Putting it into Sorcery would make an already complex framework more
> complex
> (3) I suspected it would be easier to 'control' the sharing of
> information in the AstDB - which is a specific implementation - than
> in a generic DAL like Sorcery
>
> The actual implementation I went with works as follows:
>
> * Any family in the AstDB can be marked as "shared" using a new
> function - DB_SHARED().
>   - Sharing family 'foo': DB_SHARED(put,global)=foo
>   - Deleting the shared status of 'foo': DB_SHARED(delete)=foo
>
> * The 'shared' status of a database family is independent of the
> actual data stored in the AstDB. Thus you can change whether or not a
> family is shared without impacting the underlying data.
>
> * Updating/deleting a key in a shared family causes a Stasis message
> to be fired off to consumers letting them know that the key was
> updated/deleted. The consumers choose how to send that information to
> other Asterisk instances in a cluster. Non-shared families are not
> shared (which seems obvious...) - this minimizes the traffic on Stasis
> to only the entries that should be propagated.
>
> * Families in the AstDB can be shared in one of two ways - 'global' or
> 'unique'.
>   - A 'global' shared family shares the family/key space with all
> other Asterisk instances. For example, if in a local Asterisk instance
> family 'foo' with key 'bar' has value '1', then a separate Asterisk
> instance that publishes state for 'foo/bar' with value '2' will
> overwrite the local copy of 'foo/bar'. This is useful when you want
> the same data replicated across all Asterisk instances.
>   - A 'unique' shared family replicates its state across other
> Asterisk instances, but does *not* overwrite the local copy of the
> families. Instead, remote versions of a particular family/key are
> placed into a new family on the local Asterisk instance, determined by
> the EID of the remote Asterisk instance. Using the same example, if in
> a local Asterisk instance (with EID 11:11) the family 'foo' is shared
> with key 'bar' and value '1', and a remote Asterisk instance (with EID
> ff:ff) publishes state for 'foo/bar' with value '2', then in the local
> Asterisk instance this does not overwrite the local copy. Instead, the
> remote key/value pair is stored in 'ff:ff/foo/bar' with value '2'.
> Likewise, since the local family 'foo' is shared 'uniquely, on the
> remote instance, the local copy is stored in '11:11/foo/bar' with
> value '1'. This is useful for families that may have conflicting data
> keys that need additional context dependent processing to aggregate,
> such as SIP registrations.
>
>
>
> Some possible uses:
>
> * Global shared blacklists
>
> [globals]
>
> SHARED_DB(put,global)=blacklist
>
> [default]
>
> exten => add_to_blacklist,1,NoOp()
> ; bad caller!
> same => Set(DB(blacklist/${CALLERID(name)})=)
>
> * Shared SIP registries
>
> [globals]
>
> SHARED_DB(put,unique)=registrar
>
>
> Or, really, just share any data you want between Asterisk servers. I
> suspect there's a lot of interesting ways to use this.
>
> After a bit of tweaking on the return flight, I'm fairly sure the
> concept works [1]. Before I go much further on it (that is, make it
> actually usable, clean up CLI commands, write a *lot* of tests, finish
> up the PJSIP/Corosync/etc. implementations, etc.) - does this sound
> like something that would be generally useful? Are the mechanisms for
> marking a family as being shared logical? Is there a better way to
> handle the different use cases for shared data?
>
> [1] http://svn.asterisk.org/svn/asterisk/team/mjordan/trunk-astdb-cluster/
>
> --
> Matthew Jordan
> Digium, Inc. | Director of Technology
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150315/10407cc9/attachment-0001.html>