<div dir="ltr">Hey Matt,<div><br></div><div> One of the subjects I normally tackle first, when building a federated system is the data store.</div><div>Normally, I would use something like Redis and utilize its PUB/SUB bus to propagate information<br>across nodes. </div><div><br></div><div> Having a similar mechanism with Asterisk seems like a good idea, would surely reduce much <br>pain with large scale systems. Having said that, here are a couple of things that I would love to <br>see in such a mechanism:</div><div><br></div><div>1. Data Sharding - While propagating information across the bus to the other Asterisk servers is<br> one thing, it will also be useful to make the information shardable. For example, information like<br> settings I'd love to share across nodes, however, information like SIP registrations I'd love to shard.</div><div> Why shard SIP registrations? simple, I'd love to be able to track the bouncing of registrations <br> across the network, and maybe in the future, enable a Kamailio server to bounce your sessions <br> from one server to the other, without the need to fully re-register or something like that (just a thought).</div><div><br></div><div>2. Presets - Thinking from an integrator/developer point of view, I think it would be wise to add a <br> astdb_cluster.conf file. The idea is to declare specific settings and astdb families and automaticaly<br> shared. For example, imagine the following file:</div><div><br></div><div>[general]</div><div>cluster_name = cluster.voip ; must be identical across all nodes</div><div>cluster_node = 1</div><div>cluster_secret = some.super.secret.key</div><div><br></div><div>[family1]</div><div>sharing_policy = multicast</div><div>sharing_nodes = 2,3,4,5</div><div>sync_interval = 500 ; in ms</div><div>sync_mode = ro</div><div><br></div><div>[family2]</div><div>sharing_policy = broadcast</div><div>sync_interval = 5000 ; in ms</div><div>sync_mode = rw</div><div><br></div><div> The idea is the following: some information requires rapid sync, for example, state information within a <br>queue. Other information may require slow sync - replication of registrations between nodes, or even propegation<br>of extension settings between nodes. The idea is "define it and forget about it" - the minute the sharing is <br>defined, the astdb clustering tool will take care of the rest. The "sync_mode" will set if the node is allowed to <br>update values in the astdb family. For example, some nodes may serve as hot-backups, others may serve as <br>active-active clusters.</div><div><br></div><div> One thing that I see a potential issue with is astdb locking. Mainly, how do we maintain astdb persistence <br>across all nodes. AstDB is one of those things that if it fails or becomes corrupted, your entire Asterisk becomes <br>fairly unusable (ala FreePBX style). </div><div><br></div><div> Another issue that needs tackling is split-brain. What happens when a cluster becomes split and then converges<br>back. What information is considered consistent and true? how do we go about and make sure that whatever is <br>converged is the right thing?</div><div><br></div><div>Just my 2c on the subject at hand.</div><div><br></div><div>Nir S</div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 9, 2015 at 6:35 PM, Matthew Jordan <span dir="ltr"><<a href="mailto:mjordan@digium.com" target="_blank">mjordan@digium.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey all -<br>
<br>
So, this last week I was travelling, and since flights are always a<br>
good time to write some code, I decided to play around with an idea<br>
that had been sitting in the back of my head for a little bit:<br>
clustering the Asterisk Database. I got the notion after looking at<br>
Kamailio's htable implementation, and the clustering that it can do<br>
using SIP DMQ messages. A similar mechanism in Asterisk felt rather<br>
doable, since we have:<br>
(1) the ability to pub/sub internally the state of some object<br>
(2) various clustering engines that make use of that internal pub/sub bus<br>
(3) the ability to PUBLISH state between Asterisk instances<br>
<br>
I initially debated whether it was appropriate to do in the AstDB or<br>
in Sorcery (which would be a more generic way of clustering<br>
information). In the end, I decided on the AstDB, for a few reasons:<br>
(1) Local information is already stored in there from a variety of<br>
places, while not everything uses Sorcery<br>
(2) Putting it into Sorcery would make an already complex framework more complex<br>
(3) I suspected it would be easier to 'control' the sharing of<br>
information in the AstDB - which is a specific implementation - than<br>
in a generic DAL like Sorcery<br>
<br>
The actual implementation I went with works as follows:<br>
<br>
* Any family in the AstDB can be marked as "shared" using a new<br>
function - DB_SHARED().<br>
- Sharing family 'foo': DB_SHARED(put,global)=foo<br>
- Deleting the shared status of 'foo': DB_SHARED(delete)=foo<br>
<br>
* The 'shared' status of a database family is independent of the<br>
actual data stored in the AstDB. Thus you can change whether or not a<br>
family is shared without impacting the underlying data.<br>
<br>
* Updating/deleting a key in a shared family causes a Stasis message<br>
to be fired off to consumers letting them know that the key was<br>
updated/deleted. The consumers choose how to send that information to<br>
other Asterisk instances in a cluster. Non-shared families are not<br>
shared (which seems obvious...) - this minimizes the traffic on Stasis<br>
to only the entries that should be propagated.<br>
<br>
* Families in the AstDB can be shared in one of two ways - 'global' or 'unique'.<br>
- A 'global' shared family shares the family/key space with all<br>
other Asterisk instances. For example, if in a local Asterisk instance<br>
family 'foo' with key 'bar' has value '1', then a separate Asterisk<br>
instance that publishes state for 'foo/bar' with value '2' will<br>
overwrite the local copy of 'foo/bar'. This is useful when you want<br>
the same data replicated across all Asterisk instances.<br>
- A 'unique' shared family replicates its state across other<br>
Asterisk instances, but does *not* overwrite the local copy of the<br>
families. Instead, remote versions of a particular family/key are<br>
placed into a new family on the local Asterisk instance, determined by<br>
the EID of the remote Asterisk instance. Using the same example, if in<br>
a local Asterisk instance (with EID 11:11) the family 'foo' is shared<br>
with key 'bar' and value '1', and a remote Asterisk instance (with EID<br>
ff:ff) publishes state for 'foo/bar' with value '2', then in the local<br>
Asterisk instance this does not overwrite the local copy. Instead, the<br>
remote key/value pair is stored in 'ff:ff/foo/bar' with value '2'.<br>
Likewise, since the local family 'foo' is shared 'uniquely, on the<br>
remote instance, the local copy is stored in '11:11/foo/bar' with<br>
value '1'. This is useful for families that may have conflicting data<br>
keys that need additional context dependent processing to aggregate,<br>
such as SIP registrations.<br>
<br>
<br>
<br>
Some possible uses:<br>
<br>
* Global shared blacklists<br>
<br>
[globals]<br>
<br>
SHARED_DB(put,global)=blacklist<br>
<br>
[default]<br>
<br>
exten => add_to_blacklist,1,NoOp()<br>
; bad caller!<br>
same => Set(DB(blacklist/${CALLERID(name)})=)<br>
<br>
* Shared SIP registries<br>
<br>
[globals]<br>
<br>
SHARED_DB(put,unique)=registrar<br>
<br>
<br>
Or, really, just share any data you want between Asterisk servers. I<br>
suspect there's a lot of interesting ways to use this.<br>
<br>
After a bit of tweaking on the return flight, I'm fairly sure the<br>
concept works [1]. Before I go much further on it (that is, make it<br>
actually usable, clean up CLI commands, write a *lot* of tests, finish<br>
up the PJSIP/Corosync/etc. implementations, etc.) - does this sound<br>
like something that would be generally useful? Are the mechanisms for<br>
marking a family as being shared logical? Is there a better way to<br>
handle the different use cases for shared data?<br>
<br>
[1] <a href="http://svn.asterisk.org/svn/asterisk/team/mjordan/trunk-astdb-cluster/" target="_blank">http://svn.asterisk.org/svn/asterisk/team/mjordan/trunk-astdb-cluster/</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Matthew Jordan<br>
Digium, Inc. | Director of Technology<br>
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA<br>
Check us out at: <a href="http://digium.com" target="_blank">http://digium.com</a> & <a href="http://asterisk.org" target="_blank">http://asterisk.org</a><br>
<br>
--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>
</font></span></blockquote></div><br></div></div>