<p>Università di Bologna - CESIA VoIP has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.asterisk.org/c/asterisk/+/14455">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">res_corosync: Fix crash in huge distributed environment.<br><br>1) Fix memory-leaks<br>   Added code to release ast_events extracted from corosync and stasis messages<br><br>2) Clean stasis cache when a member of the corosync cluster leaves the group<br>   Added code to remove from the stasis cache of the members remained on the<br>   group all the messages with the EID of the left member.<br>   If the device states of the left member remain in the stasis cache of other<br>   members, they will not be updated anymore and high priority cached values,<br>   like BUSY, will take precedence over current device states.<br><br>3) Stop corosync event propagation when node is not joined to the group<br>   Updated dispatch_thread_handler code to detect when asterisk is not joined<br>   to the corosync group and added some condition in publish_event_to_corosync<br>   code to send corosync messages only when joined.<br>   When a node is not joined its corosync daemon can't send messages:<br>   the cpg_mcast_joined function append new messages to the FIFO buffer until<br>   it's full and then it blocks indefinitely.<br>   In this scenario if the stasis_message_cb callback, registered by<br>   res_corosync to handle stasis messages, try to send a corosync messages,<br>   the thread of the stasis thread-pool will be blocked until the node join<br>   the corosync cluster.<br><br>ASTERISK-28888<br>Reported by: Università di Bologna - CESIA VoIP<br><br>Change-Id: Ie8e99bc23f141a73c13ae6fb1948d148d4de17f2<br>---<br>M res/res_corosync.c<br>1 file changed, 533 insertions(+), 275 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.asterisk.org:29418/asterisk refs/changes/55/14455/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/res/res_corosync.c b/res/res_corosync.c</span><br><span>index 4e96bff..04b13c9 100644</span><br><span>--- a/res/res_corosync.c</span><br><span>+++ b/res/res_corosync.c</span><br><span>@@ -48,13 +48,21 @@</span><br><span> #include "asterisk/stasis.h"</span><br><span> #include "asterisk/stasis_message_router.h"</span><br><span> #include "asterisk/stasis_system.h"</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/taskprocessor.h"</span><br><span> </span><br><span> AST_RWLOCK_DEFINE_STATIC(event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+AST_RWLOCK_DEFINE_STATIC(init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*! \brief Timeout for Corosync's poll process */</span><br><span style="color: hsl(120, 100%, 40%);">+#define COROSYNC_POLL_TIMEOUT                         (10 * 1000)</span><br><span> </span><br><span> static void publish_mwi_to_stasis(struct ast_event *event);</span><br><span> static void publish_device_state_to_stasis(struct ast_event *event);</span><br><span> static void publish_cluster_discovery_to_stasis(struct ast_event *event);</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+/*! \brief Join to corosync */</span><br><span style="color: hsl(120, 100%, 40%);">+static int corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> /*! \brief All the nodes that we're aware of */</span><br><span> static struct ao2_container *nodes;</span><br><span> </span><br><span>@@ -97,9 +105,11 @@</span><br><span>              return NULL;</span><br><span>         }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   memcpy(&node->eid, (struct ast_eid *)ast_event_get_ie_raw(event, AST_EVENT_IE_EID), sizeof(node->eid));</span><br><span style="color: hsl(120, 100%, 40%);">+     memcpy(&node->eid, (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID),</span><br><span style="color: hsl(120, 100%, 40%);">+              sizeof(node->eid));</span><br><span>    node->id = ast_event_get_ie_uint(event, AST_EVENT_IE_NODE_ID);</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_sockaddr_parse(&node->addr, ast_event_get_ie_str(event, AST_EVENT_IE_LOCAL_ADDR), PARSE_PORT_IGNORE);</span><br><span style="color: hsl(120, 100%, 40%);">+      ast_sockaddr_parse(&node->addr, ast_event_get_ie_str(event, AST_EVENT_IE_LOCAL_ADDR),</span><br><span style="color: hsl(120, 100%, 40%);">+                                     PARSE_PORT_IGNORE);</span><br><span> </span><br><span>   return node;</span><br><span> }</span><br><span>@@ -182,23 +192,24 @@</span><br><span>            return NULL;</span><br><span>         }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   event_eid = (struct ast_eid *)ast_event_get_ie_raw(payload->event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+      event_eid = (struct ast_eid *) ast_event_get_ie_raw(payload->event, AST_EVENT_IE_EID);</span><br><span> </span><br><span>        event = ast_event_new(AST_EVENT_PING,</span><br><span style="color: hsl(0, 100%, 40%);">-                           AST_EVENT_IE_EID, AST_EVENT_IE_PLTYPE_RAW, event_eid, sizeof(*event_eid),</span><br><span style="color: hsl(0, 100%, 40%);">-                               AST_EVENT_IE_END);</span><br><span style="color: hsl(120, 100%, 40%);">+                                              AST_EVENT_IE_EID, AST_EVENT_IE_PLTYPE_RAW, event_eid,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                 sizeof(*event_eid), AST_EVENT_IE_END);</span><br><span> </span><br><span>         return event;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-STASIS_MESSAGE_TYPE_DEFN_LOCAL(corosync_ping_message_type,</span><br><span style="color: hsl(0, 100%, 40%);">-        .to_event = corosync_ping_to_event, );</span><br><span style="color: hsl(120, 100%, 40%);">+STASIS_MESSAGE_TYPE_DEFN_LOCAL(corosync_ping_message_type,.to_event =</span><br><span style="color: hsl(120, 100%, 40%);">+                                                    corosync_ping_to_event,);</span><br><span> </span><br><span> /*! \brief Publish a Corosync ping to \ref stasis */</span><br><span> static void publish_corosync_ping_to_stasis(struct ast_event *event)</span><br><span> {</span><br><span>        struct corosync_ping_payload *payload;</span><br><span>       struct stasis_message *message;</span><br><span style="color: hsl(120, 100%, 40%);">+       struct ast_eid *event_eid;</span><br><span> </span><br><span>       ast_assert(ast_event_get_type(event) == AST_EVENT_PING);</span><br><span>     ast_assert(event != NULL);</span><br><span>@@ -207,11 +218,15 @@</span><br><span>           return;</span><br><span>      }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   payload = ao2_t_alloc(sizeof(*payload), corosync_ping_payload_dtor, "Create ping payload");</span><br><span style="color: hsl(120, 100%, 40%);">+ payload =</span><br><span style="color: hsl(120, 100%, 40%);">+             ao2_t_alloc(sizeof(*payload), corosync_ping_payload_dtor, "Create ping payload");</span><br><span>  if (!payload) {</span><br><span>              return;</span><br><span>      }</span><br><span style="color: hsl(0, 100%, 40%);">-       payload->event = event;</span><br><span style="color: hsl(120, 100%, 40%);">+    event_eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+ payload->event = ast_event_new(AST_EVENT_PING,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                AST_EVENT_IE_EID, AST_EVENT_IE_PLTYPE_RAW, event_eid,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                 sizeof(*event_eid), AST_EVENT_IE_END);</span><br><span> </span><br><span>        message = stasis_message_create(corosync_ping_message_type(), payload);</span><br><span>      if (!message) {</span><br><span>@@ -232,33 +247,35 @@</span><br><span>      unsigned char publish_default;</span><br><span>       unsigned char subscribe;</span><br><span>     unsigned char subscribe_default;</span><br><span style="color: hsl(0, 100%, 40%);">-        struct stasis_topic *(* topic_fn)(void);</span><br><span style="color: hsl(0, 100%, 40%);">-        struct stasis_cache *(* cache_fn)(void);</span><br><span style="color: hsl(0, 100%, 40%);">-        struct stasis_message_type *(* message_type_fn)(void);</span><br><span style="color: hsl(0, 100%, 40%);">-  void (* publish_to_stasis)(struct ast_event *);</span><br><span style="color: hsl(120, 100%, 40%);">+       struct stasis_topic *(*topic_fn) (void);</span><br><span style="color: hsl(120, 100%, 40%);">+      struct stasis_cache *(*cache_fn) (void);</span><br><span style="color: hsl(120, 100%, 40%);">+      struct stasis_message_type *(*message_type_fn) (void);</span><br><span style="color: hsl(120, 100%, 40%);">+        void (*publish_to_stasis)(struct ast_event *);</span><br><span> } event_types[] = {</span><br><span style="color: hsl(0, 100%, 40%);">-   [AST_EVENT_MWI] = { .name = "mwi",</span><br><span style="color: hsl(0, 100%, 40%);">-                        .topic_fn = ast_mwi_topic_all,</span><br><span style="color: hsl(0, 100%, 40%);">-                          .cache_fn = ast_mwi_state_cache,</span><br><span style="color: hsl(0, 100%, 40%);">-                        .message_type_fn = ast_mwi_state_type,</span><br><span style="color: hsl(0, 100%, 40%);">-                          .publish_to_stasis = publish_mwi_to_stasis, },</span><br><span style="color: hsl(0, 100%, 40%);">-      [AST_EVENT_DEVICE_STATE_CHANGE] = { .name = "device_state",</span><br><span style="color: hsl(0, 100%, 40%);">-                                       .topic_fn = ast_device_state_topic_all,</span><br><span style="color: hsl(0, 100%, 40%);">-                                         .cache_fn = ast_device_state_cache,</span><br><span style="color: hsl(0, 100%, 40%);">-                                     .message_type_fn = ast_device_state_message_type,</span><br><span style="color: hsl(0, 100%, 40%);">-                                       .publish_to_stasis = publish_device_state_to_stasis, },</span><br><span style="color: hsl(0, 100%, 40%);">-     [AST_EVENT_PING] = { .name = "ping",</span><br><span style="color: hsl(0, 100%, 40%);">-                       .publish_default = 1,</span><br><span style="color: hsl(0, 100%, 40%);">-                           .subscribe_default = 1,</span><br><span style="color: hsl(0, 100%, 40%);">-                         .topic_fn = corosync_topic,</span><br><span style="color: hsl(0, 100%, 40%);">-                             .message_type_fn = corosync_ping_message_type,</span><br><span style="color: hsl(0, 100%, 40%);">-                          .publish_to_stasis = publish_corosync_ping_to_stasis, },</span><br><span style="color: hsl(0, 100%, 40%);">-   [AST_EVENT_CLUSTER_DISCOVERY] = { .name = "cluster_discovery",</span><br><span style="color: hsl(0, 100%, 40%);">-                                          .publish_default = 1,</span><br><span style="color: hsl(0, 100%, 40%);">-                                   .subscribe_default = 1,</span><br><span style="color: hsl(0, 100%, 40%);">-                                         .topic_fn = ast_system_topic,</span><br><span style="color: hsl(0, 100%, 40%);">-                                   .message_type_fn = ast_cluster_discovery_type,</span><br><span style="color: hsl(0, 100%, 40%);">-                                          .publish_to_stasis = publish_cluster_discovery_to_stasis, },</span><br><span style="color: hsl(120, 100%, 40%);">+        [AST_EVENT_MWI] = {.name = "mwi",</span><br><span style="color: hsl(120, 100%, 40%);">+                                      .topic_fn = ast_mwi_topic_all,</span><br><span style="color: hsl(120, 100%, 40%);">+                                        .cache_fn = ast_mwi_state_cache,</span><br><span style="color: hsl(120, 100%, 40%);">+                                      .message_type_fn = ast_mwi_state_type,</span><br><span style="color: hsl(120, 100%, 40%);">+                                        .publish_to_stasis = publish_mwi_to_stasis,},</span><br><span style="color: hsl(120, 100%, 40%);">+      [AST_EVENT_DEVICE_STATE_CHANGE] = {.name = "device_state",</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                     .topic_fn = ast_device_state_topic_all,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                       .cache_fn = ast_device_state_cache,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                           .message_type_fn = ast_device_state_message_type,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                     .publish_to_stasis =</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                          publish_device_state_to_stasis,},</span><br><span style="color: hsl(120, 100%, 40%);">+  [AST_EVENT_PING] = {.name = "ping",</span><br><span style="color: hsl(120, 100%, 40%);">+                                         .publish_default = 1,</span><br><span style="color: hsl(120, 100%, 40%);">+                                         .subscribe_default = 1,</span><br><span style="color: hsl(120, 100%, 40%);">+                                               .topic_fn = corosync_topic,</span><br><span style="color: hsl(120, 100%, 40%);">+                                           .message_type_fn = corosync_ping_message_type,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                .publish_to_stasis = publish_corosync_ping_to_stasis,},</span><br><span style="color: hsl(120, 100%, 40%);">+       [AST_EVENT_CLUSTER_DISCOVERY] = {.name = "cluster_discovery",</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                        .publish_default = 1,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                         .subscribe_default = 1,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                       .topic_fn = ast_system_topic,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                         .message_type_fn = ast_cluster_discovery_type,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                        .publish_to_stasis =</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                  publish_cluster_discovery_to_stasis,},</span><br><span> };</span><br><span> </span><br><span> static struct {</span><br><span>@@ -267,20 +284,19 @@</span><br><span>         unsigned int stop:1;</span><br><span> } dispatch_thread = {</span><br><span>        .id = AST_PTHREADT_NULL,</span><br><span style="color: hsl(0, 100%, 40%);">-        .alert_pipe = { -1, -1 },</span><br><span style="color: hsl(120, 100%, 40%);">+     .alert_pipe = {-1, -1},</span><br><span> };</span><br><span> </span><br><span> static cpg_handle_t cpg_handle;</span><br><span> static corosync_cfg_handle_t cfg_handle;</span><br><span> </span><br><span> #ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(0, 100%, 40%);">-static void cfg_state_track_cb(</span><br><span style="color: hsl(0, 100%, 40%);">-             corosync_cfg_state_notification_buffer_t *notification_buffer,</span><br><span style="color: hsl(0, 100%, 40%);">-          cs_error_t error);</span><br><span style="color: hsl(120, 100%, 40%);">+static void cfg_state_track_cb(corosync_cfg_state_notification_buffer_t *</span><br><span style="color: hsl(120, 100%, 40%);">+                                                    notification_buffer, cs_error_t error);</span><br><span> #endif /* HAVE_COROSYNC_CFG_STATE_TRACK */</span><br><span> </span><br><span> static void cfg_shutdown_cb(corosync_cfg_handle_t cfg_handle,</span><br><span style="color: hsl(0, 100%, 40%);">-           corosync_cfg_shutdown_flags_t flags);</span><br><span style="color: hsl(120, 100%, 40%);">+                                                 corosync_cfg_shutdown_flags_t flags);</span><br><span> </span><br><span> static corosync_cfg_callbacks_t cfg_callbacks = {</span><br><span> #ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span>@@ -290,7 +306,8 @@</span><br><span> };</span><br><span> </span><br><span> /*! \brief Publish cluster discovery to \ref stasis */</span><br><span style="color: hsl(0, 100%, 40%);">-static void publish_cluster_discovery_to_stasis_full(struct corosync_node *node, int joined)</span><br><span style="color: hsl(120, 100%, 40%);">+static void publish_cluster_discovery_to_stasis_full(struct corosync_node *node,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                                         int joined)</span><br><span> {</span><br><span>    struct ast_json *json;</span><br><span>       struct ast_json_payload *payload;</span><br><span>@@ -302,16 +319,11 @@</span><br><span>    addr = ast_sockaddr_stringify_addr(&node->addr);</span><br><span> </span><br><span>  ast_log(AST_LOG_NOTICE, "Node %u (%s) at %s %s the cluster\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                node->id,</span><br><span style="color: hsl(0, 100%, 40%);">-            eid,</span><br><span style="color: hsl(0, 100%, 40%);">-            addr,</span><br><span style="color: hsl(0, 100%, 40%);">-           joined ? "joined" : "left");</span><br><span style="color: hsl(120, 100%, 40%);">+                      node->id, eid, addr, joined ? "joined" : "left");</span><br><span> </span><br><span>         json = ast_json_pack("{s: s, s: i, s: s, s: i}",</span><br><span style="color: hsl(0, 100%, 40%);">-              "address", addr,</span><br><span style="color: hsl(0, 100%, 40%);">-              "node_id", node->id,</span><br><span style="color: hsl(0, 100%, 40%);">-               "eid", eid,</span><br><span style="color: hsl(0, 100%, 40%);">-           "joined", joined);</span><br><span style="color: hsl(120, 100%, 40%);">+                                           "address", addr,</span><br><span style="color: hsl(120, 100%, 40%);">+                                            "node_id", node->id, "eid", eid, "joined", joined);</span><br><span>        if (!json) {</span><br><span>                 return;</span><br><span>      }</span><br><span>@@ -346,8 +358,8 @@</span><br><span> </span><br><span>  ast_assert(ast_event_get_type(event) == AST_EVENT_CLUSTER_DISCOVERY);</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-       event_eid = (struct ast_eid *)ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(0, 100%, 40%);">-    if (!ast_eid_cmp(&ast_eid_default, event_eid)) {</span><br><span style="color: hsl(120, 100%, 40%);">+  event_eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+ if (!event_eid || !ast_eid_cmp(&ast_eid_default, event_eid)) {</span><br><span>           /* Don't feed events back in that originated locally. */</span><br><span>                 return;</span><br><span>      }</span><br><span>@@ -395,7 +407,7 @@</span><br><span>      context = ast_event_get_ie_str(event, AST_EVENT_IE_CONTEXT);</span><br><span>         new_msgs = ast_event_get_ie_uint(event, AST_EVENT_IE_NEWMSGS);</span><br><span>       old_msgs = ast_event_get_ie_uint(event, AST_EVENT_IE_OLDMSGS);</span><br><span style="color: hsl(0, 100%, 40%);">-  event_eid = (struct ast_eid *)ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+  event_eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span> </span><br><span>    if (ast_strlen_zero(mailbox) || ast_strlen_zero(context)) {</span><br><span>          return;</span><br><span>@@ -409,12 +421,12 @@</span><br><span>              old_msgs = INT_MAX;</span><br><span>  }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if (ast_publish_mwi_state_full(mailbox, context, (int)new_msgs,</span><br><span style="color: hsl(0, 100%, 40%);">-                                (int)old_msgs, NULL, event_eid)) {</span><br><span style="color: hsl(120, 100%, 40%);">+     if (ast_publish_mwi_state_full(mailbox, context, (int) new_msgs,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                 (int) old_msgs, NULL, event_eid)) {</span><br><span>               char eid[18];</span><br><span>                ast_eid_to_str(eid, sizeof(eid), event_eid);</span><br><span>                 ast_log(LOG_WARNING, "Failed to publish MWI message for %s@%s from %s\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                     mailbox, context, eid);</span><br><span style="color: hsl(120, 100%, 40%);">+                               mailbox, context, eid);</span><br><span>      }</span><br><span> }</span><br><span> </span><br><span>@@ -431,7 +443,7 @@</span><br><span>     device = ast_event_get_ie_str(event, AST_EVENT_IE_DEVICE);</span><br><span>   state = ast_event_get_ie_uint(event, AST_EVENT_IE_STATE);</span><br><span>    cachable = ast_event_get_ie_uint(event, AST_EVENT_IE_CACHABLE);</span><br><span style="color: hsl(0, 100%, 40%);">- event_eid = (struct ast_eid *)ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+  event_eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span> </span><br><span>    if (ast_strlen_zero(device)) {</span><br><span>               return;</span><br><span>@@ -441,17 +453,19 @@</span><br><span>              char eid[18];</span><br><span>                ast_eid_to_str(eid, sizeof(eid), event_eid);</span><br><span>                 ast_log(LOG_WARNING, "Failed to publish device state message for %s from %s\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                       device, eid);</span><br><span style="color: hsl(120, 100%, 40%);">+                         device, eid);</span><br><span>        }</span><br><span> }</span><br><span> </span><br><span> static void cpg_deliver_cb(cpg_handle_t handle, const struct cpg_name *group_name,</span><br><span style="color: hsl(0, 100%, 40%);">-                uint32_t nodeid, uint32_t pid, void *msg, size_t msg_len);</span><br><span style="color: hsl(120, 100%, 40%);">+                                               uint32_t nodeid, uint32_t pid, void *msg, size_t msg_len);</span><br><span> </span><br><span> static void cpg_confchg_cb(cpg_handle_t handle, const struct cpg_name *group_name,</span><br><span style="color: hsl(0, 100%, 40%);">-         const struct cpg_address *member_list, size_t member_list_entries,</span><br><span style="color: hsl(0, 100%, 40%);">-              const struct cpg_address *left_list, size_t left_list_entries,</span><br><span style="color: hsl(0, 100%, 40%);">-          const struct cpg_address *joined_list, size_t joined_list_entries);</span><br><span style="color: hsl(120, 100%, 40%);">+                                              const struct cpg_address *member_list,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                size_t member_list_entries,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                   const struct cpg_address *left_list, size_t left_list_entries,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                const struct cpg_address *joined_list,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                size_t joined_list_entries);</span><br><span> </span><br><span> static cpg_callbacks_t cpg_callbacks = {</span><br><span>      .cpg_deliver_fn = cpg_deliver_cb,</span><br><span>@@ -459,33 +473,33 @@</span><br><span> };</span><br><span> </span><br><span> #ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(0, 100%, 40%);">-static void cfg_state_track_cb(</span><br><span style="color: hsl(0, 100%, 40%);">-              corosync_cfg_state_notification_buffer_t *notification_buffer,</span><br><span style="color: hsl(0, 100%, 40%);">-          cs_error_t error)</span><br><span style="color: hsl(120, 100%, 40%);">+static void cfg_state_track_cb(corosync_cfg_state_notification_buffer_t *</span><br><span style="color: hsl(120, 100%, 40%);">+                                                     notification_buffer, cs_error_t error)</span><br><span> {</span><br><span> }</span><br><span> #endif /* HAVE_COROSYNC_CFG_STATE_TRACK */</span><br><span> </span><br><span> static void cfg_shutdown_cb(corosync_cfg_handle_t cfg_handle,</span><br><span style="color: hsl(0, 100%, 40%);">-          corosync_cfg_shutdown_flags_t flags)</span><br><span style="color: hsl(120, 100%, 40%);">+                                                  corosync_cfg_shutdown_flags_t flags)</span><br><span> {</span><br><span> }</span><br><span> </span><br><span> static void cpg_deliver_cb(cpg_handle_t handle, const struct cpg_name *group_name,</span><br><span style="color: hsl(0, 100%, 40%);">-                uint32_t nodeid, uint32_t pid, void *msg, size_t msg_len)</span><br><span style="color: hsl(120, 100%, 40%);">+                                                uint32_t nodeid, uint32_t pid, void *msg, size_t msg_len)</span><br><span> {</span><br><span>    struct ast_event *event;</span><br><span>     void (*publish_handler)(struct ast_event *) = NULL;</span><br><span>  enum ast_event_type event_type;</span><br><span style="color: hsl(120, 100%, 40%);">+       struct ast_eid *event_eid;</span><br><span> </span><br><span>       if (msg_len < ast_event_minimum_length()) {</span><br><span>               ast_debug(1, "Ignoring event that's too small. %u < %u\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                     (unsigned int) msg_len,</span><br><span style="color: hsl(0, 100%, 40%);">-                 (unsigned int) ast_event_minimum_length());</span><br><span style="color: hsl(120, 100%, 40%);">+                             (unsigned int) msg_len, (unsigned int) ast_event_minimum_length());</span><br><span>                return;</span><br><span>      }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if (!ast_eid_cmp(&ast_eid_default, ast_event_get_ie_raw(msg, AST_EVENT_IE_EID))) {</span><br><span style="color: hsl(120, 100%, 40%);">+        event_eid = (struct ast_eid *) ast_event_get_ie_raw(msg, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+   if (!event_eid || !ast_eid_cmp(&ast_eid_default, event_eid)) {</span><br><span>           /* Don't feed events back in that originated locally. */</span><br><span>                 return;</span><br><span>      }</span><br><span>@@ -497,14 +511,17 @@</span><br><span>    }</span><br><span> </span><br><span>        ast_rwlock_rdlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "cpg_deliver_cb rdlock\n");</span><br><span>   publish_handler = event_types[event_type].publish_to_stasis;</span><br><span>         if (!event_types[event_type].subscribe || !publish_handler) {</span><br><span>                /* We are not configured to subscribe to these events or</span><br><span>                we have no way to publish it internally. */</span><br><span>               ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_debug(5, "cpg_deliver_cb unlock\n");</span><br><span>           return;</span><br><span>      }</span><br><span>    ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "cpg_deliver_cb unlock\n");</span><br><span> </span><br><span>       if (!(event = ast_malloc(msg_len))) {</span><br><span>                return;</span><br><span>@@ -516,13 +533,14 @@</span><br><span>              const struct ast_eid *eid;</span><br><span>           char buf[128] = "";</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-               eid = ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+          eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span>              ast_eid_to_str(buf, sizeof(buf), (struct ast_eid *) eid);</span><br><span>            ast_log(LOG_NOTICE, "Got event PING from server with EID: '%s'\n", buf);</span><br><span>   }</span><br><span>    ast_debug(5, "Publishing event %s (%u) to stasis\n",</span><br><span style="color: hsl(0, 100%, 40%);">-          ast_event_get_type_name(event), event_type);</span><br><span style="color: hsl(120, 100%, 40%);">+                    ast_event_get_type_name(event), event_type);</span><br><span>       publish_handler(event);</span><br><span style="color: hsl(120, 100%, 40%);">+       ast_free(event);</span><br><span> }</span><br><span> </span><br><span> static void publish_event_to_corosync(struct ast_event *event)</span><br><span>@@ -530,30 +548,41 @@</span><br><span>  cs_error_t cs_err;</span><br><span>   struct iovec iov;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   iov.iov_base = (void *)event;</span><br><span style="color: hsl(120, 100%, 40%);">+ iov.iov_base = (void *) event;</span><br><span>       iov.iov_len = ast_event_get_size(event);</span><br><span> </span><br><span>         ast_debug(5, "Publishing event %s (%u) to corosync\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                ast_event_get_type_name(event), ast_event_get_type(event));</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_event_get_type_name(event), ast_event_get_type(event));</span><br><span> </span><br><span>    /* The stasis subscription will only exist if we are configured to publish</span><br><span>    * these events, so just send away. */</span><br><span style="color: hsl(0, 100%, 40%);">-  if ((cs_err = cpg_mcast_joined(cpg_handle, CPG_TYPE_FIFO, &iov, 1)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-             ast_log(LOG_WARNING, "CPG mcast failed (%u) for event %s (%u)\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                     cs_err, ast_event_get_type_name(event), ast_event_get_type(event));</span><br><span style="color: hsl(120, 100%, 40%);">+   if (corosync_node_joined && !ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_debug(5, "publish_event_to_corosync rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+         if ((cs_err = cpg_mcast_joined(cpg_handle, CPG_TYPE_FIFO, &iov, 1)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                   ast_log(LOG_WARNING, "CPG mcast failed (%u) for event %s (%u)\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                   cs_err, ast_event_get_type_name(event), ast_event_get_type(event));</span><br><span style="color: hsl(120, 100%, 40%);">+           }</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "publish_event_to_corosync unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+ } else {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_log(LOG_WARNING,</span><br><span style="color: hsl(120, 100%, 40%);">+                          "CPG mcast not executed for event %s (%u): initializing CPG.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                            ast_event_get_type_name(event), ast_event_get_type(event));</span><br><span>  }</span><br><span> }</span><br><span> </span><br><span> static void publish_to_corosync(struct stasis_message *message)</span><br><span> {</span><br><span>   struct ast_event *event;</span><br><span style="color: hsl(120, 100%, 40%);">+      struct ast_eid *event_eid;</span><br><span> </span><br><span>       event = stasis_message_to_event(message);</span><br><span>    if (!event) {</span><br><span>                return;</span><br><span>      }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if (ast_eid_cmp(&ast_eid_default, ast_event_get_ie_raw(event, AST_EVENT_IE_EID))) {</span><br><span style="color: hsl(120, 100%, 40%);">+       event_eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+ if (!event_eid || ast_eid_cmp(&ast_eid_default, event_eid)) {</span><br><span>            /* If the event didn't originate from this server, don't send it back out. */</span><br><span>                ast_event_destroy(event);</span><br><span>            return;</span><br><span>@@ -563,15 +592,17 @@</span><br><span>              const struct ast_eid *eid;</span><br><span>           char buf[128] = "";</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-               eid = ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span style="color: hsl(120, 100%, 40%);">+          eid = (struct ast_eid *) ast_event_get_ie_raw(event, AST_EVENT_IE_EID);</span><br><span>              ast_eid_to_str(buf, sizeof(buf), (struct ast_eid *) eid);</span><br><span>            ast_log(LOG_NOTICE, "Sending event PING from this server with EID: '%s'\n", buf);</span><br><span>  }</span><br><span> </span><br><span>        publish_event_to_corosync(event);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_event_destroy(event);</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-static void stasis_message_cb(void *data, struct stasis_subscription *sub, struct stasis_message *message)</span><br><span style="color: hsl(120, 100%, 40%);">+static void stasis_message_cb(void *data, struct stasis_subscription *sub,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                        struct stasis_message *message)</span><br><span> {</span><br><span>       if (!message) {</span><br><span>              return;</span><br><span>@@ -593,23 +624,84 @@</span><br><span>      return 0;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+static int clear_node_cache(void *obj, void *arg, int flags)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+   struct stasis_message *cached_msg = obj;</span><br><span style="color: hsl(120, 100%, 40%);">+      struct stasis_topic *topic = arg;</span><br><span style="color: hsl(120, 100%, 40%);">+     struct stasis_message *msg;</span><br><span style="color: hsl(120, 100%, 40%);">+   struct ast_eid *msg_eid;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+    if (!cached_msg) {</span><br><span style="color: hsl(120, 100%, 40%);">+            return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+     }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+   msg_eid = (struct ast_eid *) stasis_message_eid(cached_msg);</span><br><span style="color: hsl(120, 100%, 40%);">+  if (msg_eid && ast_eid_cmp(&ast_eid_default, msg_eid)) {</span><br><span style="color: hsl(120, 100%, 40%);">+          msg = stasis_cache_clear_create(cached_msg);</span><br><span style="color: hsl(120, 100%, 40%);">+          if (msg) {</span><br><span style="color: hsl(120, 100%, 40%);">+                    stasis_publish(topic, msg);</span><br><span style="color: hsl(120, 100%, 40%);">+                   ao2_cleanup(msg);</span><br><span style="color: hsl(120, 100%, 40%);">+             }</span><br><span style="color: hsl(120, 100%, 40%);">+     }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+   return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> static void cpg_confchg_cb(cpg_handle_t handle, const struct cpg_name *group_name,</span><br><span style="color: hsl(0, 100%, 40%);">-            const struct cpg_address *member_list, size_t member_list_entries,</span><br><span style="color: hsl(0, 100%, 40%);">-              const struct cpg_address *left_list, size_t left_list_entries,</span><br><span style="color: hsl(0, 100%, 40%);">-          const struct cpg_address *joined_list, size_t joined_list_entries)</span><br><span style="color: hsl(120, 100%, 40%);">+                                               const struct cpg_address *member_list,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                size_t member_list_entries,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                   const struct cpg_address *left_list, size_t left_list_entries,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                const struct cpg_address *joined_list,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                size_t joined_list_entries)</span><br><span> {</span><br><span>  unsigned int i;</span><br><span> </span><br><span> </span><br><span>      for (i = 0; i < left_list_entries; i++) {</span><br><span>                 const struct cpg_address *cpg_node = &left_list[i];</span><br><span style="color: hsl(0, 100%, 40%);">-         struct corosync_node* node;</span><br><span style="color: hsl(120, 100%, 40%);">+           struct corosync_node *node;</span><br><span> </span><br><span>              node = ao2_find(nodes, &cpg_node->nodeid, OBJ_UNLINK | OBJ_SEARCH_KEY);</span><br><span>               if (!node) {</span><br><span>                         continue;</span><br><span>            }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+         unsigned int j;</span><br><span style="color: hsl(120, 100%, 40%);">+               for (j = 0; j < ARRAY_LEN(event_types); j++) {</span><br><span style="color: hsl(120, 100%, 40%);">+                     struct ao2_container *messages;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_rwlock_rdlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cpg_confchg_cb rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                    if (!event_types[j].subscribe) {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                             ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                            continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                     }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                   if (!event_types[j].cache_fn || !event_types[j].message_type_fn) {</span><br><span style="color: hsl(120, 100%, 40%);">+                            ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                             ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                            continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                     }</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                  messages =</span><br><span style="color: hsl(120, 100%, 40%);">+                            stasis_cache_dump_by_eid(event_types[j].cache_fn(),</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                            event_types[j].message_type_fn(), &node->eid);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                      int messages_count = ao2_container_count(messages);</span><br><span style="color: hsl(120, 100%, 40%);">+                   ast_log(LOG_NOTICE,</span><br><span style="color: hsl(120, 100%, 40%);">+                                   "Clearing %i events of type %s of node %i from stasis cache.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                    messages_count, event_types[j].name, node->id);</span><br><span style="color: hsl(120, 100%, 40%);">+                    ao2_callback(messages, OBJ_NODATA, clear_node_cache,</span><br><span style="color: hsl(120, 100%, 40%);">+                                           event_types[j].topic_fn());</span><br><span style="color: hsl(120, 100%, 40%);">+                  ast_log(LOG_NOTICE, "Cleared events of type %s from stasis cache.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                       event_types[j].name);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                       ao2_t_ref(messages, -1, "Dispose of flushed cache");</span><br><span style="color: hsl(120, 100%, 40%);">+                }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span>          publish_cluster_discovery_to_stasis_full(node, 0);</span><br><span>           ao2_ref(node, -1);</span><br><span>   }</span><br><span>@@ -624,22 +716,30 @@</span><br><span>            struct ao2_container *messages;</span><br><span> </span><br><span>          ast_rwlock_rdlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_debug(5, "cpg_confchg_cb rdlock\n");</span><br><span>           if (!event_types[i].publish) {</span><br><span>                       ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span>                   continue;</span><br><span>            }</span><br><span> </span><br><span>                if (!event_types[i].cache_fn || !event_types[i].message_type_fn) {</span><br><span>                   ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span>                   continue;</span><br><span>            }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-               messages = stasis_cache_dump_by_eid(event_types[i].cache_fn(),</span><br><span style="color: hsl(0, 100%, 40%);">-                  event_types[i].message_type_fn(),</span><br><span style="color: hsl(0, 100%, 40%);">-                       &ast_eid_default);</span><br><span>               ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_debug(5, "cpg_confchg_cb unlock\n");</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+                messages =</span><br><span style="color: hsl(120, 100%, 40%);">+                    stasis_cache_dump_by_eid(event_types[i].cache_fn(),</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                    event_types[i].message_type_fn(), &ast_eid_default);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           int messages_count = ao2_container_count(messages);</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_log(LOG_NOTICE, "Sending %i events of type %s to corosync.\n", messages_count,</span><br><span style="color: hsl(120, 100%, 40%);">+                          event_types[i].name);</span><br><span>                ao2_callback(messages, OBJ_NODATA, dump_cache_cb, NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_log(LOG_NOTICE, "Sent events of type %s to corosync.\n", event_types[i].name);</span><br><span> </span><br><span>             ao2_t_ref(messages, -1, "Dispose of dumped cache");</span><br><span>        }</span><br><span>@@ -658,53 +758,77 @@</span><br><span>    char buf[128];</span><br><span>       int res;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-    if ((cs_err = corosync_cfg_local_get(cfg_handle, &node_id)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-             ast_log(LOG_WARNING, "Failed to extract Corosync node ID for this node. Not informing cluster of existance.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-              return;</span><br><span style="color: hsl(120, 100%, 40%);">+       if (!ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_debug(5, "send_cluster_notify rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+             if ((cs_err = corosync_cfg_local_get(cfg_handle, &node_id)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                   ast_log(LOG_WARNING,</span><br><span style="color: hsl(120, 100%, 40%);">+                                  "Failed to extract Corosync node ID for this node. Not informing cluster of existance.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                 return;</span><br><span style="color: hsl(120, 100%, 40%);">+               }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           if (((cs_err =</span><br><span style="color: hsl(120, 100%, 40%);">+                          corosync_cfg_get_node_addrs(cfg_handle, node_id, 1, &num_addrs,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                           &corosync_addr)) != CS_OK) || (num_addrs < 1)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_log(LOG_WARNING,</span><br><span style="color: hsl(120, 100%, 40%);">+                                  "Failed to get local Corosync address. Not informing cluster of existance.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                     return;</span><br><span style="color: hsl(120, 100%, 40%);">+               }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "send_cluster_notify unlock\n");</span><br><span>      }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if (((cs_err = corosync_cfg_get_node_addrs(cfg_handle, node_id, 1, &num_addrs, &corosync_addr)) != CS_OK) || (num_addrs < 1)) {</span><br><span style="color: hsl(0, 100%, 40%);">-              ast_log(LOG_WARNING, "Failed to get local Corosync address. Not informing cluster of existance.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-          return;</span><br><span style="color: hsl(0, 100%, 40%);">- }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       sa = (struct sockaddr *)corosync_addr.address;</span><br><span style="color: hsl(0, 100%, 40%);">-  sa_len = (size_t)corosync_addr.address_length;</span><br><span style="color: hsl(120, 100%, 40%);">+        sa = (struct sockaddr *) corosync_addr.address;</span><br><span style="color: hsl(120, 100%, 40%);">+       sa_len = (size_t) corosync_addr.address_length;</span><br><span>      if ((res = getnameinfo(sa, sa_len, buf, sizeof(buf), NULL, 0, NI_NUMERICHOST))) {</span><br><span style="color: hsl(0, 100%, 40%);">-               ast_log(LOG_WARNING, "Failed to determine name of local Corosync address: %s (%d). Not informing cluster of existance.\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                    gai_strerror(res), res);</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_log(LOG_WARNING,</span><br><span style="color: hsl(120, 100%, 40%);">+                          "Failed to determine name of local Corosync address: %s (%d). Not informing cluster of existance.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                               gai_strerror(res), res);</span><br><span>             return;</span><br><span>      }</span><br><span> </span><br><span>        event = ast_event_new(AST_EVENT_CLUSTER_DISCOVERY,</span><br><span style="color: hsl(0, 100%, 40%);">-                                  AST_EVENT_IE_NODE_ID, AST_EVENT_IE_PLTYPE_UINT, node_id,</span><br><span style="color: hsl(0, 100%, 40%);">-                                AST_EVENT_IE_LOCAL_ADDR, AST_EVENT_IE_PLTYPE_STR, buf,</span><br><span style="color: hsl(0, 100%, 40%);">-                                  AST_EVENT_IE_END);</span><br><span style="color: hsl(120, 100%, 40%);">+                                                  AST_EVENT_IE_NODE_ID, AST_EVENT_IE_PLTYPE_UINT, node_id,</span><br><span style="color: hsl(120, 100%, 40%);">+                                              AST_EVENT_IE_LOCAL_ADDR, AST_EVENT_IE_PLTYPE_STR, buf,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                AST_EVENT_IE_END);</span><br><span>         publish_event_to_corosync(event);</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_free(event);</span><br><span style="color: hsl(120, 100%, 40%);">+      ast_event_destroy(event);</span><br><span> }</span><br><span> </span><br><span> static void *dispatch_thread_handler(void *data)</span><br><span> {</span><br><span>  cs_error_t cs_err;</span><br><span>   struct pollfd pfd[3] = {</span><br><span style="color: hsl(0, 100%, 40%);">-                { .events = POLLIN, },</span><br><span style="color: hsl(0, 100%, 40%);">-          { .events = POLLIN, },</span><br><span style="color: hsl(0, 100%, 40%);">-          { .events = POLLIN, },</span><br><span style="color: hsl(120, 100%, 40%);">+                {.events = POLLIN,},</span><br><span style="color: hsl(120, 100%, 40%);">+          {.events = POLLIN,},</span><br><span style="color: hsl(120, 100%, 40%);">+          {.events = POLLIN,},</span><br><span>         };</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-  if ((cs_err = cpg_fd_get(cpg_handle, &pfd[0].fd)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-               ast_log(LOG_ERROR, "Failed to get CPG fd.  This module is now broken.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+  if (!ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_debug(5, "dispatch_thread_handler rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+           if ((cs_err = cpg_fd_get(cpg_handle, &pfd[0].fd)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_log(LOG_ERROR, "Failed to get CPG fd.  This module is now broken.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                  ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                   return NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           if ((cs_err = corosync_cfg_fd_get(cfg_handle, &pfd[1].fd)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                    ast_log(LOG_ERROR, "Failed to get CFG fd.  This module is now broken.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                  ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                   return NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           pfd[2].fd = dispatch_thread.alert_pipe[0];</span><br><span style="color: hsl(120, 100%, 40%);">+            ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+   } else {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_log(LOG_ERROR,</span><br><span style="color: hsl(120, 100%, 40%);">+                            "Failed to get fd: initiliazing CPG.  This module is now broken.\n");</span><br><span>              return NULL;</span><br><span>         }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       if ((cs_err = corosync_cfg_fd_get(cfg_handle, &pfd[1].fd)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-              ast_log(LOG_ERROR, "Failed to get CFG fd.  This module is now broken.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-            return NULL;</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       pfd[2].fd = dispatch_thread.alert_pipe[0];</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>   send_cluster_notify();</span><br><span>       while (!dispatch_thread.stop) {</span><br><span>              int res;</span><br><span>@@ -715,72 +839,156 @@</span><br><span>            pfd[1].revents = 0;</span><br><span>          pfd[2].revents = 0;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-         res = ast_poll(pfd, ARRAY_LEN(pfd), -1);</span><br><span style="color: hsl(120, 100%, 40%);">+              res = ast_poll(pfd, ARRAY_LEN(pfd), COROSYNC_POLL_TIMEOUT);</span><br><span>          if (res == -1 && errno != EINTR && errno != EAGAIN) {</span><br><span>                        ast_log(LOG_ERROR, "poll() error: %s (%d)\n", strerror(errno), errno);</span><br><span style="color: hsl(0, 100%, 40%);">-                        continue;</span><br><span style="color: hsl(0, 100%, 40%);">-               }</span><br><span style="color: hsl(120, 100%, 40%);">+                     cs_err = CS_ERR_BAD_HANDLE;</span><br><span style="color: hsl(120, 100%, 40%);">+           } else if (res == 0) {</span><br><span style="color: hsl(120, 100%, 40%);">+                        unsigned int local_nodeid;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-          if (pfd[0].revents & POLLIN) {</span><br><span style="color: hsl(0, 100%, 40%);">-                      if ((cs_err = cpg_dispatch(cpg_handle, CS_DISPATCH_ALL)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                            ast_log(LOG_WARNING, "Failed CPG dispatch: %u\n", cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                  if (!ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_debug(5, "dispatch_thread_handler rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                           if ((cs_err = cpg_local_get(cpg_handle, &local_nodeid)) == CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                       struct cpg_name name;</span><br><span style="color: hsl(120, 100%, 40%);">+                                 struct cpg_address address[CPG_MEMBERS_MAX];</span><br><span style="color: hsl(120, 100%, 40%);">+                                  int entries = CPG_MEMBERS_MAX;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                                      ast_copy_string(name.value, "asterisk", sizeof(name.value));</span><br><span style="color: hsl(120, 100%, 40%);">+                                        name.length = strlen(name.value);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     if ((cs_err =</span><br><span style="color: hsl(120, 100%, 40%);">+                                          cpg_membership_get(cpg_handle, &name, address,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                   &entries)) == CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                            int i;</span><br><span style="color: hsl(120, 100%, 40%);">+                                                int found = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                                              ast_debug(1, "CPG group has %i node membership\n", entries);</span><br><span style="color: hsl(120, 100%, 40%);">+                                                for (i = 0; (i < entries) && !found; i++) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                                        if (address[i].nodeid == local_nodeid)</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                found = 1;</span><br><span style="color: hsl(120, 100%, 40%);">+                                            }</span><br><span style="color: hsl(120, 100%, 40%);">+                                             if (!found) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                                 ast_log(LOG_WARNING, "Failed to check CPG node membership\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                                      corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                                                     cs_err = CS_ERR_BAD_HANDLE;</span><br><span style="color: hsl(120, 100%, 40%);">+                                           }</span><br><span style="color: hsl(120, 100%, 40%);">+                                     } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                                              ast_log(LOG_WARNING, "Failed to get CPG node membership: %u\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                                             cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                              corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                                             cs_err = CS_ERR_BAD_HANDLE;</span><br><span style="color: hsl(120, 100%, 40%);">+                                   }</span><br><span style="color: hsl(120, 100%, 40%);">+                             } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                                      ast_log(LOG_WARNING, "Failed to get CPG local node id: %u\n", cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                      corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                                     cs_err = CS_ERR_BAD_HANDLE;</span><br><span style="color: hsl(120, 100%, 40%);">+                           }</span><br><span style="color: hsl(120, 100%, 40%);">+                             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                   } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_log(LOG_WARNING,</span><br><span style="color: hsl(120, 100%, 40%);">+                                          "Failed to check CPG node membership: initializing CPG.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                             cs_err = CS_ERR_BAD_HANDLE;</span><br><span style="color: hsl(120, 100%, 40%);">+                   }</span><br><span style="color: hsl(120, 100%, 40%);">+             } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                      if (!ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_debug(5, "dispatch_thread_handler rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                           if (pfd[0].revents & POLLIN) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    if ((cs_err = cpg_dispatch(cpg_handle, CS_DISPATCH_ALL)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                          ast_log(LOG_WARNING, "Failed CPG dispatch: %u\n", cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                  }</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                           if (pfd[1].revents & POLLIN) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    if ((cs_err =</span><br><span style="color: hsl(120, 100%, 40%);">+                                          corosync_cfg_dispatch(cfg_handle, CS_DISPATCH_ALL)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                              ast_log(LOG_WARNING, "Failed CFG dispatch: %u\n", cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                  }</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span style="color: hsl(120, 100%, 40%);">+                             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                   } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_log(LOG_WARNING, "Failed to dispatch: initializing CPG.\n");</span><br><span>                   }</span><br><span>            }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-               if (pfd[1].revents & POLLIN) {</span><br><span style="color: hsl(0, 100%, 40%);">-                      if ((cs_err = corosync_cfg_dispatch(cfg_handle, CS_DISPATCH_ALL)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                           ast_log(LOG_WARNING, "Failed CFG dispatch: %u\n", cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-                    }</span><br><span style="color: hsl(0, 100%, 40%);">-               }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>            if (cs_err == CS_ERR_LIBRARY || cs_err == CS_ERR_BAD_HANDLE) {</span><br><span style="color: hsl(0, 100%, 40%);">-                  struct cpg_name name;</span><br><span> </span><br><span>                    /* If corosync gets restarted out from under Asterisk, try to recover. */</span><br><span> </span><br><span>                        ast_log(LOG_NOTICE, "Attempting to recover from corosync failure.\n");</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                    if ((cs_err = corosync_cfg_initialize(&cfg_handle, &cfg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                         ast_log(LOG_ERROR, "Failed to initialize cfg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-                          sleep(5);</span><br><span style="color: hsl(0, 100%, 40%);">-                               continue;</span><br><span style="color: hsl(0, 100%, 40%);">-                       }</span><br><span style="color: hsl(120, 100%, 40%);">+                     if (!ast_rwlock_trywrlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_debug(5, "dispatch_thread_handler wrlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                           struct cpg_name name;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                       if ((cs_err = cpg_initialize(&cpg_handle, &cpg_callbacks) != CS_OK)) {</span><br><span style="color: hsl(0, 100%, 40%);">-                          ast_log(LOG_ERROR, "Failed to initialize cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-                          sleep(5);</span><br><span style="color: hsl(0, 100%, 40%);">-                               continue;</span><br><span style="color: hsl(0, 100%, 40%);">-                       }</span><br><span style="color: hsl(120, 100%, 40%);">+                             corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                             if (cpg_handle && (cs_err = cpg_finalize(cpg_handle)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                     ast_log(LOG_ERROR, "Failed to finalize cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                          }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                   if ((cs_err = cpg_fd_get(cpg_handle, &pfd[0].fd)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                               ast_log(LOG_ERROR, "Failed to get CPG fd.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-                                sleep(5);</span><br><span style="color: hsl(0, 100%, 40%);">-                               continue;</span><br><span style="color: hsl(0, 100%, 40%);">-                       }</span><br><span style="color: hsl(120, 100%, 40%);">+                             if (cfg_handle && (cs_err = corosync_cfg_finalize(cfg_handle)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    ast_log(LOG_ERROR, "Failed to finalize cfg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                          }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                   if ((cs_err = corosync_cfg_fd_get(cfg_handle, &pfd[1].fd)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                              ast_log(LOG_ERROR, "Failed to get CFG fd.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-                                sleep(5);</span><br><span style="color: hsl(0, 100%, 40%);">-                               continue;</span><br><span style="color: hsl(0, 100%, 40%);">-                       }</span><br><span style="color: hsl(120, 100%, 40%);">+                             if ((cs_err =</span><br><span style="color: hsl(120, 100%, 40%);">+                                  corosync_cfg_initialize(&cfg_handle, &cfg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    ast_log(LOG_ERROR, "Failed to initialize cfg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                   sleep(5);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                   ast_copy_string(name.value, "asterisk", sizeof(name.value));</span><br><span style="color: hsl(0, 100%, 40%);">-                  name.length = strlen(name.value);</span><br><span style="color: hsl(0, 100%, 40%);">-                       if ((cs_err = cpg_join(cpg_handle, &name)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                              ast_log(LOG_ERROR, "Failed to join cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-                                sleep(5);</span><br><span style="color: hsl(0, 100%, 40%);">-                               continue;</span><br><span style="color: hsl(0, 100%, 40%);">-                       }</span><br><span style="color: hsl(120, 100%, 40%);">+                             if ((cs_err = cpg_initialize(&cpg_handle, &cpg_callbacks) != CS_OK)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_log(LOG_ERROR, "Failed to initialize cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                   sleep(5);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                   ast_log(LOG_NOTICE, "Corosync recovery complete.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-                 send_cluster_notify();</span><br><span style="color: hsl(120, 100%, 40%);">+                                if ((cs_err = cpg_fd_get(cpg_handle, &pfd[0].fd)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                     ast_log(LOG_ERROR, "Failed to get CPG fd.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                      ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                   sleep(5);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                           if ((cs_err = corosync_cfg_fd_get(cfg_handle, &pfd[1].fd)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    ast_log(LOG_ERROR, "Failed to get CFG fd.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                      ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                   sleep(5);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                           ast_copy_string(name.value, "asterisk", sizeof(name.value));</span><br><span style="color: hsl(120, 100%, 40%);">+                                name.length = strlen(name.value);</span><br><span style="color: hsl(120, 100%, 40%);">+                             if ((cs_err = cpg_join(cpg_handle, &name)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                    ast_log(LOG_ERROR, "Failed to join cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                                      ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                        ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                                   sleep(5);</span><br><span style="color: hsl(120, 100%, 40%);">+                                     continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                             }</span><br><span style="color: hsl(120, 100%, 40%);">+                             corosync_node_joined = 1;</span><br><span style="color: hsl(120, 100%, 40%);">+                             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                                ast_debug(5, "dispatch_thread_handler unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                           ast_log(LOG_NOTICE, "Corosync recovery complete.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                               send_cluster_notify();</span><br><span style="color: hsl(120, 100%, 40%);">+                        } else {</span><br><span style="color: hsl(120, 100%, 40%);">+                              ast_log(LOG_NOTICE,</span><br><span style="color: hsl(120, 100%, 40%);">+                                           "Failed to recover from corosync failure: initializing CPG.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                    }</span><br><span>            }</span><br><span>    }</span><br><span> </span><br><span>        return NULL;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-static char *corosync_show_members(struct ast_cli_entry *e, int cmd, struct ast_cli_args *a)</span><br><span style="color: hsl(120, 100%, 40%);">+static char *corosync_show_members(struct ast_cli_entry *e, int cmd,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                  struct ast_cli_args *a)</span><br><span> {</span><br><span>      cs_error_t cs_err;</span><br><span>   cpg_iteration_handle_t cpg_iter;</span><br><span>@@ -791,76 +999,83 @@</span><br><span>     case CLI_INIT:</span><br><span>               e->command = "corosync show members";</span><br><span>           e->usage =</span><br><span style="color: hsl(0, 100%, 40%);">-                   "Usage: corosync show members\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                      "       Show corosync cluster members\n";</span><br><span style="color: hsl(120, 100%, 40%);">+                   "Usage: corosync show members\n" "       Show corosync cluster members\n";</span><br><span>               return NULL;</span><br><span> </span><br><span>     case CLI_GENERATE:</span><br><span style="color: hsl(0, 100%, 40%);">-              return NULL;    /* no completion */</span><br><span style="color: hsl(120, 100%, 40%);">+           return NULL;                    /* no completion */</span><br><span>  }</span><br><span> </span><br><span>        if (a->argc != e->args) {</span><br><span>              return CLI_SHOWUSAGE;</span><br><span>        }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   cs_err = cpg_iteration_initialize(cpg_handle, CPG_ITERATION_ALL, NULL, &cpg_iter);</span><br><span style="color: hsl(120, 100%, 40%);">+        if (!ast_rwlock_tryrdlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_debug(5, "corosync_show_members rdlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+             cs_err = cpg_iteration_initialize(cpg_handle, CPG_ITERATION_ALL, NULL, &cpg_iter);</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-      if (cs_err != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-          ast_cli(a->fd, "Failed to initialize CPG iterator.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-            return CLI_FAILURE;</span><br><span style="color: hsl(0, 100%, 40%);">-     }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_cli(a->fd, "\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                    "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "=== Cluster members =========================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "===\n");</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- for (i = 1, cs_err = cpg_iteration_next(cpg_iter, &cpg_desc);</span><br><span style="color: hsl(0, 100%, 40%);">-                       cs_err == CS_OK;</span><br><span style="color: hsl(0, 100%, 40%);">-                        cs_err = cpg_iteration_next(cpg_iter, &cpg_desc), i++) {</span><br><span style="color: hsl(0, 100%, 40%);">-#ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(0, 100%, 40%);">-                corosync_cfg_node_address_t addrs[8];</span><br><span style="color: hsl(0, 100%, 40%);">-           int num_addrs = 0;</span><br><span style="color: hsl(0, 100%, 40%);">-              unsigned int j;</span><br><span style="color: hsl(0, 100%, 40%);">-#endif</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-           ast_cli(a->fd, "=== Node %u\n", i);</span><br><span style="color: hsl(0, 100%, 40%);">-                ast_cli(a->fd, "=== --> Group: %s\n", cpg_desc.group.value);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-#ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(0, 100%, 40%);">-                /*</span><br><span style="color: hsl(0, 100%, 40%);">-               * Corosync 2.x cfg lib needs to allocate 1M on stack after calling</span><br><span style="color: hsl(0, 100%, 40%);">-              * corosync_cfg_get_node_addrs. netconsole thread has allocated only 0.5M</span><br><span style="color: hsl(0, 100%, 40%);">-                * resulting in crash.</span><br><span style="color: hsl(0, 100%, 40%);">-           */</span><br><span style="color: hsl(0, 100%, 40%);">-             cs_err = corosync_cfg_get_node_addrs(cfg_handle, cpg_desc.nodeid,</span><br><span style="color: hsl(0, 100%, 40%);">-                               ARRAY_LEN(addrs), &num_addrs, addrs);</span><br><span>            if (cs_err != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-                  ast_log(LOG_WARNING, "Failed to get node addresses\n");</span><br><span style="color: hsl(0, 100%, 40%);">-                       continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_cli(a->fd, "Failed to initialize CPG iterator: %u.\n", cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                      cpg_iteration_finalize(cpg_iter);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "corosync_show_members unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                     return CLI_FAILURE;</span><br><span>          }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-           for (j = 0; j < num_addrs; j++) {</span><br><span style="color: hsl(0, 100%, 40%);">-                    struct sockaddr *sa = (struct sockaddr *) addrs[j].address;</span><br><span style="color: hsl(0, 100%, 40%);">-                     size_t sa_len = (size_t) addrs[j].address_length;</span><br><span style="color: hsl(0, 100%, 40%);">-                       char buf[128];</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_cli(a->fd, "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                              "=============================================================\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                           "=== Cluster members =========================================\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                           "=============================================================\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                           "===\n");</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                 getnameinfo(sa, sa_len, buf, sizeof(buf), NULL, 0, NI_NUMERICHOST);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-                     ast_cli(a->fd, "=== --> Address %u: %s\n", j + 1, buf);</span><br><span style="color: hsl(0, 100%, 40%);">-         }</span><br><span style="color: hsl(0, 100%, 40%);">-#else</span><br><span style="color: hsl(0, 100%, 40%);">-          ast_cli(a->fd, "=== --> Nodeid: %"PRIu32"\n", cpg_desc.nodeid);</span><br><span style="color: hsl(120, 100%, 40%);">+             for (i = 1, cs_err = cpg_iteration_next(cpg_iter, &cpg_desc);</span><br><span style="color: hsl(120, 100%, 40%);">+                      cs_err == CS_OK; cs_err = cpg_iteration_next(cpg_iter, &cpg_desc), i++) {</span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(120, 100%, 40%);">+                  corosync_cfg_node_address_t addrs[8];</span><br><span style="color: hsl(120, 100%, 40%);">+                 int num_addrs = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                    unsigned int j;</span><br><span> #endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                   ast_cli(a->fd, "=== Node %u\n", i);</span><br><span style="color: hsl(120, 100%, 40%);">+                      ast_cli(a->fd, "=== --> Group: %s\n", cpg_desc.group.value);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef HAVE_COROSYNC_CFG_STATE_TRACK</span><br><span style="color: hsl(120, 100%, 40%);">+                  /*</span><br><span style="color: hsl(120, 100%, 40%);">+                     * Corosync 2.x cfg lib needs to allocate 1M on stack after calling</span><br><span style="color: hsl(120, 100%, 40%);">+                    * corosync_cfg_get_node_addrs. netconsole thread has allocated only 0.5M</span><br><span style="color: hsl(120, 100%, 40%);">+                      * resulting in crash.</span><br><span style="color: hsl(120, 100%, 40%);">+                         */</span><br><span style="color: hsl(120, 100%, 40%);">+                   cs_err = corosync_cfg_get_node_addrs(cfg_handle, cpg_desc.nodeid,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                              ARRAY_LEN(addrs), &num_addrs, addrs);</span><br><span style="color: hsl(120, 100%, 40%);">+                    if (cs_err != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                                ast_log(LOG_WARNING, "Failed to get node addresses\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                             continue;</span><br><span style="color: hsl(120, 100%, 40%);">+                     }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                   for (j = 0; j < num_addrs; j++) {</span><br><span style="color: hsl(120, 100%, 40%);">+                          struct sockaddr *sa = (struct sockaddr *) addrs[j].address;</span><br><span style="color: hsl(120, 100%, 40%);">+                           size_t sa_len = (size_t) addrs[j].address_length;</span><br><span style="color: hsl(120, 100%, 40%);">+                             char buf[128];</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                              getnameinfo(sa, sa_len, buf, sizeof(buf), NULL, 0, NI_NUMERICHOST);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                         ast_cli(a->fd, "=== --> Address %u: %s\n", j + 1, buf);</span><br><span style="color: hsl(120, 100%, 40%);">+                       }</span><br><span style="color: hsl(120, 100%, 40%);">+#else</span><br><span style="color: hsl(120, 100%, 40%);">+                      ast_cli(a->fd, "=== --> Nodeid: %" PRIu32 "\n", cpg_desc.nodeid);</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+           }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_cli(a->fd, "===\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                           "=============================================================\n" "\n");</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                cpg_iteration_finalize(cpg_iter);</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "corosync_show_members unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+     } else {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_cli(a->fd, "Failed to initialize CPG iterator: initializing CPG.\n");</span><br><span>       }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   ast_cli(a->fd, "===\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                    "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                     "\n");</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- cpg_iteration_finalize(cpg_iter);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>    return CLI_SUCCESS;</span><br><span> }</span><br><span> </span><br><span>@@ -880,7 +1095,7 @@</span><br><span>          return NULL;</span><br><span> </span><br><span>     case CLI_GENERATE:</span><br><span style="color: hsl(0, 100%, 40%);">-              return NULL;    /* no completion */</span><br><span style="color: hsl(120, 100%, 40%);">+           return NULL;                    /* no completion */</span><br><span>  }</span><br><span> </span><br><span>        if (a->argc != e->args) {</span><br><span>@@ -893,14 +1108,14 @@</span><br><span>             return CLI_FAILURE;</span><br><span>  }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   ast_rwlock_rdlock(&event_types_lock);</span><br><span>    event_types[AST_EVENT_PING].publish_to_stasis(event);</span><br><span style="color: hsl(0, 100%, 40%);">-   ast_rwlock_unlock(&event_types_lock);</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ ast_event_destroy(event);</span><br><span>    return CLI_SUCCESS;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-static char *corosync_show_config(struct ast_cli_entry *e, int cmd, struct ast_cli_args *a)</span><br><span style="color: hsl(120, 100%, 40%);">+static char *corosync_show_config(struct ast_cli_entry *e, int cmd,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                            struct ast_cli_args *a)</span><br><span> {</span><br><span>       unsigned int i;</span><br><span> </span><br><span>@@ -913,7 +1128,7 @@</span><br><span>           return NULL;</span><br><span> </span><br><span>     case CLI_GENERATE:</span><br><span style="color: hsl(0, 100%, 40%);">-              return NULL;    /* no completion */</span><br><span style="color: hsl(120, 100%, 40%);">+           return NULL;                    /* no completion */</span><br><span>  }</span><br><span> </span><br><span>        if (a->argc != e->args) {</span><br><span>@@ -921,16 +1136,15 @@</span><br><span>     }</span><br><span> </span><br><span>        ast_cli(a->fd, "\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                    "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "=== res_corosync config =====================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-             "===\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       "=============================================================\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                   "=== res_corosync config =====================================\n"</span><br><span style="color: hsl(120, 100%, 40%);">+                   "=============================================================\n" "===\n");</span><br><span> </span><br><span>  ast_rwlock_rdlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "corosync_show_config rdlock\n");</span><br><span>     for (i = 0; i < ARRAY_LEN(event_types); i++) {</span><br><span>            if (event_types[i].publish) {</span><br><span style="color: hsl(0, 100%, 40%);">-                   ast_cli(a->fd, "=== ==> Publishing Event Type: %s\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                                   event_types[i].name);</span><br><span style="color: hsl(120, 100%, 40%);">+                 ast_cli(a->fd, "=== ==> Publishing Event Type: %s\n", event_types[i].name);</span><br><span>          }</span><br><span>            if (event_types[i].subscribe) {</span><br><span>                      ast_cli(a->fd, "=== ==> Subscribing to Event Type: %s\n",</span><br><span>@@ -938,10 +1152,10 @@</span><br><span>           }</span><br><span>    }</span><br><span>    ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "corosync_show_config unlock\n");</span><br><span> </span><br><span>         ast_cli(a->fd, "===\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                    "=============================================================\n"</span><br><span style="color: hsl(0, 100%, 40%);">-                     "\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       "=============================================================\n" "\n");</span><br><span> </span><br><span>     return CLI_SUCCESS;</span><br><span> }</span><br><span>@@ -988,6 +1202,7 @@</span><br><span>      unsigned int i;</span><br><span> </span><br><span>  ast_rwlock_wrlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "load_general_config wrlock\n");</span><br><span> </span><br><span>  for (i = 0; i < ARRAY_LEN(event_types); i++) {</span><br><span>            event_types[i].publish = event_types[i].publish_default;</span><br><span>@@ -1009,17 +1224,16 @@</span><br><span>                   event_types[i].sub = stasis_forward_all(event_types[i].topic_fn(),</span><br><span>                                                                                                   corosync_topic());</span><br><span>                   stasis_message_router_add(stasis_router,</span><br><span style="color: hsl(0, 100%, 40%);">-                                                  event_types[i].message_type_fn(),</span><br><span style="color: hsl(0, 100%, 40%);">-                                               stasis_message_cb,</span><br><span style="color: hsl(0, 100%, 40%);">-                                              NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                        event_types[i].message_type_fn(),</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                     stasis_message_cb, NULL);</span><br><span>          } else if (!event_types[i].publish && event_types[i].sub) {</span><br><span>                  event_types[i].sub = stasis_forward_cancel(event_types[i].sub);</span><br><span style="color: hsl(0, 100%, 40%);">-                 stasis_message_router_remove(stasis_router,</span><br><span style="color: hsl(0, 100%, 40%);">-                                                  event_types[i].message_type_fn());</span><br><span style="color: hsl(120, 100%, 40%);">+                       stasis_message_router_remove(stasis_router, event_types[i].message_type_fn());</span><br><span>               }</span><br><span>    }</span><br><span> </span><br><span>        ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+     ast_debug(5, "load_general_config unlock\n");</span><br><span> </span><br><span>  return res;</span><br><span> }</span><br><span>@@ -1059,17 +1273,39 @@</span><br><span>   if (stasis_router) {</span><br><span> </span><br><span>             /* Unsubscribe all topic forwards and cancel all message routes */</span><br><span style="color: hsl(0, 100%, 40%);">-              ast_rwlock_wrlock(&event_types_lock);</span><br><span>            for (i = 0; i < ARRAY_LEN(event_types); i++) {</span><br><span style="color: hsl(120, 100%, 40%);">+                     struct ao2_container *messages = NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+                        unsigned char subscribe = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_rwlock_wrlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cleanup_module wrlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                    subscribe = event_types[i].subscribe;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span>                      if (event_types[i].sub) {</span><br><span>                            event_types[i].sub = stasis_forward_cancel(event_types[i].sub);</span><br><span>                              stasis_message_router_remove(stasis_router,</span><br><span style="color: hsl(0, 100%, 40%);">-                                                          event_types[i].message_type_fn());</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                        event_types[i].message_type_fn());</span><br><span>                  }</span><br><span>                    event_types[i].publish = 0;</span><br><span>                  event_types[i].subscribe = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+                 ast_rwlock_unlock(&event_types_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_debug(5, "cleanup_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+                  if (subscribe && event_types[i].cache_fn && event_types[i].message_type_fn) {</span><br><span style="color: hsl(120, 100%, 40%);">+                         messages =</span><br><span style="color: hsl(120, 100%, 40%);">+                                    stasis_cache_dump_all(event_types[i].cache_fn(),</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                event_types[i].message_type_fn());</span><br><span style="color: hsl(120, 100%, 40%);">+                          int messages_count = ao2_container_count(messages);</span><br><span style="color: hsl(120, 100%, 40%);">+                           ast_log(LOG_NOTICE,</span><br><span style="color: hsl(120, 100%, 40%);">+                                           "Clearing %i events of type %s of other nodes from stasis cache.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                                messages_count, event_types[i].name);</span><br><span style="color: hsl(120, 100%, 40%);">+                         ao2_callback(messages, OBJ_NODATA, clear_node_cache,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                   event_types[i].topic_fn());</span><br><span style="color: hsl(120, 100%, 40%);">+                          ast_log(LOG_NOTICE, "Cleared events of type %s from stasis cache.\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                               event_types[i].name);</span><br><span style="color: hsl(120, 100%, 40%);">+                         ao2_t_ref(messages, -1, "Dispose of flushed cache");</span><br><span style="color: hsl(120, 100%, 40%);">+                        }</span><br><span>            }</span><br><span style="color: hsl(0, 100%, 40%);">-               ast_rwlock_unlock(&event_types_lock);</span><br><span> </span><br><span>                stasis_message_router_unsubscribe_and_join(stasis_router);</span><br><span>           stasis_router = NULL;</span><br><span>@@ -1085,8 +1321,7 @@</span><br><span>        if (dispatch_thread.id != AST_PTHREADT_NULL) {</span><br><span>               char meepmeep = 'x';</span><br><span>                 dispatch_thread.stop = 1;</span><br><span style="color: hsl(0, 100%, 40%);">-               if (ast_carefulwrite(dispatch_thread.alert_pipe[1], &meepmeep, 1,</span><br><span style="color: hsl(0, 100%, 40%);">-                                   5000) == -1) {</span><br><span style="color: hsl(120, 100%, 40%);">+                if (ast_carefulwrite(dispatch_thread.alert_pipe[1], &meepmeep, 1, 5000) == -1) {</span><br><span>                         ast_log(LOG_ERROR, "Failed to write to pipe: %s (%d)\n",</span><br><span>                                   strerror(errno), errno);</span><br><span>             }</span><br><span>@@ -1103,16 +1338,21 @@</span><br><span>          dispatch_thread.alert_pipe[1] = -1;</span><br><span>  }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if (cpg_handle && (cs_err = cpg_finalize(cpg_handle)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-               ast_log(LOG_ERROR, "Failed to finalize cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-       cpg_handle = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+       if (!ast_rwlock_trywrlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              ast_debug(5, "cleanup_module wrlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+            if (cpg_handle && (cs_err = cpg_finalize(cpg_handle)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                     ast_log(LOG_ERROR, "Failed to finalize cpg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+             cpg_handle = 0;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-     if (cfg_handle && (cs_err = corosync_cfg_finalize(cfg_handle)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-              ast_log(LOG_ERROR, "Failed to finalize cfg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+          if (cfg_handle && (cs_err = corosync_cfg_finalize(cfg_handle)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                    ast_log(LOG_ERROR, "Failed to finalize cfg (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+             cfg_handle = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+               corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "cleanup_module unlock\n");</span><br><span>   }</span><br><span style="color: hsl(0, 100%, 40%);">-       cfg_handle = 0;</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>      ao2_cleanup(nodes);</span><br><span>  nodes = NULL;</span><br><span> }</span><br><span>@@ -1144,6 +1384,8 @@</span><br><span>           ast_log(AST_LOG_ERROR, "Failed to create message router for corosync topic\n");</span><br><span>            goto failed;</span><br><span>         }</span><br><span style="color: hsl(120, 100%, 40%);">+     stasis_message_router_set_congestion_limits(stasis_router, -1,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                                10 * AST_TASKPROCESSOR_HIGH_WATER_LEVEL);</span><br><span> </span><br><span>        if (STASIS_MESSAGE_TYPE_INIT(corosync_ping_message_type) != 0) {</span><br><span>             ast_log(AST_LOG_ERROR, "Failed to initialize corosync ping message type\n");</span><br><span>@@ -1155,42 +1397,58 @@</span><br><span>             goto failed;</span><br><span>         }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if ((cs_err = corosync_cfg_initialize(&cfg_handle, &cfg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-         ast_log(LOG_ERROR, "Failed to initialize cfg: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+       if (!ast_rwlock_trywrlock(&init_cpg_lock)) {</span><br><span style="color: hsl(120, 100%, 40%);">+              corosync_node_joined = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+             ast_debug(5, "load_module wrlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+               if ((cs_err = corosync_cfg_initialize(&cfg_handle, &cfg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                       ast_log(LOG_ERROR, "Failed to initialize cfg: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                       ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "load_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       goto failed;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           if ((cs_err = cpg_initialize(&cpg_handle, &cpg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_log(LOG_ERROR, "Failed to initialize cpg: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                       ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "load_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       goto failed;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_copy_string(name.value, "asterisk", sizeof(name.value));</span><br><span style="color: hsl(120, 100%, 40%);">+                name.length = strlen(name.value);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           if ((cs_err = cpg_join(cpg_handle, &name)) != CS_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+                    ast_log(LOG_ERROR, "Failed to join: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(120, 100%, 40%);">+                 ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "load_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       goto failed;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           if (pipe(dispatch_thread.alert_pipe) == -1) {</span><br><span style="color: hsl(120, 100%, 40%);">+                 ast_log(LOG_ERROR, "Failed to create alert pipe: %s (%d)\n",</span><br><span style="color: hsl(120, 100%, 40%);">+                                        strerror(errno), errno);</span><br><span style="color: hsl(120, 100%, 40%);">+                      ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                        ast_debug(5, "load_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                       goto failed;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+             corosync_node_joined = 1;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_rwlock_unlock(&init_cpg_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+                ast_debug(5, "load_module unlock\n");</span><br><span style="color: hsl(120, 100%, 40%);">+               if (corosync_pthread_create_background(&dispatch_thread.id, NULL,</span><br><span style="color: hsl(120, 100%, 40%);">+                                                                                    dispatch_thread_handler, NULL)) {</span><br><span style="color: hsl(120, 100%, 40%);">+                  ast_log(LOG_ERROR, "Error starting CPG dispatch thread.\n");</span><br><span style="color: hsl(120, 100%, 40%);">+                        goto failed;</span><br><span style="color: hsl(120, 100%, 40%);">+          }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+           ast_cli_register_multiple(corosync_cli, ARRAY_LEN(corosync_cli));</span><br><span style="color: hsl(120, 100%, 40%);">+     } else {</span><br><span>             goto failed;</span><br><span>         }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   if ((cs_err = cpg_initialize(&cpg_handle, &cpg_callbacks)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-          ast_log(LOG_ERROR, "Failed to initialize cpg: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-         goto failed;</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_copy_string(name.value, "asterisk", sizeof(name.value));</span><br><span style="color: hsl(0, 100%, 40%);">-  name.length = strlen(name.value);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       if ((cs_err = cpg_join(cpg_handle, &name)) != CS_OK) {</span><br><span style="color: hsl(0, 100%, 40%);">-              ast_log(LOG_ERROR, "Failed to join: (%d)\n", (int) cs_err);</span><br><span style="color: hsl(0, 100%, 40%);">-           goto failed;</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       if (pipe(dispatch_thread.alert_pipe) == -1) {</span><br><span style="color: hsl(0, 100%, 40%);">-           ast_log(LOG_ERROR, "Failed to create alert pipe: %s (%d)\n",</span><br><span style="color: hsl(0, 100%, 40%);">-                          strerror(errno), errno);</span><br><span style="color: hsl(0, 100%, 40%);">-                goto failed;</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       if (corosync_pthread_create_background(&dispatch_thread.id, NULL,</span><br><span style="color: hsl(0, 100%, 40%);">-                   dispatch_thread_handler, NULL)) {</span><br><span style="color: hsl(0, 100%, 40%);">-               ast_log(LOG_ERROR, "Error starting CPG dispatch thread.\n");</span><br><span style="color: hsl(0, 100%, 40%);">-          goto failed;</span><br><span style="color: hsl(0, 100%, 40%);">-    }</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_cli_register_multiple(corosync_cli, ARRAY_LEN(corosync_cli));</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>    return AST_MODULE_LOAD_SUCCESS;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-failed:</span><br><span style="color: hsl(120, 100%, 40%);">+  failed:</span><br><span>        cleanup_module();</span><br><span> </span><br><span>        return AST_MODULE_LOAD_DECLINE;</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/c/asterisk/+/14455">change 14455</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/c/asterisk/+/14455"/><meta itemprop="name" content="View Change"/></div></div>

<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: 13 </div>
<div style="display:none"> Gerrit-Change-Id: Ie8e99bc23f141a73c13ae6fb1948d148d4de17f2 </div>
<div style="display:none"> Gerrit-Change-Number: 14455 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Università di Bologna - CESIA VoIP <cesia.voip@unibo.it> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>