<p>Matt Jordan has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.asterisk.org/c/asterisk/+/11373">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">res_prometheus: Add Asterisk endpoint metrics<br><br>This patch adds basic Asterisk endpoint statistics to the res_prometheus<br>module. This includes:<br><br>* asterisk_endpoints_state: The current state (unknown, online, offline)<br> for each defined endpoint.<br><br>* asterisk_endpoints_channels_count: The current number of channels<br> associated with a given endpoint.<br><br>* asterisk_endpoints_count: The current number of defined endpoints.<br><br>In all cases, enough information is provided with each endpoint metric<br>to determine a unique instance of Asterisk that provided the data, as well<br>as the underlying technology and resource definition.<br><br>ASTERISK-28403<br><br>Change-Id: I46443963330c206a7d12722d08dcaabef672310e<br>---<br>A res/prometheus/endpoints.c<br>M res/prometheus/prometheus_internal.h<br>M res/res_prometheus.c<br>3 files changed, 203 insertions(+), 1 deletion(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.asterisk.org:29418/asterisk refs/changes/73/11373/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/res/prometheus/endpoints.c b/res/prometheus/endpoints.c</span><br><span>new file mode 100644</span><br><span>index 0000000..b3d06bf</span><br><span>--- /dev/null</span><br><span>+++ b/res/prometheus/endpoints.c</span><br><span>@@ -0,0 +1,194 @@</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Asterisk -- An open source telephony toolkit.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * Copyright (C) 2019 Sangoma, Inc.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * Matt Jordan <mjordan@digium.com></span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * See http://www.asterisk.org for more information about</span><br><span style="color: hsl(120, 100%, 40%);">+ * the Asterisk project. Please do not directly contact</span><br><span style="color: hsl(120, 100%, 40%);">+ * any of the maintainers of this project for assistance;</span><br><span style="color: hsl(120, 100%, 40%);">+ * the project provides a web site, mailing lists and IRC</span><br><span style="color: hsl(120, 100%, 40%);">+ * channels for your use.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is free software, distributed under the terms of</span><br><span style="color: hsl(120, 100%, 40%);">+ * the GNU General Public License Version 2. See the LICENSE file</span><br><span style="color: hsl(120, 100%, 40%);">+ * at the top of the source tree.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \file</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Prometheus Endpoint Metrics</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \author Matt Jordan <mjordan@digium.com></span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk.h"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/stasis_endpoints.h"</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/res_prometheus.h"</span><br><span style="color: hsl(120, 100%, 40%);">+#include "prometheus_internal.h"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define ENDPOINTS_STATE_HELP "Individual endpoint states. 0=unknown; 1=offline; 2=online."</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define ENDPOINTS_CHANNELS_COUNT_HELP "Count of the number of channels currently existing that are associated with the endpoint."</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Callback function to get an endpoint's current state</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param metric The metric to populate</span><br><span style="color: hsl(120, 100%, 40%);">+ * \snapshot Endpoint snapshot</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+static void get_endpoint_state(struct prometheus_metric *metric, struct ast_endpoint_snapshot *snapshot)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ snprintf(metric->value, sizeof(metric->value), "%d", snapshot->state);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Callback function to get the current number of channel's associated with an endpoint</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param metric The metric to populate</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param snapshot Endpoint snapshot</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+static void get_endpoint_channel_count(struct prometheus_metric *metric, struct ast_endpoint_snapshot *snapshot)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ snprintf(metric->value, sizeof(metric->value), "%d", snapshot->num_channels);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Helper struct for generating individual endpoint stats</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+struct endpoint_metric_defs {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Help text to display</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ const char *help;</span><br><span style="color: hsl(120, 100%, 40%);">+ /*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Name of the metric</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ const char *name;</span><br><span style="color: hsl(120, 100%, 40%);">+ /*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Callback function to generate a metric value for a given endpoint</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ void (* const get_value)(struct prometheus_metric *metric, struct ast_endpoint_snapshot *snapshot);</span><br><span style="color: hsl(120, 100%, 40%);">+} endpoint_metric_defs[] = {</span><br><span style="color: hsl(120, 100%, 40%);">+ {</span><br><span style="color: hsl(120, 100%, 40%);">+ .help = ENDPOINTS_STATE_HELP,</span><br><span style="color: hsl(120, 100%, 40%);">+ .name = "asterisk_endpoints_state",</span><br><span style="color: hsl(120, 100%, 40%);">+ .get_value = get_endpoint_state,</span><br><span style="color: hsl(120, 100%, 40%);">+ },</span><br><span style="color: hsl(120, 100%, 40%);">+ {</span><br><span style="color: hsl(120, 100%, 40%);">+ .help = ENDPOINTS_CHANNELS_COUNT_HELP,</span><br><span style="color: hsl(120, 100%, 40%);">+ .name = "asterisk_endpoints_channels_count",</span><br><span style="color: hsl(120, 100%, 40%);">+ .get_value = get_endpoint_channel_count,</span><br><span style="color: hsl(120, 100%, 40%);">+ },</span><br><span style="color: hsl(120, 100%, 40%);">+};</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Callback invoked when Prometheus scrapes the server</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param response The response to populate with formatted metrics</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+static void endpoints_scrape_cb(struct ast_str **response)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ struct ao2_container *endpoints;</span><br><span style="color: hsl(120, 100%, 40%);">+ struct ao2_iterator it_endpoints;</span><br><span style="color: hsl(120, 100%, 40%);">+ struct stasis_message *message;</span><br><span style="color: hsl(120, 100%, 40%);">+ struct prometheus_metric *endpoint_metrics;</span><br><span style="color: hsl(120, 100%, 40%);">+ char eid_str[32];</span><br><span style="color: hsl(120, 100%, 40%);">+ int i, j, num_endpoints;</span><br><span style="color: hsl(120, 100%, 40%);">+ struct prometheus_metric endpoint_count = PROMETHEUS_METRIC_STATIC_INITIALIZATION(</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_GAUGE,</span><br><span style="color: hsl(120, 100%, 40%);">+ "asterisk_endpoints_count",</span><br><span style="color: hsl(120, 100%, 40%);">+ "Current endpoint count.",</span><br><span style="color: hsl(120, 100%, 40%);">+ NULL</span><br><span style="color: hsl(120, 100%, 40%);">+ );</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_eid_to_str(eid_str, sizeof(eid_str), &ast_eid_default);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ endpoints = stasis_cache_dump(ast_endpoint_cache(), ast_endpoint_snapshot_type());</span><br><span style="color: hsl(120, 100%, 40%);">+ if (!endpoints) {</span><br><span style="color: hsl(120, 100%, 40%);">+ return;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ num_endpoints = ao2_container_count(endpoints);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Current endpoint count */</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_SET_LABEL(&endpoint_count, 0, "eid", eid_str);</span><br><span style="color: hsl(120, 100%, 40%);">+ snprintf(endpoint_count.value, sizeof(endpoint_count.value), "%d", num_endpoints);</span><br><span style="color: hsl(120, 100%, 40%);">+ prometheus_metric_to_string(&endpoint_count, response);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (num_endpoints == 0) {</span><br><span style="color: hsl(120, 100%, 40%);">+ ao2_ref(endpoints, -1);</span><br><span style="color: hsl(120, 100%, 40%);">+ return;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ endpoint_metrics = ast_calloc(ARRAY_LEN(endpoint_metric_defs) * num_endpoints, sizeof(*endpoint_metrics));</span><br><span style="color: hsl(120, 100%, 40%);">+ if (!endpoint_metrics) {</span><br><span style="color: hsl(120, 100%, 40%);">+ ao2_ref(endpoints, -1);</span><br><span style="color: hsl(120, 100%, 40%);">+ return;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Endpoint dependent values */</span><br><span style="color: hsl(120, 100%, 40%);">+ for (i = 0; i < ARRAY_LEN(endpoint_metric_defs); i++) {</span><br><span style="color: hsl(120, 100%, 40%);">+ it_endpoints = ao2_iterator_init(endpoints, 0);</span><br><span style="color: hsl(120, 100%, 40%);">+ for (j = 0; (message = ao2_iterator_next(&it_endpoints)); ao2_ref(message, -1), j++) {</span><br><span style="color: hsl(120, 100%, 40%);">+ struct ast_endpoint_snapshot *snapshot = stasis_message_data(message);</span><br><span style="color: hsl(120, 100%, 40%);">+ int index = i * num_endpoints + j;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ endpoint_metrics[index].type = PROMETHEUS_METRIC_GAUGE;</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_copy_string(endpoint_metrics[index].name, endpoint_metric_defs[i].name, sizeof(endpoint_metrics[index].name));</span><br><span style="color: hsl(120, 100%, 40%);">+ endpoint_metrics[index].help = endpoint_metric_defs[i].help;</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_SET_LABEL(&endpoint_metrics[index], 0, "eid", eid_str);</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_SET_LABEL(&endpoint_metrics[index], 1, "id", (snapshot->id));</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_SET_LABEL(&endpoint_metrics[index], 2, "tech", (snapshot->tech));</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMETHEUS_METRIC_SET_LABEL(&endpoint_metrics[index], 3, "resource", (snapshot->resource));</span><br><span style="color: hsl(120, 100%, 40%);">+ endpoint_metric_defs[i].get_value(&endpoint_metrics[index], snapshot);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (j != 0) {</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_LIST_INSERT_TAIL(&endpoint_metrics[i * num_endpoints].children, &endpoint_metrics[index], entry);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ ao2_iterator_destroy(&it_endpoints);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ prometheus_metric_to_string(&endpoint_metrics[i * num_endpoints], response);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_free(endpoint_metrics);</span><br><span style="color: hsl(120, 100%, 40%);">+ ao2_ref(endpoints, -1);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+struct prometheus_callback endpoints_callback = {</span><br><span style="color: hsl(120, 100%, 40%);">+ .name = "Endpoints callback",</span><br><span style="color: hsl(120, 100%, 40%);">+ .callback_fn = endpoints_scrape_cb,</span><br><span style="color: hsl(120, 100%, 40%);">+};</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Callback invoked when the core module is unloaded</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+static void endpoint_metrics_unload_cb(void)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ prometheus_callback_unregister(&endpoints_callback);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \internal</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Metrics provider definition</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+static struct prometheus_metrics_provider provider = {</span><br><span style="color: hsl(120, 100%, 40%);">+ .name = "endpoints",</span><br><span style="color: hsl(120, 100%, 40%);">+ .unload_cb = endpoint_metrics_unload_cb,</span><br><span style="color: hsl(120, 100%, 40%);">+};</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+int endpoint_metrics_init(void)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ prometheus_metrics_provider_register(&provider);</span><br><span style="color: hsl(120, 100%, 40%);">+ prometheus_callback_register(&endpoints_callback);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span>diff --git a/res/prometheus/prometheus_internal.h b/res/prometheus/prometheus_internal.h</span><br><span>index 06cff98..7eb4457 100644</span><br><span>--- a/res/prometheus/prometheus_internal.h</span><br><span>+++ b/res/prometheus/prometheus_internal.h</span><br><span>@@ -38,4 +38,12 @@</span><br><span> */</span><br><span> int channel_metrics_init(void);</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Initialize endpoint metrics</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \retval 0 success</span><br><span style="color: hsl(120, 100%, 40%);">+ * \retval -1 error</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+int endpoint_metrics_init(void);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> #endif /* #define PROMETHEUS_INTERNAL_H__ */</span><br><span>diff --git a/res/res_prometheus.c b/res/res_prometheus.c</span><br><span>index ef5a7dd..05fc1ed 100644</span><br><span>--- a/res/res_prometheus.c</span><br><span>+++ b/res/res_prometheus.c</span><br><span>@@ -920,7 +920,7 @@</span><br><span> goto cleanup;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- if (channel_metrics_init()) {</span><br><span style="color: hsl(120, 100%, 40%);">+ if (channel_metrics_init() || endpoint_metrics_init()) {</span><br><span> goto cleanup;</span><br><span> }</span><br><span> </span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/c/asterisk/+/11373">change 11373</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/c/asterisk/+/11373"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I46443963330c206a7d12722d08dcaabef672310e </div>
<div style="display:none"> Gerrit-Change-Number: 11373 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Matt Jordan <mjordan@digium.com> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>