[asterisk-dev] Higher scheduling priority for voice handling

J.A. Bezemer J.A.Bezemer+asteriskdev at opensourcepartners.nl
Fri Feb 3 14:04:23 CST 2012


Hi all,

Thanks for the totally wonderful work you're doing on Asterisk, making
it such a pleasure to operate and use.

One of our Asterisk instances happens to run on a relatively low-spec
single-core machine. Calls are always SIP to SIP but routed through
Asterisk for monitor purposes etc. Per active call, CPU usage is about
1.5% (alaw, no transcoding), which is totally acceptable.

However, we get considerable and annoying audio hiccups whenever
Asterisk is doing something "heavy", for example a full reload or just
dialplan reload, or when someone dials into some complex extension that
spawns multiple parallel LOCAL chans doing all kinds of interesting
stuff. And of course "core show translation recalc".

(To repeat our experience properly on a lightning-fast multi-core
machine: boot linux with maxcpus=0, which turns off SMP; and slow down
the CPU using for example /sys/.../thermal/cooling_device*. Also note
that our system is still a fair bit more powerful than the plug
computers that other folks are running Asterisk on.)

Asterisk is running at realtime priority (SCHED_RR), but the problem is
that _all_ of asterisk is running at _the same_ priority. And at the
same priority, Linux apparently doesn't like to preempt running threads
very quickly. (Also see sched_setscheduler(2) and
sched_rr_get_interval(2).)

We investigated if we could run all important voice handling at a higher
scheduling priority. Turns out that voice handling occurs at rather many
places, but once these are identified, it's a piece of cake. We actually
used not one but two extra scheduling priority levels: 12 for real
"person-to-person" voice channels, 11 for other audio, and the existing
default of 10 for everything else.

Results are phenomenal: we now have no audio hiccups whatsoever.

So with this we humbly suggest that you could look at this issue more
thoroughly than we could, for make benefit all glorious low-spec
Asterisk systems in the world.


Below is our totally-quick'n'dirty "proof of concept" patch against
Asterisk 1.8.9.0 (should apply widely) of which we are rather ashamed,
but we stopped worrying after reaching the Works For Us[TM] stage. This
covers all the usual audio handling we have, including file playback,
MoH, tone generators, early media, and apparently also ChanSpy without
us even trying. Still to do are probably Record and voicemail-record.
Patch is public domain (or CC0, at your option), but please re-implement
in a proper way.


Best regards, and keep up the good work,

Anne Bezemer


PS. We came across an existing weird pthread_setschedparam() call in
features.c:bridge_call_thread_launch(). At least it seems wrong, since
SCHED_RR requires a priority value >=1; also why would one want to set
the policy there anyway? The history is as follows:

https://code.asterisk.org/code/changelog/asterisk?cs=7035
https://code.asterisk.org/code/changelog/asterisk?cs=4677
https://issues.asterisk.org/view.php?id=3241
https://issues.asterisk.org/view.php?id=2460

...which seems to suggest that the original author intended to run at
SCHED_RR with "any undefined priority" (since no
pthread_attr_setschedparam is used), not being aware that SCHED_RR and
priority are actually inherited across threads. Also, the explicit
setting would even have been ignored since PTHREAD_EXPLICIT_SCHED is not
set. So, please add some explanatory comments if this is actually
useful/required for any purpose; or else just drop this stuff.


PS2. Searching for anyone else having tried this didn't turn up any
results; possibly because both realtime and priority have totally
different meanings in Asterisk. So for findability: we're talking about
asterisk realtime scheduler policy realtime scheduling priority
SCHED_RR sched_setscheduler pthread_setschedparam rt_priority rtprio
chrt.


diff -ur asterisk-1.8.9.0-orig/main/channel.c asterisk-1.8.9.0/main/channel.c
--- asterisk-1.8.9.0-orig/main/channel.c	2011-12-17 00:51:13.000000000 +0100
+++ asterisk-1.8.9.0/main/channel.c	2012-01-31 20:31:00.000000000 +0100
@@ -68,6 +68,32 @@
  #include "asterisk/global_datastores.h"
  #include "asterisk/data.h"

+#include <pthread.h>
+#include <string.h>
+void highprio_set(char *name, int defaultprio, int setprio)
+{
+  int policy;
+  struct sched_param param;
+  int res;
+  if (pthread_getschedparam(pthread_self(), &policy, &param) != 0)
+    return;
+  if (policy != SCHED_RR || param.sched_priority < defaultprio)
+    return;
+  if (param.sched_priority == setprio)
+    return;
+  param.sched_priority = setprio;
+  res = pthread_setschedparam(pthread_self(), SCHED_RR, &param);
+  ast_verb(3, "highprio going to %d on %s thread %p: %s\n",
+	setprio, (name == NULL) ? "NULL" : name, (void *) pthread_self(),
+	strerror(res));
+}
+
+#define HIGHPRIO_DEFAULT  10
+#define HIGHPRIO_VOICE    12
+#define HIGHPRIO_REST     11
+void highprio_set(char *name, int defaultprio, int setprio);
+static void *silence_generator_alloc(struct ast_channel *chan, void *data);
+
  #ifdef HAVE_EPOLL
  #include <sys/epoll.h>
  #endif
@@ -2803,6 +2829,7 @@
  		if (chan->generator && chan->generator->release) {
  			chan->generator->release(chan, chan->generatordata);
  		}
+		highprio_set(chan->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  	}
  	chan->generatordata = NULL;
  	chan->generator = NULL;
@@ -3037,6 +3064,7 @@
  		ast_channel_set_fd(chan, AST_GENERATOR_FD, -1);
  		ast_clear_flag(chan, AST_FLAG_WRITE_INT);
  		ast_settimeout(chan, 0, NULL, NULL);
+		highprio_set(chan->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  	}
  	ast_channel_unlock(chan);
  }
@@ -3080,11 +3108,15 @@
  		if (chan->generator && chan->generator->release)
  			chan->generator->release(chan, chan->generatordata);
  		chan->generatordata = NULL;
+		highprio_set(chan->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  	}
  	if (gen->alloc && !(chan->generatordata = gen->alloc(chan, params))) {
  		res = -1;
  	}
  	if (!res) {
+		/* no need to have silence at high-prio */
+		if (gen->alloc != silence_generator_alloc)
+			highprio_set(chan->name, HIGHPRIO_DEFAULT, HIGHPRIO_REST);
  		ast_settimeout(chan, 50, generator_force, chan);
  		chan->generator = gen;
  	}
@@ -7334,6 +7366,8 @@
  	ast_indicate(c0, AST_CONTROL_SRCUPDATE);
  	ast_indicate(c1, AST_CONTROL_SRCUPDATE);

+	highprio_set(c0->name, HIGHPRIO_DEFAULT, HIGHPRIO_VOICE);
+
  	for (/* ever */;;) {
  		struct timeval now = { 0, };
  		int to;
@@ -7452,6 +7486,7 @@

  				c0->_bridge = NULL;
  				c1->_bridge = NULL;
+				highprio_set(c0->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  				return res;
  			} else {
  				ast_clear_flag(c0, AST_FLAG_NBRIDGE);
@@ -7477,6 +7512,7 @@
  			if (ast_channel_make_compatible(c0, c1)) {
  				ast_log(LOG_WARNING, "Can't make %s and %s compatible\n", c0->name, c1->name);
  				manager_bridge_event(0, 1, c0, c1);
+				highprio_set(c0->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  				return AST_BRIDGE_FAILED;
  			}
  			o0nativeformats = c0->nativeformats;
@@ -7494,6 +7530,8 @@
  		}
  	}

+	highprio_set(c0->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
+
  	ast_clear_flag(c0, AST_FLAG_END_DTMF_ONLY);
  	ast_clear_flag(c1, AST_FLAG_END_DTMF_ONLY);

diff -ur asterisk-1.8.9.0-orig/main/file.c asterisk-1.8.9.0/main/file.c
--- asterisk-1.8.9.0-orig/main/file.c	2012-01-24 17:19:31.000000000 +0100
+++ asterisk-1.8.9.0/main/file.c	2012-01-31 20:31:11.000000000 +0100
@@ -48,6 +48,11 @@
  #include "asterisk/astobj2.h"
  #include "asterisk/test.h"

+#define HIGHPRIO_DEFAULT  10
+#define HIGHPRIO_VOICE    12
+#define HIGHPRIO_REST     11
+void highprio_set(char *name, int defaultprio, int setprio);
+
  /*
   * The following variable controls the layout of localized sound files.
   * If 0, use the historical layout with prefix just before the filename
@@ -138,6 +143,8 @@
  		tmp->vstream = NULL;
  	}

+	highprio_set(tmp->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
+
  	ast_channel_unlock(tmp);

  	return 0;
@@ -986,6 +993,7 @@
  		return -1;
  	if (vfs && ast_applystream(chan, vfs))
  		return -1;
+	highprio_set(chan->name, HIGHPRIO_DEFAULT, HIGHPRIO_REST);
  	res = ast_playstream(fs);
  	if (!res && vfs)
  		res = ast_playstream(vfs);
diff -ur asterisk-1.8.9.0-orig/main/asterisk.c asterisk-1.8.9.0/main/asterisk.c
--- asterisk-1.8.9.0-orig/main/asterisk.c	2011-09-13 23:33:20.000000000 +0200
+++ asterisk-1.8.9.0/main/asterisk.c	2012-01-31 20:30:58.000000000 +0100
@@ -146,6 +146,11 @@

  #include "../defaults.h"

+#define HIGHPRIO_DEFAULT  10
+#define HIGHPRIO_VOICE    12
+#define HIGHPRIO_REST     11
+void highprio_set(char *name, int defaultprio, int setprio);
+
  #ifndef AF_LOCAL
  #define AF_LOCAL AF_UNIX
  #define PF_LOCAL PF_UNIX
@@ -1045,7 +1050,8 @@

  	if (pid == 0) {
  #ifdef HAVE_CAP
-		cap_t cap = cap_from_text("cap_net_admin-eip");
+		cap_t cap = cap_from_text("cap_net_admin-eip cap_sys_nice-eip");
+		/* actually, "" should be enough to drop everything */

  		if (cap_set_proc(cap)) {
  			/* Careful with order! Logging cannot happen after we close FDs */
@@ -3117,6 +3123,9 @@
  	struct stat canary_stat;
  	struct timeval now;

+	/* voice threads should not block us */
+	highprio_set("canary_thread", HIGHPRIO_DEFAULT, HIGHPRIO_VOICE);
+
  	/* Give the canary time to sing */
  	sleep(120);

@@ -3524,8 +3533,9 @@
  		if (has_cap) {
  			cap_t cap;

-			cap = cap_from_text("cap_net_admin=eip");
+			cap = cap_from_text("cap_net_admin=eip cap_sys_nice=eip");

+			/* drops all unmentioned capabilities permanently */
  			if (cap_set_proc(cap))
  				ast_log(LOG_WARNING, "Unable to install capabilities.\n");

diff -ur asterisk-1.8.9.0-orig/main/app.c asterisk-1.8.9.0/main/app.c
--- asterisk-1.8.9.0-orig/main/app.c	2011-09-21 00:38:54.000000000 +0200
+++ asterisk-1.8.9.0/main/app.c	2012-01-31 20:30:55.000000000 +0100
@@ -2141,7 +2141,8 @@
  	} else {
  		/* Child */
  #ifdef HAVE_CAP
-		cap_t cap = cap_from_text("cap_net_admin-eip");
+		cap_t cap = cap_from_text("cap_net_admin-eip cap_sys_nice-eip");
+		/* actually, "" should be enough to drop everything */

  		if (cap_set_proc(cap)) {
  			ast_log(LOG_WARNING, "Unable to remove capabilities.\n");
diff -ur asterisk-1.8.9.0-orig/apps/app_dial.c asterisk-1.8.9.0/apps/app_dial.c
--- asterisk-1.8.9.0-orig/apps/app_dial.c	2011-12-23 16:24:33.000000000 +0100
+++ asterisk-1.8.9.0/apps/app_dial.c	2012-01-31 20:27:41.000000000 +0100
@@ -68,6 +68,32 @@
  #include "asterisk/indications.h"
  #include "asterisk/framehook.h"

+#include <pthread.h>
+#include <string.h>
+/* One way or another, dlopen fails with "undefined symbol" if referenced
+   to function in main process; so include a full local copy instead. */
+static void highprio_set(char *name, int defaultprio, int setprio)
+{
+  int policy;
+  struct sched_param param;
+  int res;
+  if (pthread_getschedparam(pthread_self(), &policy, &param) != 0)
+    return;
+  if (policy != SCHED_RR || param.sched_priority < defaultprio)
+    return;
+  if (param.sched_priority == setprio)
+    return;
+  param.sched_priority = setprio;
+  res = pthread_setschedparam(pthread_self(), SCHED_RR, &param);
+  ast_verb(3, "highprio going to %d on %s thread %p: %s\n",
+	setprio, (name == NULL) ? "NULL" : name, (void *) pthread_self(),
+	strerror(res));
+}
+
+#define HIGHPRIO_DEFAULT  10
+#define HIGHPRIO_VOICE    12
+#define HIGHPRIO_REST     11
+
  /*** DOCUMENTATION
  	<application name="Dial" language="en_US">
  		<synopsis>
@@ -1061,6 +1087,11 @@
  		ast_poll_channel_add(in, epollo->chan);
  #endif

+	/* Do high-prio forwarding of early audio (search "early audio" below).
+	   caller <- callee could be "if (single)", but we actually forward
+	   caller -> callee(s) too. */
+	highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_REST);
+
  	while (*to && !peer) {
  		struct chanlist *o;
  		int pos = 0; /* how many channels do we handle */
@@ -1091,6 +1122,7 @@
  			if (is_cc_recall) {
  				ast_cc_failed(cc_recall_core_id, "Everyone is busy/congested for the recall. How sad");
  			}
+			highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  			return NULL;
  		}
  		winner = ast_waitfor_n(watchers, pos, to);
@@ -1375,6 +1407,7 @@
  				case AST_FRAME_VOICE:
  				case AST_FRAME_IMAGE:
  				case AST_FRAME_TEXT:
+					/* this forwards early audio caller <- callee   can be single-destination only */
  					if (!ast_test_flag64(outgoing, OPT_RINGBACK | OPT_MUSICBACK) && ast_write(in, f)) {
  						ast_log(LOG_WARNING, "Unable to write frametype: %d\n",
  							f->frametype);
@@ -1414,6 +1447,7 @@
  				if (is_cc_recall) {
  					ast_cc_completed(in, "CC completed, although the caller hung up (cancelled)");
  				}
+				highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  				return NULL;
  			}

@@ -1434,6 +1468,7 @@
  						if (is_cc_recall) {
  							ast_cc_completed(in, "CC completed, but the caller used DTMF to exit");
  						}
+						highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  						return NULL;
  					}
  					ast_channel_unlock(in);
@@ -1449,6 +1484,7 @@
  					if (is_cc_recall) {
  						ast_cc_completed(in, "CC completed, but the caller hung up with DTMF");
  					}
+					highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
  					return NULL;
  				}
  			}
@@ -1472,6 +1508,7 @@
  				case AST_FRAME_TEXT:
  				case AST_FRAME_DTMF_BEGIN:
  				case AST_FRAME_DTMF_END:
+					/* this forwards early audio caller -> callee(s)   can be multi-destination */
  					if (ast_write(o->chan, f)) {
  						ast_log(LOG_WARNING, "Unable to forward frametype: %d\n",
  							f->frametype);
@@ -1513,6 +1550,8 @@
  			ast_cdr_noanswer(in->cdr);
  	}

+	highprio_set(in->name, HIGHPRIO_DEFAULT, HIGHPRIO_DEFAULT);
+
  #ifdef HAVE_EPOLL
  	for (epollo = outgoing; epollo; epollo = epollo->next) {
  		if (epollo->chan)
diff -ur asterisk-1.8.9.0-orig/res/res_musiconhold.c asterisk-1.8.9.0/res/res_musiconhold.c
--- asterisk-1.8.9.0-orig/res/res_musiconhold.c	2011-12-27 21:48:11.000000000 +0100
+++ asterisk-1.8.9.0/res/res_musiconhold.c	2012-01-31 20:31:54.000000000 +0100
@@ -67,6 +67,32 @@
  #include "asterisk/time.h"
  #include "asterisk/poll-compat.h"

+#include <pthread.h>
+#include <string.h>
+/* One way or another, dlopen fails with "undefined symbol" if referenced
+   to function in main process; so include a full local copy instead. */
+static void highprio_set(char *name, int defaultprio, int setprio)
+{
+  int policy;
+  struct sched_param param;
+  int res;
+  if (pthread_getschedparam(pthread_self(), &policy, &param) != 0)
+    return;
+  if (policy != SCHED_RR || param.sched_priority < defaultprio)
+    return;
+  if (param.sched_priority == setprio)
+    return;
+  param.sched_priority = setprio;
+  res = pthread_setschedparam(pthread_self(), SCHED_RR, &param);
+  ast_verb(3, "highprio going to %d on %s thread %p: %s\n",
+	setprio, (name == NULL) ? "NULL" : name, (void *) pthread_self(),
+	strerror(res));
+}
+
+#define HIGHPRIO_DEFAULT  10
+#define HIGHPRIO_VOICE    12
+#define HIGHPRIO_REST     11
+
  #define INITIAL_NUM_FILES   8
  #define HANDLE_REF	1
  #define DONT_UNREF	0
@@ -632,6 +658,8 @@
  	int len;
  	struct timeval deadline, tv_tmp;

+	highprio_set("monmp3thread", HIGHPRIO_DEFAULT, HIGHPRIO_REST);
+
  	deadline.tv_sec = 0;
  	deadline.tv_usec = 0;
  	for(;/* ever */;) {



More information about the asterisk-dev mailing list