[Asterisk-code-review] PJSIP: Prevent deadlock due to dialog/transaction lock inver... (asterisk[13])

[Mark Michelson]

Mark Michelson has uploaded a new change for review.

  https://gerrit.asterisk.org/1930

Change subject: PJSIP: Prevent deadlock due to dialog/transaction lock inversion.
......................................................................

PJSIP: Prevent deadlock due to dialog/transaction lock inversion.

A deadlock was observed where the monitor thread was stuck, therefore
resulting in no incoming SIP traffic being processed.

The problem occurred when two 200 OK responses arrived in response to a
terminating NOTIFY request sent from Asterisk. The first 200 OK was
dispatched to a threadpool worker, who locked the corresponding
transaction. The second 200 OK arrived, resulting in the monitor thread
locking the dialog. At this point, the two threads are at odds, because
the monitor thread attempts to lock the transaction, and the threadpool
thread loops attempting to try to lock the dialog.

In this case, the fix is to not have the monitor thread attempt to hold
both the dialog and transaction locks at the same time. Instead, we
release the dialog lock before attempting to lock the transaction.

There have also been some debug messages added to the process in an
attempt to make it more clear what is going on in the process.

ASTERISK-25668 #close
Reported by Mark Michelson

Change-Id: I4db0705f1403737b4360e33a8e6276805d086d4a
---
M res/res_pjsip/pjsip_distributor.c
1 file changed, 11 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.asterisk.org:29418/asterisk refs/changes/30/1930/1

diff --git a/res/res_pjsip/pjsip_distributor.c b/res/res_pjsip/pjsip_distributor.c
index 0e0e90f..d4f4ca2 100644
--- a/res/res_pjsip/pjsip_distributor.c
+++ b/res/res_pjsip/pjsip_distributor.c
@@ -105,6 +105,7 @@
 		serializer_name = tsx->last_tx->mod_data[distributor_mod.id];
 		if (!ast_strlen_zero(serializer_name)) {
 			serializer = ast_taskprocessor_get(serializer_name, TPS_REF_IF_EXISTS);
+			ast_debug(3, "Found serializer %p on transaction %s\n", serializer, tsx->obj_name);
 		}
 	}
 
@@ -253,27 +254,31 @@
 	pjsip_dialog *dlg = find_dialog(rdata);
 	struct distributor_dialog_data *dist = NULL;
 	struct ast_taskprocessor *serializer = NULL;
-	struct ast_taskprocessor *req_serializer = NULL;
 	pjsip_rx_data *clone;
 
 	if (dlg) {
+		ast_debug(3, "Searching for serializer on dialog %s for %s\n",
+				dlg->obj_name, rdata->msg_info.info);
 		dist = pjsip_dlg_get_mod_data(dlg, distributor_mod.id);
 		if (dist) {
-			serializer = dist->serializer;
+			serializer = ao2_bump(dist->serializer);
+			ast_debug(3, "Found serializer %p on dialog %s\n", serializer, dlg->obj_name);
 		}
+		pjsip_dlg_dec_lock(dlg);
 	}
 
 	if (serializer) {
 		/* We have a serializer so we know where to send the message. */
 	} else if (rdata->msg_info.msg->type == PJSIP_RESPONSE_MSG) {
-		req_serializer = find_request_serializer(rdata);
-		serializer = req_serializer;
+		ast_debug(3, "No dialog serializer for response %s. Using request transaction as basis\n",
+				rdata->msg_info.info);
+		serializer = find_request_serializer(rdata);
 	} else if (!pjsip_method_cmp(&rdata->msg_info.msg->line.req.method, &pjsip_cancel_method)
 		|| !pjsip_method_cmp(&rdata->msg_info.msg->line.req.method, &pjsip_bye_method)) {
 		/* We have a BYE or CANCEL request without a serializer. */
 		pjsip_endpt_respond_stateless(ast_sip_get_pjsip_endpoint(), rdata,
 			PJSIP_SC_CALL_TSX_DOES_NOT_EXIST, NULL, NULL, NULL);
-		goto end;
+		return PJ_TRUE;
 	}
 
 	pjsip_rx_data_clone(rdata, 0, &clone);
@@ -296,11 +301,7 @@
 		ast_sip_push_task(serializer, distribute, clone);
 	}
 
-end:
-	if (dlg) {
-		pjsip_dlg_dec_lock(dlg);
-	}
-	ast_taskprocessor_unreference(req_serializer);
+	ast_taskprocessor_unreference(serializer);
 
 	return PJ_TRUE;
 }

-- 
To view, visit https://gerrit.asterisk.org/1930
To unsubscribe, visit https://gerrit.asterisk.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4db0705f1403737b4360e33a8e6276805d086d4a
Gerrit-PatchSet: 1
Gerrit-Project: asterisk
Gerrit-Branch: 13
Gerrit-Owner: Mark Michelson <mmichelson at digium.com>