<p>Joshua Colp <strong>merged</strong> this change.</p><p><a href="https://gerrit.asterisk.org/9317">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Richard Mudgett: Looks good to me, but someone else must approve
Joshua Colp: Looks good to me, approved; Approved for Submit
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">main/cdr.c: Alleviate CDR deadlock<br><br>There is a rare case (do to the infrequent timing involved) where<br>CDR submission threads in batch mode can deadlock with a currently<br>running CDR batch process. This patch should remove the need for<br>holding the lock in the scheduler and should clean a few code<br>paths up that inconsistently submitted new work to the CDR batch<br>processor.<br><br>ASTERISK-27909<br><br>Change-Id: I6333e865db7c593c102c2fd948cecdb96481974d<br>Reported-by: Denis Lebedev<br>---<br>M main/cdr.c<br>1 file changed, 16 insertions(+), 19 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/main/cdr.c b/main/cdr.c</span><br><span>index 5286733..62fcdf5 100644</span><br><span>--- a/main/cdr.c</span><br><span>+++ b/main/cdr.c</span><br><span>@@ -3785,6 +3785,7 @@</span><br><span> static int submit_scheduled_batch(const void *data)</span><br><span> {</span><br><span> struct module_config *mod_cfg;</span><br><span style="color: hsl(120, 100%, 40%);">+ int nextms;</span><br><span> </span><br><span> cdr_submit_batch(0);</span><br><span> </span><br><span>@@ -3793,25 +3794,23 @@</span><br><span> return 0;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- /* manually reschedule from this point in time */</span><br><span style="color: hsl(0, 100%, 40%);">- ast_mutex_lock(&cdr_sched_lock);</span><br><span style="color: hsl(0, 100%, 40%);">- cdr_sched = ast_sched_add(sched, mod_cfg->general->batch_settings.time * 1000, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(0, 100%, 40%);">- ast_mutex_unlock(&cdr_sched_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Calculate the next scheduled interval */</span><br><span style="color: hsl(120, 100%, 40%);">+ nextms = mod_cfg->general->batch_settings.time * 1000;</span><br><span> </span><br><span> ao2_cleanup(mod_cfg);</span><br><span style="color: hsl(0, 100%, 40%);">- /* returning zero so the scheduler does not automatically reschedule */</span><br><span style="color: hsl(0, 100%, 40%);">- return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return nextms;</span><br><span> }</span><br><span> </span><br><span> /*! Do not hold the batch lock while calling this function */</span><br><span style="color: hsl(0, 100%, 40%);">-static void submit_unscheduled_batch(void)</span><br><span style="color: hsl(120, 100%, 40%);">+static void start_batch_mode(void)</span><br><span> {</span><br><span> /* Prevent two deletes from happening at the same time */</span><br><span> ast_mutex_lock(&cdr_sched_lock);</span><br><span> /* this is okay since we are not being called from within the scheduler */</span><br><span> AST_SCHED_DEL(sched, cdr_sched);</span><br><span> /* schedule the submission to occur ASAP (1 ms) */</span><br><span style="color: hsl(0, 100%, 40%);">- cdr_sched = ast_sched_add(sched, 1, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+ cdr_sched = ast_sched_add_variable(sched, 1, submit_scheduled_batch, NULL, 1);</span><br><span> ast_mutex_unlock(&cdr_sched_lock);</span><br><span> </span><br><span> /* signal the do_cdr thread to wakeup early and do some work (that lazy thread ;) */</span><br><span>@@ -3876,9 +3875,9 @@</span><br><span> }</span><br><span> ast_mutex_unlock(&cdr_batch_lock);</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- /* Don't call submit_unscheduled_batch with the cdr_batch_lock held */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Don't submit a batch with cdr_batch_lock held */</span><br><span> if (submit_batch) {</span><br><span style="color: hsl(0, 100%, 40%);">- submit_unscheduled_batch();</span><br><span style="color: hsl(120, 100%, 40%);">+ start_batch_mode();</span><br><span> }</span><br><span> }</span><br><span> </span><br><span>@@ -3892,7 +3891,7 @@</span><br><span> struct timeval now;</span><br><span> schedms = ast_sched_wait(sched);</span><br><span> /* this shouldn't happen, but provide a 1 second default just in case */</span><br><span style="color: hsl(0, 100%, 40%);">- if (schedms <= 0)</span><br><span style="color: hsl(120, 100%, 40%);">+ if (schedms < 0)</span><br><span> schedms = 1000;</span><br><span> now = ast_tvadd(ast_tvnow(), ast_samp2tv(schedms, 1000));</span><br><span> timeout.tv_sec = now.tv_sec;</span><br><span>@@ -3902,7 +3901,7 @@</span><br><span> ast_cond_timedwait(&cdr_pending_cond, &cdr_pending_lock, &timeout);</span><br><span> numevents = ast_sched_runq(sched);</span><br><span> ast_mutex_unlock(&cdr_pending_lock);</span><br><span style="color: hsl(0, 100%, 40%);">- ast_debug(2, "Processed %d scheduled CDR batches from the run queue\n", numevents);</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_debug(2, "Processed %d CDR batches from the run queue\n", numevents);</span><br><span> }</span><br><span> </span><br><span> return NULL;</span><br><span>@@ -4220,7 +4219,7 @@</span><br><span> } else if (!ast_test_flag(&mod_cfg->general->settings, CDR_BATCHMODE)) {</span><br><span> ast_cli(a->fd, "Cannot submit CDR batch: batch mode not enabled.\n");</span><br><span> } else {</span><br><span style="color: hsl(0, 100%, 40%);">- submit_unscheduled_batch();</span><br><span style="color: hsl(120, 100%, 40%);">+ start_batch_mode();</span><br><span> ast_cli(a->fd, "Submitted CDRs to backend engines for processing. This may take a while.\n");</span><br><span> }</span><br><span> ao2_cleanup(mod_cfg);</span><br><span>@@ -4336,7 +4335,7 @@</span><br><span> aco_option_register(&cfg_info, "scheduleronly", ACO_EXACT, general_options, DEFAULT_BATCH_SCHEDULER_ONLY, OPT_BOOLFLAG_T, 1, FLDSET(struct ast_cdr_config, batch_settings.settings), BATCH_MODE_SCHEDULER_ONLY);</span><br><span> aco_option_register(&cfg_info, "safeshutdown", ACO_EXACT, general_options, DEFAULT_BATCH_SAFE_SHUTDOWN, OPT_BOOLFLAG_T, 1, FLDSET(struct ast_cdr_config, batch_settings.settings), BATCH_MODE_SAFE_SHUTDOWN);</span><br><span> aco_option_register(&cfg_info, "size", ACO_EXACT, general_options, DEFAULT_BATCH_SIZE, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.size), 0, MAX_BATCH_SIZE);</span><br><span style="color: hsl(0, 100%, 40%);">- aco_option_register(&cfg_info, "time", ACO_EXACT, general_options, DEFAULT_BATCH_TIME, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.time), 0, MAX_BATCH_TIME);</span><br><span style="color: hsl(120, 100%, 40%);">+ aco_option_register(&cfg_info, "time", ACO_EXACT, general_options, DEFAULT_BATCH_TIME, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.time), 1, MAX_BATCH_TIME);</span><br><span> }</span><br><span> </span><br><span> if (aco_process_config(&cfg_info, reload) == ACO_PROCESS_ERROR) {</span><br><span>@@ -4397,8 +4396,6 @@</span><br><span> </span><br><span> static void cdr_enable_batch_mode(struct ast_cdr_config *config)</span><br><span> {</span><br><span style="color: hsl(0, 100%, 40%);">- SCOPED_LOCK(batch, &cdr_batch_lock, ast_mutex_lock, ast_mutex_unlock);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> /* Only create the thread level portions once */</span><br><span> if (cdr_thread == AST_PTHREADT_NULL) {</span><br><span> ast_cond_init(&cdr_pending_cond, NULL);</span><br><span>@@ -4408,9 +4405,9 @@</span><br><span> }</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- /* Kill the currently scheduled item */</span><br><span style="color: hsl(0, 100%, 40%);">- AST_SCHED_DEL(sched, cdr_sched);</span><br><span style="color: hsl(0, 100%, 40%);">- cdr_sched = ast_sched_add(sched, config->batch_settings.time * 1000, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Start the batching process */</span><br><span style="color: hsl(120, 100%, 40%);">+ start_batch_mode();</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> ast_log(LOG_NOTICE, "CDR batch mode logging enabled, first of either size %u or time %u seconds.\n",</span><br><span> config->batch_settings.size, config->batch_settings.time);</span><br><span> }</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/9317">change 9317</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/9317"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I6333e865db7c593c102c2fd948cecdb96481974d </div>
<div style="display:none"> Gerrit-Change-Number: 9317 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins2 </div>
<div style="display:none"> Gerrit-Reviewer: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Matthew Fredrickson <creslin@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Richard Mudgett <rmudgett@digium.com> </div>