<p>Joshua Colp <strong>merged</strong> this change.</p><p><a href="https://gerrit.asterisk.org/9317">View Change</a></p><div style="white-space:pre-wrap">Approvals:
  Richard Mudgett: Looks good to me, but someone else must approve
  Joshua Colp: Looks good to me, approved; Approved for Submit

</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">main/cdr.c: Alleviate CDR deadlock<br><br>There is a rare case (do to the infrequent timing involved) where<br>CDR submission threads in batch mode can deadlock with a currently<br>running CDR batch process.  This patch should remove the need for<br>holding the lock in the scheduler and should clean a few code<br>paths up that inconsistently submitted new work to the CDR batch<br>processor.<br><br>ASTERISK-27909<br><br>Change-Id: I6333e865db7c593c102c2fd948cecdb96481974d<br>Reported-by: Denis Lebedev<br>---<br>M main/cdr.c<br>1 file changed, 16 insertions(+), 19 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/main/cdr.c b/main/cdr.c</span><br><span>index 5286733..62fcdf5 100644</span><br><span>--- a/main/cdr.c</span><br><span>+++ b/main/cdr.c</span><br><span>@@ -3785,6 +3785,7 @@</span><br><span> static int submit_scheduled_batch(const void *data)</span><br><span> {</span><br><span>    struct module_config *mod_cfg;</span><br><span style="color: hsl(120, 100%, 40%);">+        int nextms;</span><br><span> </span><br><span>      cdr_submit_batch(0);</span><br><span> </span><br><span>@@ -3793,25 +3794,23 @@</span><br><span>           return 0;</span><br><span>    }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   /* manually reschedule from this point in time */</span><br><span style="color: hsl(0, 100%, 40%);">-       ast_mutex_lock(&cdr_sched_lock);</span><br><span style="color: hsl(0, 100%, 40%);">-    cdr_sched = ast_sched_add(sched, mod_cfg->general->batch_settings.time * 1000, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(0, 100%, 40%);">-     ast_mutex_unlock(&cdr_sched_lock);</span><br><span style="color: hsl(120, 100%, 40%);">+        /* Calculate the next scheduled interval */</span><br><span style="color: hsl(120, 100%, 40%);">+   nextms = mod_cfg->general->batch_settings.time * 1000;</span><br><span> </span><br><span>     ao2_cleanup(mod_cfg);</span><br><span style="color: hsl(0, 100%, 40%);">-   /* returning zero so the scheduler does not automatically reschedule */</span><br><span style="color: hsl(0, 100%, 40%);">- return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+   return nextms;</span><br><span> }</span><br><span> </span><br><span> /*! Do not hold the batch lock while calling this function */</span><br><span style="color: hsl(0, 100%, 40%);">-static void submit_unscheduled_batch(void)</span><br><span style="color: hsl(120, 100%, 40%);">+static void start_batch_mode(void)</span><br><span> {</span><br><span>  /* Prevent two deletes from happening at the same time */</span><br><span>    ast_mutex_lock(&cdr_sched_lock);</span><br><span>         /* this is okay since we are not being called from within the scheduler */</span><br><span>   AST_SCHED_DEL(sched, cdr_sched);</span><br><span>     /* schedule the submission to occur ASAP (1 ms) */</span><br><span style="color: hsl(0, 100%, 40%);">-      cdr_sched = ast_sched_add(sched, 1, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+    cdr_sched = ast_sched_add_variable(sched, 1, submit_scheduled_batch, NULL, 1);</span><br><span>       ast_mutex_unlock(&cdr_sched_lock);</span><br><span> </span><br><span>   /* signal the do_cdr thread to wakeup early and do some work (that lazy thread ;) */</span><br><span>@@ -3876,9 +3875,9 @@</span><br><span>         }</span><br><span>    ast_mutex_unlock(&cdr_batch_lock);</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-      /* Don't call submit_unscheduled_batch with the cdr_batch_lock held */</span><br><span style="color: hsl(120, 100%, 40%);">+    /* Don't submit a batch with cdr_batch_lock held */</span><br><span>      if (submit_batch) {</span><br><span style="color: hsl(0, 100%, 40%);">-             submit_unscheduled_batch();</span><br><span style="color: hsl(120, 100%, 40%);">+           start_batch_mode();</span><br><span>  }</span><br><span> }</span><br><span> </span><br><span>@@ -3892,7 +3891,7 @@</span><br><span>           struct timeval now;</span><br><span>          schedms = ast_sched_wait(sched);</span><br><span>             /* this shouldn't happen, but provide a 1 second default just in case */</span><br><span style="color: hsl(0, 100%, 40%);">-            if (schedms <= 0)</span><br><span style="color: hsl(120, 100%, 40%);">+          if (schedms < 0)</span><br><span>                  schedms = 1000;</span><br><span>              now = ast_tvadd(ast_tvnow(), ast_samp2tv(schedms, 1000));</span><br><span>            timeout.tv_sec = now.tv_sec;</span><br><span>@@ -3902,7 +3901,7 @@</span><br><span>                 ast_cond_timedwait(&cdr_pending_cond, &cdr_pending_lock, &timeout);</span><br><span>              numevents = ast_sched_runq(sched);</span><br><span>           ast_mutex_unlock(&cdr_pending_lock);</span><br><span style="color: hsl(0, 100%, 40%);">-                ast_debug(2, "Processed %d scheduled CDR batches from the run queue\n", numevents);</span><br><span style="color: hsl(120, 100%, 40%);">+         ast_debug(2, "Processed %d CDR batches from the run queue\n", numevents);</span><br><span>  }</span><br><span> </span><br><span>        return NULL;</span><br><span>@@ -4220,7 +4219,7 @@</span><br><span>         } else if (!ast_test_flag(&mod_cfg->general->settings, CDR_BATCHMODE)) {</span><br><span>           ast_cli(a->fd, "Cannot submit CDR batch: batch mode not enabled.\n");</span><br><span>   } else {</span><br><span style="color: hsl(0, 100%, 40%);">-                submit_unscheduled_batch();</span><br><span style="color: hsl(120, 100%, 40%);">+           start_batch_mode();</span><br><span>          ast_cli(a->fd, "Submitted CDRs to backend engines for processing.  This may take a while.\n");</span><br><span>  }</span><br><span>    ao2_cleanup(mod_cfg);</span><br><span>@@ -4336,7 +4335,7 @@</span><br><span>                aco_option_register(&cfg_info, "scheduleronly", ACO_EXACT, general_options, DEFAULT_BATCH_SCHEDULER_ONLY, OPT_BOOLFLAG_T, 1, FLDSET(struct ast_cdr_config, batch_settings.settings), BATCH_MODE_SCHEDULER_ONLY);</span><br><span>               aco_option_register(&cfg_info, "safeshutdown", ACO_EXACT, general_options, DEFAULT_BATCH_SAFE_SHUTDOWN, OPT_BOOLFLAG_T, 1, FLDSET(struct ast_cdr_config, batch_settings.settings), BATCH_MODE_SAFE_SHUTDOWN);</span><br><span>          aco_option_register(&cfg_info, "size", ACO_EXACT, general_options, DEFAULT_BATCH_SIZE, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.size), 0, MAX_BATCH_SIZE);</span><br><span style="color: hsl(0, 100%, 40%);">-                aco_option_register(&cfg_info, "time", ACO_EXACT, general_options, DEFAULT_BATCH_TIME, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.time), 0, MAX_BATCH_TIME);</span><br><span style="color: hsl(120, 100%, 40%);">+              aco_option_register(&cfg_info, "time", ACO_EXACT, general_options, DEFAULT_BATCH_TIME, OPT_UINT_T, PARSE_IN_RANGE, FLDSET(struct ast_cdr_config, batch_settings.time), 1, MAX_BATCH_TIME);</span><br><span>     }</span><br><span> </span><br><span>        if (aco_process_config(&cfg_info, reload) == ACO_PROCESS_ERROR) {</span><br><span>@@ -4397,8 +4396,6 @@</span><br><span> </span><br><span> static void cdr_enable_batch_mode(struct ast_cdr_config *config)</span><br><span> {</span><br><span style="color: hsl(0, 100%, 40%);">-      SCOPED_LOCK(batch, &cdr_batch_lock, ast_mutex_lock, ast_mutex_unlock);</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span>   /* Only create the thread level portions once */</span><br><span>     if (cdr_thread == AST_PTHREADT_NULL) {</span><br><span>               ast_cond_init(&cdr_pending_cond, NULL);</span><br><span>@@ -4408,9 +4405,9 @@</span><br><span>          }</span><br><span>    }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-   /* Kill the currently scheduled item */</span><br><span style="color: hsl(0, 100%, 40%);">- AST_SCHED_DEL(sched, cdr_sched);</span><br><span style="color: hsl(0, 100%, 40%);">-        cdr_sched = ast_sched_add(sched, config->batch_settings.time * 1000, submit_scheduled_batch, NULL);</span><br><span style="color: hsl(120, 100%, 40%);">+        /* Start the batching process */</span><br><span style="color: hsl(120, 100%, 40%);">+      start_batch_mode();</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span>        ast_log(LOG_NOTICE, "CDR batch mode logging enabled, first of either size %u or time %u seconds.\n",</span><br><span>                       config->batch_settings.size, config->batch_settings.time);</span><br><span> }</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/9317">change 9317</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/9317"/><meta itemprop="name" content="View Change"/></div></div>

<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I6333e865db7c593c102c2fd948cecdb96481974d </div>
<div style="display:none"> Gerrit-Change-Number: 9317 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins2 </div>
<div style="display:none"> Gerrit-Reviewer: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Matthew Fredrickson <creslin@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Richard Mudgett <rmudgett@digium.com> </div>