<p>Friendly Automation <strong>submitted</strong> this change.</p><p><a href="https://gerrit.asterisk.org/c/asterisk/+/16656">View Change</a></p><div style="white-space:pre-wrap">Approvals:
George Joseph: Looks good to me, approved
Friendly Automation: Approved for Submit
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">ast_coredumper: Refactor to better find things<br><br>The search for a running asterisk when --running is used<br>has been greatly simplified and in the event it doesn't<br>work, you can now specify a pid to use on the command<br>line with --pid.<br><br>The search for asterisk modules when --tarball-coredumps<br>is used has been enhanced to have a better chance of finding<br>them and in the event it doesn't work, you can now specify<br>--libdir on the command line to indicate the library directory<br>where they were installed.<br><br>The DATEFORMAT variable was renamed to DATEOPTS and is now<br>passed to the 'date' utility rather than running DATEFORMAT<br>as a command.<br><br>The coredump and output files are now renamed with DATEOPTS.<br>This can be disabled by specifying --no-rename.<br><br>Several confusing and conflicting options were removed:<br>--append-coredumps<br>--conffile<br>--no-default-search<br>--tarball-uniqueid<br><br>The script was re-structured to make it easier for follow.<br><br>Change-Id: I674be64bdde3ef310b6a551d4911c3b600ffee59<br>---<br>M configs/samples/ast_debug_tools.conf.sample<br>M contrib/scripts/ast_coredumper<br>A doc/CHANGES-staging/ast_coredumper.txt<br>3 files changed, 567 insertions(+), 488 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/configs/samples/ast_debug_tools.conf.sample b/configs/samples/ast_debug_tools.conf.sample</span><br><span>index 0f90f85..34200fa 100644</span><br><span>--- a/configs/samples/ast_debug_tools.conf.sample</span><br><span>+++ b/configs/samples/ast_debug_tools.conf.sample</span><br><span>@@ -4,46 +4,55 @@</span><br><span> # "sourced" by bash and must adhere to bash semantics.</span><br><span> #</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-# A list of coredumps and/or coredump search patterns.</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# The following settings are used by ast_coredumper</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# COREDUMPS is a a list of coredumps and/or coredump</span><br><span style="color: hsl(120, 100%, 40%);">+# search patterns.</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span> # Bash extended globs are enabled and any resulting files</span><br><span> # that aren't actually coredumps are silently ignored</span><br><span> # so you can be liberal with the globs.</span><br><span> #</span><br><span style="color: hsl(0, 100%, 40%);">-# If your patterns contains spaces be sure to only quote</span><br><span style="color: hsl(120, 100%, 40%);">+# If your patterns contain spaces be sure to only quote</span><br><span> # the portion of the pattern that DOESN'T contain wildcard</span><br><span> # expressions. If you quote the whole pattern, it won't</span><br><span> # be expanded and the glob characters will be treated as</span><br><span> # literals.</span><br><span> #</span><br><span> # The exclusion of files ending ".txt" is just for</span><br><span style="color: hsl(0, 100%, 40%);">-# demonstration purposes as non-coredumps will be ignored</span><br><span style="color: hsl(0, 100%, 40%);">-# anyway.</span><br><span style="color: hsl(0, 100%, 40%);">-COREDUMPS=(/tmp/core[-._]asterisk!(*.txt) /tmp/core[-._]$(hostname)!(*.txt))</span><br><span style="color: hsl(120, 100%, 40%);">+# demonstration purposes as non-asterisk-coredumps will be</span><br><span style="color: hsl(120, 100%, 40%);">+# ignored anyway.</span><br><span style="color: hsl(120, 100%, 40%);">+COREDUMPS=( /tmp/core!(*.txt) )</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-# The directory to contain output files and work directories.</span><br><span style="color: hsl(120, 100%, 40%);">+# OUTPUTDIR is the directory to contain output files and</span><br><span style="color: hsl(120, 100%, 40%);">+# work directories. </span><br><span> # For output from existing core files, the default is the</span><br><span> # directory that the core file is found in. For core files</span><br><span> # produced from a running process, the default is /tmp.</span><br><span> OUTPUTDIR=/tmp</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-# Date command for the "running" coredump and tarballs.</span><br><span style="color: hsl(0, 100%, 40%);">-# DATEFORMAT will be executed to get the timestamp.</span><br><span style="color: hsl(0, 100%, 40%);">-# Don't put quotes around the format string or they'll be</span><br><span style="color: hsl(0, 100%, 40%);">-# treated as literal characters. Also be aware of colons</span><br><span style="color: hsl(0, 100%, 40%);">-# in the output as you can't upload files with colons in</span><br><span style="color: hsl(0, 100%, 40%);">-# the name to Jira.</span><br><span style="color: hsl(120, 100%, 40%);">+# DATEOPTS is passed to the 'date' utility and is</span><br><span style="color: hsl(120, 100%, 40%);">+# used to set the timestamp used to create the</span><br><span style="color: hsl(120, 100%, 40%);">+# name of the output files and to rename the coredump.</span><br><span> #</span><br><span style="color: hsl(0, 100%, 40%);">-# Unix timestamp</span><br><span style="color: hsl(0, 100%, 40%);">-#DATEFORMAT='date +%s.%N'</span><br><span style="color: hsl(0, 100%, 40%);">-#</span><br><span style="color: hsl(0, 100%, 40%);">-# Unix timestamp on *BSD/MacOS after installing coreutils</span><br><span style="color: hsl(0, 100%, 40%);">-#DATEFORMAT='gdate +%s.%N'</span><br><span style="color: hsl(120, 100%, 40%);">+# Beware of colons in the output as you can't upload</span><br><span style="color: hsl(120, 100%, 40%);">+# files with colons in the name to Jira.</span><br><span style="color: hsl(120, 100%, 40%);">+# The preferred timestamp format is readable GMT.</span><br><span> #</span><br><span> # Readable GMT</span><br><span style="color: hsl(0, 100%, 40%);">-#DATEFORMAT='date -u +%FT%H-%M-%S%z'</span><br><span style="color: hsl(120, 100%, 40%);">+DATEOPTS='-u +%FT%H-%M-%SZ'</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# Unix timestamp</span><br><span style="color: hsl(120, 100%, 40%);">+#DATEOPTS='+%s.%N'</span><br><span> #</span><br><span> # Readable Local time</span><br><span style="color: hsl(0, 100%, 40%);">-DATEFORMAT='date +%FT%H-%M-%S%z'</span><br><span style="color: hsl(120, 100%, 40%);">+#DATEOPTS='+%FT%H-%M-%S%z'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# The following settings are used by ast_loggrabber</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span> </span><br><span> # A list of log files and/or log file search patterns using the</span><br><span> # same syntax as COREDUMPS.</span><br><span>diff --git a/contrib/scripts/ast_coredumper b/contrib/scripts/ast_coredumper</span><br><span>index 9d9f8bc..2868312 100755</span><br><span>--- a/contrib/scripts/ast_coredumper</span><br><span>+++ b/contrib/scripts/ast_coredumper</span><br><span>@@ -1,46 +1,478 @@</span><br><span> #!/usr/bin/env bash</span><br><span> # Turn on extended globbing</span><br><span> shopt -s extglob</span><br><span style="color: hsl(120, 100%, 40%);">+shopt -s nullglob</span><br><span> # Bail on any error</span><br><span> set -e</span><br><span> </span><br><span> prog=$(basename $0)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+# NOTE: <(cmd) is a bash construct that returns a temporary file name</span><br><span style="color: hsl(120, 100%, 40%);">+# from which the command output can be read. In this case, we're</span><br><span style="color: hsl(120, 100%, 40%);">+# extracting the block of text delimited by '#@@@FUNCSSTART@@@'</span><br><span style="color: hsl(120, 100%, 40%);">+# and '#@@@FUNCSEND@@@' from this file and 'source'ing it to</span><br><span style="color: hsl(120, 100%, 40%);">+# get some functions.</span><br><span style="color: hsl(120, 100%, 40%);">+source <(sed -n -r -e "/^#@@@FUNCSSTART@@@/,\${p;/^#@@@FUNCSEND@@@/q}" $0 | sed '1d;$d')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# The "!(*.txt)" is a bash construct that excludes files ending with .txt</span><br><span style="color: hsl(120, 100%, 40%);">+# from the glob match.</span><br><span style="color: hsl(120, 100%, 40%);">+declare -a COREDUMPS=( /tmp/core!(*.txt) )</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# A line starting with ': ' is a POSIX construct that makes the shell</span><br><span style="color: hsl(120, 100%, 40%);">+# perform the operation but ignore the result. This is an alternative to</span><br><span style="color: hsl(120, 100%, 40%);">+# having to do RUNNING=${RUNNING:=false} to set defaults.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+: ${ASTERISK_BIN:=$(which asterisk)}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${DATEOPTS='-u +%FT%H-%M-%SZ'}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${DELETE_COREDUMPS_AFTER:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${DELETE_RESULTS_AFTER:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${DRY_RUN:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${GDB:=$(which gdb)}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${HELP:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${LATEST:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${OUTPUTDIR:=/tmp}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${PROMPT:=true}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${RUNNING:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${RENAME:=true}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${TARBALL_CONFIG:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${TARBALL_COREDUMPS:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+: ${TARBALL_RESULTS:=false}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+COMMANDLINE_COREDUMPS=false</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Read config files from most important to least important.</span><br><span style="color: hsl(120, 100%, 40%);">+# Variables set on the command line or environment always take precedence.</span><br><span style="color: hsl(120, 100%, 40%);">+[ -f ./ast_debug_tools.conf ] && source ./ast_debug_tools.conf</span><br><span style="color: hsl(120, 100%, 40%);">+[ -f ~/ast_debug_tools.conf ] && source ~/ast_debug_tools.conf</span><br><span style="color: hsl(120, 100%, 40%);">+[ -f /etc/asterisk/ast_debug_tools.conf ] && source /etc/asterisk/ast_debug_tools.conf</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if [ -n "${DATEFORMAT}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ err <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ The DATEFORMAT variable in your ast_debug_tools.conf file has been</span><br><span style="color: hsl(120, 100%, 40%);">+ replaced with DATEOPTS which has a different format. See the latest</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_debug_tools.conf sample file for more information.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+ </span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+for a in "$@" ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ $a == "--RUNNING" ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ RUNNING=true</span><br><span style="color: hsl(120, 100%, 40%);">+ PROMPT=false</span><br><span style="color: hsl(120, 100%, 40%);">+ elif [[ $a =~ --no-([^=]+)$ ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ var=${BASH_REMATCH[1]//-/_}</span><br><span style="color: hsl(120, 100%, 40%);">+ eval ${var^^}="false"</span><br><span style="color: hsl(120, 100%, 40%);">+ elif [[ $a =~ --([^=]+)$ ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ var=${BASH_REMATCH[1]//-/_}</span><br><span style="color: hsl(120, 100%, 40%);">+ eval ${var^^}="true"</span><br><span style="color: hsl(120, 100%, 40%);">+ elif [[ $a =~ --([^=]+)=(.+)$ ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ var=${BASH_REMATCH[1]//-/_}</span><br><span style="color: hsl(120, 100%, 40%);">+ eval ${var^^}=${BASH_REMATCH[2]}</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ if ! $COMMANDLINE_COREDUMPS ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ COMMANDLINE_COREDUMPS=true</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS=()</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS+=( "$a" )</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+done</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if $HELP ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ print_help</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 0</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+check_gdb</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if [ -z "${ASTERISK_BIN}" -o ! -x "${ASTERISK_BIN}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ The asterisk binary specified (${ASTERISK_BIN})</span><br><span style="color: hsl(120, 100%, 40%);">+ was not found or is not executable. Use the '--asterisk-bin'</span><br><span style="color: hsl(120, 100%, 40%);">+ option to specify a valid binary.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if [ $EUID -ne 0 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -13 "You must be root to use $prog."</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if [ -z "${OUTPUTDIR}" -o ! -d "${OUTPUTDIR}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -20 "OUTPUTDIR ${OUTPUTDIR} doesn't exists or is not a directory"</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if $RUNNING ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ MAIN_PID=$(find_pid)</span><br><span style="color: hsl(120, 100%, 40%);">+ # If find_pid returns an error, the shell will automatically exit.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # We only want to process the coredump from the running process.</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS=( )</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ msg "Found a single asterisk instance running as process $MAIN_PID"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if $PROMPT ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ read -p "WARNING: Taking a core dump of the running asterisk instance will suspend call processing while the dump is saved. Do you wish to continue? (y/N) " answer</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ answer=Y</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ "$answer" =~ ^[Yy] ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ df=$(date ${DATEOPTS})</span><br><span style="color: hsl(120, 100%, 40%);">+ cf="${OUTPUTDIR}/core-asterisk-running-$df"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "$(S_COR ${DRY_RUN} 'Simulating dumping' 'Dumping') running asterisk process to $cf"</span><br><span style="color: hsl(120, 100%, 40%);">+ if ${DRY_RUN} ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Would run: ${GDB} ${ASTERISK_BIN} -p $MAIN_PID -q --batch --ex gcore $cf"</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ ${GDB} ${ASTERISK_BIN} -p $MAIN_PID -q --batch --ex "gcore $cf" >/dev/null 2>&1</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "$(S_COR ${DRY_RUN} 'Simulated dump' 'Dump') is complete."</span><br><span style="color: hsl(120, 100%, 40%);">+ </span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS=( "$cf" )</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ die -125 "Aborting dump of running process"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ </span><br><span style="color: hsl(120, 100%, 40%);">+ $DRY_RUN && exit 0</span><br><span style="color: hsl(120, 100%, 40%);">+else</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # At this point, all glob entries that match files should be expanded.</span><br><span style="color: hsl(120, 100%, 40%);">+ # Any entries that don't exist are probably globs that didn't match anything</span><br><span style="color: hsl(120, 100%, 40%);">+ # and need to be pruned. Any non coredumps are also pruned.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for i in ${!COREDUMPS[@]} ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ! -f "${COREDUMPS[$i]}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ unset "COREDUMPS[$i]"</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ # Some versions of 'file' don't allow only the first n bytes of the</span><br><span style="color: hsl(120, 100%, 40%);">+ # file to be processed so we use dd to grab just the first 32 bytes.</span><br><span style="color: hsl(120, 100%, 40%);">+ mimetype=$(dd if="${COREDUMPS[$i]}" bs=32 count=1 2>/dev/null | file -bi -)</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ ! "$mimetype" =~ coredump ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ unset "COREDUMPS[$i]"</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Let's make sure it's an asterisk coredump by dumping the notes</span><br><span style="color: hsl(120, 100%, 40%);">+ # section of the file and grepping for "asterisk".</span><br><span style="color: hsl(120, 100%, 40%);">+ readelf -n "${COREDUMPS[$i]}" | grep -q "asterisk" || {</span><br><span style="color: hsl(120, 100%, 40%);">+ unset "COREDUMPS[$i]"</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ${#COREDUMPS[@]} -eq 0 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 "No coredumps found"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Sort and weed out any dups</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS=( $(ls -t "${COREDUMPS[@]}" 2>/dev/null | uniq ) )</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ${#COREDUMPS[@]} -eq 0 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 "No coredumps found"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if $LATEST ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS=( "${COREDUMPS[0]}" )</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if [ ${#COREDUMPS[@]} -eq 0 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 "No coredumps found"</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Extract the gdb scripts from the end of this script</span><br><span style="color: hsl(120, 100%, 40%);">+# and save them to /tmp/.gdbinit, then add a trap to</span><br><span style="color: hsl(120, 100%, 40%);">+# clean it up.</span><br><span style="color: hsl(120, 100%, 40%);">+gdbinit=${OUTPUTDIR}/.ast_coredumper.gdbinit</span><br><span style="color: hsl(120, 100%, 40%);">+trap "rm $gdbinit" EXIT</span><br><span style="color: hsl(120, 100%, 40%);">+ss=`egrep -n "^#@@@SCRIPTSTART@@@" $0 |cut -f1 -d:`</span><br><span style="color: hsl(120, 100%, 40%);">+tail -n +${ss} $0 >$gdbinit</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Now iterate over the coredumps and dump the debugging info</span><br><span style="color: hsl(120, 100%, 40%);">+for i in "${!COREDUMPS[@]}" ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ cf=$(realpath -e ${COREDUMPS[$i]} || : )</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$cf" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Processing $cf"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if ! $RUNNING && ! [[ "$cf" =~ "running" ]] && $RENAME ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ df=$(date -r $cf ${DATEOPTS})</span><br><span style="color: hsl(120, 100%, 40%);">+ cfdir=$(dirname "$cf")</span><br><span style="color: hsl(120, 100%, 40%);">+ newcf="${cfdir}/core-asterisk-${df}"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ "${newcf}" != "${cf}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Renaming $cf to $cfdir/core-asterisk-${df}"</span><br><span style="color: hsl(120, 100%, 40%);">+ mv "$cf" "${cfdir}/core-asterisk-${df}"</span><br><span style="color: hsl(120, 100%, 40%);">+ cf="${cfdir}/core-asterisk-${df}"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ cfname=`basename ${cf}`</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Produce all the output files</span><br><span style="color: hsl(120, 100%, 40%);">+ ${GDB} -n --batch -q --ex "source $gdbinit" "${ASTERISK_BIN}" "$cf" 2>/dev/null | (</span><br><span style="color: hsl(120, 100%, 40%);">+ of=/dev/null</span><br><span style="color: hsl(120, 100%, 40%);">+ while IFS= read line ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ "$line" =~ !@!@!@!\ ([^\ ]+)\ !@!@!@! ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ of=${OUTPUTDIR}/${cfname}-${BASH_REMATCH[1]}</span><br><span style="color: hsl(120, 100%, 40%);">+ of=${of//:/-}</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -f "$of"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Creating $of"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ echo -e $"$line" >> "$of"</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+ )</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if $TARBALL_COREDUMPS ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ # We need to change occurrences of ':' to '-' because</span><br><span style="color: hsl(120, 100%, 40%);">+ # Jira won't let you attach a file with colons in the name.</span><br><span style="color: hsl(120, 100%, 40%);">+ cfname=${cfname//:/-}</span><br><span style="color: hsl(120, 100%, 40%);">+ tf=${OUTPUTDIR}/${cfname}.tar.gz</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Creating ${tf}"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ dest=${OUTPUTDIR}/${cfname}.output</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf ${dest} 2>/dev/null || :</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ libdir=""</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -n "${LIBDIR}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ LIBDIR=$(realpath "${LIBDIR}")</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ! -d "${LIBDIR}/asterisk/modules" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ ${LIBDIR}/asterisk/modules does not exist.</span><br><span style="color: hsl(120, 100%, 40%);">+ The library specified by --libdir or LIBDIR ${LIBDIR})</span><br><span style="color: hsl(120, 100%, 40%);">+ either does not exist or does not contain an "asterisk/modules" directory.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ libdir=${LIBDIR}</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ abits=$(file -b ${ASTERISK_BIN} | sed -n -r -e "s/.*(32|64)-bit.*/\1/p")</span><br><span style="color: hsl(120, 100%, 40%);">+ declare -a searchorder</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ $abits -eq 32 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ searchorder=( /lib /usr/lib /usr/lib32 /usr/local/lib )</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ searchorder=( /usr/lib64 /usr/local/lib64 /usr/lib /usr/local/lib /lib )</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ for d in ${searchorder[@]} ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ testmod="${d}/asterisk/modules/bridge_simple.so"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -e "${testmod}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ lbits=$(file -b ${ASTERISK_BIN} | sed -n -r -e "s/.*(32|64)-bit.*/\1/p")</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ $lbits -eq $abits ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ libdir=$d</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "${libdir}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ No standard systemlibrary directory contained asterisk modules.</span><br><span style="color: hsl(120, 100%, 40%);">+ Please specify the correct system library directory</span><br><span style="color: hsl(120, 100%, 40%);">+ with the --libdir option or the LIBDIR variable.</span><br><span style="color: hsl(120, 100%, 40%);">+ ${LIBDIR}/asterisk/modules must exist.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ mkdir -p ${dest}/tmp ${dest}/${libdir}/asterisk ${dest}/etc ${dest}/usr/sbin</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ln -s ${cf} ${dest}/tmp/${cfname}</span><br><span style="color: hsl(120, 100%, 40%);">+ cp ${OUTPUTDIR}/${cfname}*.txt ${dest}/tmp/</span><br><span style="color: hsl(120, 100%, 40%);">+ [ -f /etc/os-release ] && cp /etc/os-release ${dest}/etc/</span><br><span style="color: hsl(120, 100%, 40%);">+ if $TARBALL_CONFIG ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ cp -a /etc/asterisk ${dest}/etc/</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ cp -a /${libdir}/libasterisk* ${dest}/${libdir}/</span><br><span style="color: hsl(120, 100%, 40%);">+ cp -a /${libdir}/asterisk/* ${dest}/${libdir}/asterisk/</span><br><span style="color: hsl(120, 100%, 40%);">+ cp -a /usr/sbin/asterisk ${dest}/usr/sbin</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf ${tf}</span><br><span style="color: hsl(120, 100%, 40%);">+ tar -chzf ${tf} --transform="s/^[.]/${cfname}.output/" -C ${dest} .</span><br><span style="color: hsl(120, 100%, 40%);">+ sleep 3</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf ${dest}</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Created $tf"</span><br><span style="color: hsl(120, 100%, 40%);">+ elif $TARBALL_RESULTS ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ cfname=${cfname//:/-}</span><br><span style="color: hsl(120, 100%, 40%);">+ tf=${OUTPUTDIR}/${cfname}.tar.gz</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Creating ${tf}"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ dest=${OUTPUTDIR}/${cfname}.output</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf ${dest} 2>/dev/null || :</span><br><span style="color: hsl(120, 100%, 40%);">+ mkdir -p ${dest}</span><br><span style="color: hsl(120, 100%, 40%);">+ cp ${OUTPUTDIR}/${cfname}*.txt ${dest}/</span><br><span style="color: hsl(120, 100%, 40%);">+ if $TARBALL_CONFIG ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ mkdir -p ${dest}/etc</span><br><span style="color: hsl(120, 100%, 40%);">+ cp -a /etc/asterisk ${dest}/etc/</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ tar -chzf ${tf} --transform="s/^[.]/${cfname}/" -C ${dest} .</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf ${dest}</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Created $tf"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if $DELETE_COREDUMPS_AFTER ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf "${cf}"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if $DELETE_RESULTS_AFTER ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -rf "${cf//:/-}"-{brief,full,thread1,locks,info}.txt</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+done</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+exit</span><br><span style="color: hsl(120, 100%, 40%);">+# @formatter:off</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#@@@FUNCSSTART@@@</span><br><span> print_help() {</span><br><span style="color: hsl(0, 100%, 40%);">-cat <<EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ sed -n -r -e "/^#@@@HELPSTART@@@/,\${p;/^#@@@HELPEND@@@/q}" $0 | sed '1d;$d'</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 1</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+err() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$1" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ cat >&2</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "$1" >&2</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+msg() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$1" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ cat</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "$1"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+die() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ $1 =~ ^-([0-9]+) ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ RC=${BASH_REMATCH[1]}</span><br><span style="color: hsl(120, 100%, 40%);">+ shift</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ err "$1"</span><br><span style="color: hsl(120, 100%, 40%);">+ exit ${RC:-1}</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+S_COR() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if $1 ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo -n "$2"</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ echo -n "$3"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+check_gdb() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "${GDB}" -o ! -x "${GDB}" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ ${GDB} seems to not be installed.</span><br><span style="color: hsl(120, 100%, 40%);">+ Please install gdb or use the '--gdb' option to</span><br><span style="color: hsl(120, 100%, 40%);">+ point to a valid executable.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ result=$($GDB --batch --ex "python print('hello')" 2>/dev/null || : )</span><br><span style="color: hsl(120, 100%, 40%);">+ if [[ ! "$result" =~ ^hello$ ]] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -2 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ $GDB does not support python.</span><br><span style="color: hsl(120, 100%, 40%);">+ Use the '--gdb' option to point to one that does.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+find_pid() {</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -n "$PID" ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ # Make sure it's at least all numeric</span><br><span style="color: hsl(120, 100%, 40%);">+ [[ $PID =~ ^[0-9]+$ ]] || die -22 $"Pid $PID is invalid."</span><br><span style="color: hsl(120, 100%, 40%);">+ # Make sure it exists</span><br><span style="color: hsl(120, 100%, 40%);">+ cmd=$(ps -p $PID -o comm=) || die -22 "Pid $PID is not a valid process."</span><br><span style="color: hsl(120, 100%, 40%);">+ # Make sure the program (without path) is "asterisk"</span><br><span style="color: hsl(120, 100%, 40%);">+ [ "$cmd" == "asterisk" ] || die -22 "Pid $PID is '$cmd' not 'asterisk'."</span><br><span style="color: hsl(120, 100%, 40%);">+ echo $PID</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Some versions of pgrep can't display the program arguments</span><br><span style="color: hsl(120, 100%, 40%);">+ # so we'll just get the pids that exactly match a program</span><br><span style="color: hsl(120, 100%, 40%);">+ # name of "asterisk".</span><br><span style="color: hsl(120, 100%, 40%);">+ pids=$( pgrep -d ',' -x "asterisk")</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z ${pids} ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -3 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ No running asterisk instances detected.</span><br><span style="color: hsl(120, 100%, 40%);">+ If you know the pid of the process you want to dump,</span><br><span style="color: hsl(120, 100%, 40%);">+ supply it on the command line with --pid=<pid>.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Now that we have the pids, let's get the command and</span><br><span style="color: hsl(120, 100%, 40%);">+ # its args. We'll add them to an array indexed by pid.</span><br><span style="color: hsl(120, 100%, 40%);">+ declare -a candidates</span><br><span style="color: hsl(120, 100%, 40%);">+ while read LINE ; do</span><br><span style="color: hsl(120, 100%, 40%);">+ [[ $LINE =~ ([0-9]+)[\ ]+([^\ ]+)[\ ]+(.*) ]] || continue</span><br><span style="color: hsl(120, 100%, 40%);">+ pid=${BASH_REMATCH[1]}</span><br><span style="color: hsl(120, 100%, 40%);">+ prog=${BASH_REMATCH[2]}</span><br><span style="color: hsl(120, 100%, 40%);">+ args=${BASH_REMATCH[3]}</span><br><span style="color: hsl(120, 100%, 40%);">+ # If you run "asterisk -(rRx)", pgrep will find the process (which we</span><br><span style="color: hsl(120, 100%, 40%);">+ # really don't want) but thankfully, asterisk.c resets argv[0] to</span><br><span style="color: hsl(120, 100%, 40%);">+ # "rasterisk" so the output of ps will show that. This is an easy</span><br><span style="color: hsl(120, 100%, 40%);">+ # filter to weed out remote consoles.</span><br><span style="color: hsl(120, 100%, 40%);">+ [[ "$prog" == "rasterisk" ]] && continue;</span><br><span style="color: hsl(120, 100%, 40%);">+ candidates[$pid]="${prog}^${args}"</span><br><span style="color: hsl(120, 100%, 40%);">+ done < <(ps -o pid= -o command= -p $pids)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ${#candidates[@]} -eq 0 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -3 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ No running asterisk instances detected.</span><br><span style="color: hsl(120, 100%, 40%);">+ If you know the pid of the process you want to dump,</span><br><span style="color: hsl(120, 100%, 40%);">+ supply it on the command line with --pid=<pid>.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ ${#candidates[@]} -gt 1 ] ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ die -22 <<-EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ Detected more than one asterisk process running.</span><br><span style="color: hsl(120, 100%, 40%);">+ $(printf "%8s %s\n" "PID" "COMMAND")</span><br><span style="color: hsl(120, 100%, 40%);">+ $(for p in ${!candidates[@]} ; do printf "%8s %s\n" $p "${candidates[$p]//^/ }" ; done )</span><br><span style="color: hsl(120, 100%, 40%);">+ If you know the pid of the process you want to dump,</span><br><span style="color: hsl(120, 100%, 40%);">+ supply it on the command line with --pid=<pid>.</span><br><span style="color: hsl(120, 100%, 40%);">+ EOF</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ echo ${!candidates[@]}</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+#@@@FUNCSEND@@@</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#@@@HELPSTART@@@</span><br><span> NAME</span><br><span> $prog - Dump and/or format asterisk coredump files</span><br><span> </span><br><span> SYNOPSIS</span><br><span style="color: hsl(0, 100%, 40%);">- $prog [ --help ] [ --running | --RUNNING ] [ --latest ]</span><br><span style="color: hsl(120, 100%, 40%);">+ $prog [ --help ] [ --running | --RUNNING ] [ --pid="pid" ]</span><br><span style="color: hsl(120, 100%, 40%);">+ [ --latest ] [ --OUTPUTDIR="path" ]</span><br><span style="color: hsl(120, 100%, 40%);">+ [ --libdir="path" ] [ --asterisk-bin="path" ]</span><br><span style="color: hsl(120, 100%, 40%);">+ [ --gdb="path" ] [ --rename ] [ --dateformat="date options" ]</span><br><span> [ --tarball-coredumps ] [ --delete-coredumps-after ]</span><br><span> [ --tarball-results ] [ --delete-results-after ]</span><br><span style="color: hsl(0, 100%, 40%);">- [ --tarball-config ] [ --tarball-uniqueid="<uniqueid>" ]</span><br><span style="color: hsl(0, 100%, 40%);">- [ --no-default-search ] [ --append-coredumps ]</span><br><span style="color: hsl(0, 100%, 40%);">- [ --asterisk-bin="path" ]</span><br><span style="color: hsl(120, 100%, 40%);">+ [ --tarball-config ]</span><br><span> [ <coredump> | <pattern> ... ]</span><br><span> </span><br><span> DESCRIPTION</span><br><span> </span><br><span> Extracts backtraces and lock tables from Asterisk coredump files.</span><br><span style="color: hsl(0, 100%, 40%);">- For each coredump found, 4 new result files are created:</span><br><span style="color: hsl(0, 100%, 40%);">- - <coredump>.brief.txt: The output of "thread apply all bt".</span><br><span style="color: hsl(120, 100%, 40%);">+ For each coredump found, 5 new result files are created:</span><br><span style="color: hsl(120, 100%, 40%);">+ - <coredump>-brief.txt: The output of "thread apply all bt".</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- - <coredump>.thread1.txt: The output of "thread apply 1 bt full".</span><br><span style="color: hsl(120, 100%, 40%);">+ - <coredump>-full.txt: The output of "thread apply all bt full".</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- - <coredump>.full.txt: The output of "thread apply all bt full".</span><br><span style="color: hsl(120, 100%, 40%);">+ - <coredump>-info.txt: State info like taskprocessors, channels, etc</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- - <coredump>.locks.txt: If asterisk was compiled with</span><br><span style="color: hsl(120, 100%, 40%);">+ - <coredump>-locks.txt: If asterisk was compiled with</span><br><span> "DEBUG_THREADS", this file will contain a dump of the locks</span><br><span> table similar to doing a "core show locks" from the asterisk</span><br><span> CLI.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- Optional features:</span><br><span style="color: hsl(0, 100%, 40%);">- - The running asterisk process can be suspended and dumped.</span><br><span style="color: hsl(0, 100%, 40%);">- - The coredumps can be merged into a tarball.</span><br><span style="color: hsl(0, 100%, 40%);">- - The coredumps can be deleted after processing.</span><br><span style="color: hsl(0, 100%, 40%);">- - The results files can be merged into a tarball.</span><br><span style="color: hsl(0, 100%, 40%);">- - The results files can be deleted after processing.</span><br><span style="color: hsl(120, 100%, 40%);">+ - <coredump>-thread1.txt: The output of "thread apply 1 bt full".</span><br><span> </span><br><span> Options:</span><br><span> </span><br><span>@@ -49,32 +481,62 @@</span><br><span> </span><br><span> --running</span><br><span> Create a coredump from the running asterisk instance and</span><br><span style="color: hsl(0, 100%, 40%);">- process it along with any other coredumps found (if any).</span><br><span style="color: hsl(120, 100%, 40%);">+ process it.</span><br><span> WARNING: This WILL interrupt call processing. You will be</span><br><span style="color: hsl(0, 100%, 40%);">- asked to confirm. The coredump will be written to /tmp if</span><br><span style="color: hsl(0, 100%, 40%);">- $OUTPUTDIR is not defined.</span><br><span style="color: hsl(120, 100%, 40%);">+ asked to confirm.</span><br><span> </span><br><span> --RUNNING</span><br><span> Same as --running but without the confirmation prompt.</span><br><span> DANGEROUS!!</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ --pid=<asterisk main process pid></span><br><span style="color: hsl(120, 100%, 40%);">+ If you are trying to get a dump of the running asterisk</span><br><span style="color: hsl(120, 100%, 40%);">+ instance, specifying its pid on the command line will</span><br><span style="color: hsl(120, 100%, 40%);">+ bypass the complex logic used to figure it out.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> --latest</span><br><span> Process only the latest coredump from those specified (based</span><br><span style="color: hsl(0, 100%, 40%);">- on last-modified time). If a dump of the running process was</span><br><span style="color: hsl(0, 100%, 40%);">- requested, it is always included in addition to the latest</span><br><span style="color: hsl(0, 100%, 40%);">- from the existing coredumps.</span><br><span style="color: hsl(120, 100%, 40%);">+ on last-modified time). Only needed when --running was not</span><br><span style="color: hsl(120, 100%, 40%);">+ specified and there is more that one coredump matched.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --outputdir=<output directory></span><br><span style="color: hsl(120, 100%, 40%);">+ The directory into which output products will be saved.</span><br><span style="color: hsl(120, 100%, 40%);">+ Default: same directory as coredump</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --libdir=<shared libs directory></span><br><span style="color: hsl(120, 100%, 40%);">+ The directory where the libasterisk* shared libraries and</span><br><span style="color: hsl(120, 100%, 40%);">+ the asterisk/modules directory are located. The common</span><br><span style="color: hsl(120, 100%, 40%);">+ directories like /usr/lib, /usr/lib64, etc are automatically</span><br><span style="color: hsl(120, 100%, 40%);">+ searches so only use this option when your asterisk install</span><br><span style="color: hsl(120, 100%, 40%);">+ is non-standard.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --asterisk-bin=<asterisk binary></span><br><span style="color: hsl(120, 100%, 40%);">+ Path to the asterisk binary.</span><br><span style="color: hsl(120, 100%, 40%);">+ Default: look for asterisk in the PATH.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --gdb=<path_to_gdb></span><br><span style="color: hsl(120, 100%, 40%);">+ gdb must have python support built-in. Most do.</span><br><span style="color: hsl(120, 100%, 40%);">+ Default: /usr/bin/gdb</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --dateformat=<date options></span><br><span style="color: hsl(120, 100%, 40%);">+ Passed to the 'date' utility to construct dates.</span><br><span style="color: hsl(120, 100%, 40%);">+ The default is '-u +%FT%H-%M-%SZ' which results</span><br><span style="color: hsl(120, 100%, 40%);">+ in a UTC timestamp.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ --rename</span><br><span style="color: hsl(120, 100%, 40%);">+ Causes the coredump to be renamed using DATEOPTS</span><br><span style="color: hsl(120, 100%, 40%);">+ and the output files to be named accordingly.</span><br><span style="color: hsl(120, 100%, 40%);">+ This is the default. To disable renaming, specify</span><br><span style="color: hsl(120, 100%, 40%);">+ --no-rename</span><br><span> </span><br><span> --tarball-coredumps</span><br><span style="color: hsl(0, 100%, 40%);">- Creates a gzipped tarball of coredumps processed, their</span><br><span style="color: hsl(0, 100%, 40%);">- results txt files and copies of /etc/os-release,</span><br><span style="color: hsl(0, 100%, 40%);">- /usr/sbin/asterisk, /usr/lib(64)/libasterisk* and</span><br><span style="color: hsl(0, 100%, 40%);">- /usr/lib(64)/asterisk as those files are needed to properly</span><br><span style="color: hsl(0, 100%, 40%);">- examine the coredump. The file will be named</span><br><span style="color: hsl(0, 100%, 40%);">- $OUTPUTDIR/asterisk.<timestamp>.coredumps.tar.gz or</span><br><span style="color: hsl(0, 100%, 40%);">- $OUTPUTDIR/asterisk-<uniqueid>.coredumps.tar.gz if</span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-uniqueid was specified.</span><br><span style="color: hsl(0, 100%, 40%);">- WARNING: This file could 1gb in size!</span><br><span style="color: hsl(0, 100%, 40%);">- Mutually exclusive with --tartball-results</span><br><span style="color: hsl(120, 100%, 40%);">+ Creates a gzipped tarball of each coredump processed, their</span><br><span style="color: hsl(120, 100%, 40%);">+ results txt files, a copy of /etc/os-release, the</span><br><span style="color: hsl(120, 100%, 40%);">+ asterisk binary, and all modules.</span><br><span style="color: hsl(120, 100%, 40%);">+ The file will be named like the coredump with '.tar.gz'</span><br><span style="color: hsl(120, 100%, 40%);">+ appended.</span><br><span style="color: hsl(120, 100%, 40%);">+ WARNING: This file could be quite large!</span><br><span style="color: hsl(120, 100%, 40%);">+ Mutually exclusive with --tarball-results</span><br><span> </span><br><span> --delete-coredumps-after</span><br><span> Deletes all processed coredumps regardless of whether</span><br><span>@@ -84,11 +546,11 @@</span><br><span> Creates a gzipped tarball of all result files produced.</span><br><span> The tarball name will be:</span><br><span> $OUTPUTDIR/asterisk.<timestamp>.results.tar.gz</span><br><span style="color: hsl(0, 100%, 40%);">- Mutually exclusive with --tartball-coredumps</span><br><span style="color: hsl(120, 100%, 40%);">+ Mutually exclusive with --tarball-coredumps</span><br><span> </span><br><span> --delete-results-after</span><br><span> Deletes all processed results regardless of whether</span><br><span style="color: hsl(0, 100%, 40%);">- a tarball was created. It probably doesn't make sense</span><br><span style="color: hsl(120, 100%, 40%);">+ a tarball was created. It probably does not make sense</span><br><span> to use this option unless you have also specified</span><br><span> --tarball-results.</span><br><span> </span><br><span>@@ -96,54 +558,32 @@</span><br><span> Adds the contents of /etc/asterisk to the tarball created</span><br><span> with --tarball-coredumps or --tarball-results.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-uniqueid="<uniqueid>"</span><br><span style="color: hsl(0, 100%, 40%);">- Normally DATEFORMAT is used to make the tarballs unique</span><br><span style="color: hsl(0, 100%, 40%);">- but you can use your own unique id in the tarball names</span><br><span style="color: hsl(0, 100%, 40%);">- such as the Jira issue id.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- --no-default-search</span><br><span style="color: hsl(0, 100%, 40%);">- Ignore COREDUMPS from the config files and process only</span><br><span style="color: hsl(0, 100%, 40%);">- coredumps listed on the command line (if any) and/or</span><br><span style="color: hsl(0, 100%, 40%);">- the running asterisk instance (if requested).</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- --append-coredumps</span><br><span style="color: hsl(0, 100%, 40%);">- Append any coredumps specified on the command line to the</span><br><span style="color: hsl(0, 100%, 40%);">- config file specified ones instead of overriding them.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- --asterisk-binary</span><br><span style="color: hsl(0, 100%, 40%);">- Path to the asterisk binary. Default: look for asterisk</span><br><span style="color: hsl(0, 100%, 40%);">- in the PATH.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> <coredump> | <pattern></span><br><span style="color: hsl(0, 100%, 40%);">- A list of coredumps or coredump search patterns. Unless</span><br><span style="color: hsl(0, 100%, 40%);">- --append-coredumps was specified, these entries will override</span><br><span style="color: hsl(0, 100%, 40%);">- those specified in the config files.</span><br><span style="color: hsl(120, 100%, 40%);">+ A list of coredumps or coredump search patterns. These</span><br><span style="color: hsl(120, 100%, 40%);">+ will override the default and those specified in the config files.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- Any resulting file that isn't actually a coredump is silently</span><br><span style="color: hsl(0, 100%, 40%);">- ignored. If your patterns contains spaces be sure to only</span><br><span style="color: hsl(0, 100%, 40%);">- quote the portion of the pattern that DOESN'T contain wildcard</span><br><span style="color: hsl(0, 100%, 40%);">- expressions. If you quote the whole pattern, it won't be</span><br><span style="color: hsl(0, 100%, 40%);">- expanded.</span><br><span style="color: hsl(120, 100%, 40%);">+ The default pattern is "/tmp/core!(*.txt)"</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- If --no-default-search is specified and no files are specified</span><br><span style="color: hsl(0, 100%, 40%);">- on the command line, then the only the running asterisk process</span><br><span style="color: hsl(0, 100%, 40%);">- will be dumped (if requested). Otherwise if no files are</span><br><span style="color: hsl(0, 100%, 40%);">- specified on the command line the value of COREDUMPS from</span><br><span style="color: hsl(0, 100%, 40%);">- ast_debug_tools.conf will be used. Failing that, the following</span><br><span style="color: hsl(0, 100%, 40%);">- patterns will be used:</span><br><span style="color: hsl(0, 100%, 40%);">- /tmp/core[-._]asterisk!(*.txt)</span><br><span style="color: hsl(0, 100%, 40%);">- /tmp/core[-._]\$(hostname)!(*.txt)</span><br><span style="color: hsl(120, 100%, 40%);">+ The "!(*.txt)" tells bash to ignore any files that match</span><br><span style="color: hsl(120, 100%, 40%);">+ the base pattern and end in ".txt". It$'s not strictly</span><br><span style="color: hsl(120, 100%, 40%);">+ needed as non asterisk coredumps are always ignored.</span><br><span> </span><br><span> NOTES</span><br><span style="color: hsl(0, 100%, 40%);">- You must be root to use $prog.</span><br><span style="color: hsl(120, 100%, 40%);">+ You must be root to use this program.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- $OUTPUTDIR can be read from the current environment or from the</span><br><span style="color: hsl(0, 100%, 40%);">- ast_debug_tools.conf file described below. If not specified,</span><br><span style="color: hsl(0, 100%, 40%);">- work products are placed in the same directory as the core file.</span><br><span style="color: hsl(120, 100%, 40%);">+ All options except "running", "RUNNING" and "pid" can be</span><br><span style="color: hsl(120, 100%, 40%);">+ specified in the ast_debug_tools.conf file.</span><br><span style="color: hsl(120, 100%, 40%);">+ Option names must be translated to upper case and their '-'</span><br><span style="color: hsl(120, 100%, 40%);">+ characters replaced by '_'. Boolean options must be set to</span><br><span style="color: hsl(120, 100%, 40%);">+ 'true' or 'false' (lower case, without the quotes).</span><br><span style="color: hsl(120, 100%, 40%);">+ Examples:</span><br><span style="color: hsl(120, 100%, 40%);">+ TARBALL_RESULTS=true</span><br><span style="color: hsl(120, 100%, 40%);">+ RENAME=false</span><br><span style="color: hsl(120, 100%, 40%);">+ ASTERISK_BIN=/usr/sbin/asterisk</span><br><span> </span><br><span> The script relies on not only bash, but also recent GNU date and</span><br><span> gdb with python support. *BSD operating systems may require</span><br><span style="color: hsl(0, 100%, 40%);">- installation of the 'coreutils' and 'devel/gdb' packagess and minor</span><br><span style="color: hsl(120, 100%, 40%);">+ installation of the 'coreutils' and 'devel/gdb' packages and minor</span><br><span> tweaking of the ast_debug_tools.conf file.</span><br><span> </span><br><span> Any files output will have ':' characters changed to '-'. This is</span><br><span>@@ -155,403 +595,10 @@</span><br><span> ~/ast_debug_tools.conf</span><br><span> ./ast_debug_tools.conf</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # This file is used by the Asterisk debug tools.</span><br><span style="color: hsl(0, 100%, 40%);">- # Unlike other Asterisk config files, this one is</span><br><span style="color: hsl(0, 100%, 40%);">- # "sourced" by bash and must adhere to bash semantics.</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(120, 100%, 40%);">+ See the configs/samples/ast_debug_tools.conf file in the asterisk</span><br><span style="color: hsl(120, 100%, 40%);">+ source tree for more info.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- # A list of coredumps and/or coredump search patterns.</span><br><span style="color: hsl(0, 100%, 40%);">- # Bash extended globs are enabled and any resulting files</span><br><span style="color: hsl(0, 100%, 40%);">- # that aren't actually coredumps are silently ignored</span><br><span style="color: hsl(0, 100%, 40%);">- # so you can be liberal with the globs.</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # If your patterns contains spaces be sure to only quote</span><br><span style="color: hsl(0, 100%, 40%);">- # the portion of the pattern that DOESN'T contain wildcard</span><br><span style="color: hsl(0, 100%, 40%);">- # expressions. If you quote the whole pattern, it won't</span><br><span style="color: hsl(0, 100%, 40%);">- # be expanded and the glob characters will be treated as</span><br><span style="color: hsl(0, 100%, 40%);">- # literals.</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # The exclusion of files ending ".txt" is just for</span><br><span style="color: hsl(0, 100%, 40%);">- # demonstration purposes as non-coredumps will be ignored</span><br><span style="color: hsl(0, 100%, 40%);">- # anyway.</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS=(/tmp/core[-._]asterisk!(*.txt) /tmp/core[-._]\$(hostname)!(*.txt))</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # The directory to contain output files and work directories.</span><br><span style="color: hsl(0, 100%, 40%);">- # For output from existing core files, the default is the</span><br><span style="color: hsl(0, 100%, 40%);">- # directory that the core file is found in. For core files</span><br><span style="color: hsl(0, 100%, 40%);">- # produced from a running process, the default is /tmp.</span><br><span style="color: hsl(0, 100%, 40%);">- OUTPUTDIR=/some/directory</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # Date command for the "running" coredump and tarballs.</span><br><span style="color: hsl(0, 100%, 40%);">- # DATEFORMAT will be executed to get the timestamp.</span><br><span style="color: hsl(0, 100%, 40%);">- # Don't put quotes around the format string or they'll be</span><br><span style="color: hsl(0, 100%, 40%);">- # treated as literal characters. Also be aware of colons</span><br><span style="color: hsl(0, 100%, 40%);">- # in the output as you can't upload files with colons in</span><br><span style="color: hsl(0, 100%, 40%);">- # the name to Jira.</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # Unix timestamp</span><br><span style="color: hsl(0, 100%, 40%);">- #DATEFORMAT='date +%s.%N'</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # *BSD/MacOS doesn't support %N but after installing GNU</span><br><span style="color: hsl(0, 100%, 40%);">- # coreutils...</span><br><span style="color: hsl(0, 100%, 40%);">- #DATEFORMAT='gdate +%s.%N'</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # Readable GMT</span><br><span style="color: hsl(0, 100%, 40%);">- #DATEFORMAT='date -u +%FT%H-%M-%S%z'</span><br><span style="color: hsl(0, 100%, 40%);">- #</span><br><span style="color: hsl(0, 100%, 40%);">- # Readable Local time</span><br><span style="color: hsl(0, 100%, 40%);">- DATEFORMAT='date +%FT%H-%M-%S%z'</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-EOF</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">-}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ $EUID -ne 0 ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "You must be root to use $prog."</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-running=false</span><br><span style="color: hsl(0, 100%, 40%);">-RUNNING=false</span><br><span style="color: hsl(0, 100%, 40%);">-latest=false</span><br><span style="color: hsl(0, 100%, 40%);">-tarball_coredumps=false</span><br><span style="color: hsl(0, 100%, 40%);">-tarball_config=false</span><br><span style="color: hsl(0, 100%, 40%);">-delete_coredumps_after=false</span><br><span style="color: hsl(0, 100%, 40%);">-tarball_results=false</span><br><span style="color: hsl(0, 100%, 40%);">-delete_results_after=false</span><br><span style="color: hsl(0, 100%, 40%);">-append_coredumps=false</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-declare -a COREDUMPS</span><br><span style="color: hsl(0, 100%, 40%);">-declare -a ARGS_COREDUMPS</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# readconf reads a bash-sourceable file and sets variables</span><br><span style="color: hsl(0, 100%, 40%);">-# that havn't already been set. This allows variables set</span><br><span style="color: hsl(0, 100%, 40%);">-# on the command line or that are already in the environment</span><br><span style="color: hsl(0, 100%, 40%);">-# to take precedence over those read from the file.</span><br><span style="color: hsl(0, 100%, 40%);">-#</span><br><span style="color: hsl(0, 100%, 40%);">-# Setting the values can't be done in a subshell so you can't</span><br><span style="color: hsl(0, 100%, 40%);">-# just pipe the output of sed into the while.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-readconf() {</span><br><span style="color: hsl(0, 100%, 40%);">- while read line ; do</span><br><span style="color: hsl(0, 100%, 40%);">- v=${line%%=*}</span><br><span style="color: hsl(0, 100%, 40%);">- [ -z "${!v}" ] && eval $line || :</span><br><span style="color: hsl(0, 100%, 40%);">- done <<EOF</span><br><span style="color: hsl(0, 100%, 40%);">-$( sed -r -e "/\s*#/d" -e "/^\s*$/d" $1 )</span><br><span style="color: hsl(0, 100%, 40%);">-EOF</span><br><span style="color: hsl(0, 100%, 40%);">-}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Read config files from most important to least important.</span><br><span style="color: hsl(0, 100%, 40%);">-# Variable set on the command line or environment always take precedence.</span><br><span style="color: hsl(0, 100%, 40%);">-[ -f ./ast_debug_tools.conf ] && readconf ./ast_debug_tools.conf</span><br><span style="color: hsl(0, 100%, 40%);">-[ -f ~/ast_debug_tools.conf ] && readconf ~/ast_debug_tools.conf</span><br><span style="color: hsl(0, 100%, 40%);">-[ -f /etc/asterisk/ast_debug_tools.conf ] && readconf /etc/asterisk/ast_debug_tools.conf</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# For *BSD, the preferred gdb may be in /usr/local/bin so we</span><br><span style="color: hsl(0, 100%, 40%);">-# need to search for one that supports python.</span><br><span style="color: hsl(0, 100%, 40%);">-for g in $(which -a gdb) ; do</span><br><span style="color: hsl(0, 100%, 40%);">- result=$($g --batch --ex "python print('hello')" 2>/dev/null || : )</span><br><span style="color: hsl(0, 100%, 40%);">- if [[ "$result" =~ ^hello$ ]] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- GDB=$g</span><br><span style="color: hsl(0, 100%, 40%);">- break</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ -z "$GDB" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "No suitable gdb was found in $PATH"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ -n "$OUTPUTDIR" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- if [ ! -d "$OUTPUTDIR" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "OUTPUTDIR $OUTPUTDIR doesn't exists or is not a directory"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ ${#COREDUMPS[@]} -eq 0 ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS+=(/tmp/core[-._]asterisk!(*.txt) /tmp/core[-._]$(hostname)!(*.txt))</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-DATEFORMAT=${DATEFORMAT:-'date +%FT%H-%M-%S%z'}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Use "$@" (with the quotes) so spaces in patterns or</span><br><span style="color: hsl(0, 100%, 40%);">-# file names are preserved.</span><br><span style="color: hsl(0, 100%, 40%);">-# Later on when we have to iterate over COREDUMPS, we always</span><br><span style="color: hsl(0, 100%, 40%);">-# use the indexes rather than trying to expand the values of COREDUMPS</span><br><span style="color: hsl(0, 100%, 40%);">-# just in case.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-for a in "$@" ; do</span><br><span style="color: hsl(0, 100%, 40%);">- case "$a" in</span><br><span style="color: hsl(0, 100%, 40%);">- --running)</span><br><span style="color: hsl(0, 100%, 40%);">- running=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --RUNNING)</span><br><span style="color: hsl(0, 100%, 40%);">- RUNNING=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --no-default-search)</span><br><span style="color: hsl(0, 100%, 40%);">- # Clean out COREDUMPS from config files</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS=()</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --latest)</span><br><span style="color: hsl(0, 100%, 40%);">- latest=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-coredumps)</span><br><span style="color: hsl(0, 100%, 40%);">- tarball_coredumps=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-config)</span><br><span style="color: hsl(0, 100%, 40%);">- tarball_config=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --delete-coredumps-after)</span><br><span style="color: hsl(0, 100%, 40%);">- delete_coredumps_after=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-results)</span><br><span style="color: hsl(0, 100%, 40%);">- tarball_results=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --delete-results-after)</span><br><span style="color: hsl(0, 100%, 40%);">- delete_results_after=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --append-coredumps)</span><br><span style="color: hsl(0, 100%, 40%);">- append_coredumps=true</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --tarball-uniqueid=*)</span><br><span style="color: hsl(0, 100%, 40%);">- tarball_uniqueid=${a#*=}</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --asterisk-bin=*)</span><br><span style="color: hsl(0, 100%, 40%);">- asterisk_bin=${a#*=}</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- --help|-*)</span><br><span style="color: hsl(0, 100%, 40%);">- print_help</span><br><span style="color: hsl(0, 100%, 40%);">- ;;</span><br><span style="color: hsl(0, 100%, 40%);">- *)</span><br><span style="color: hsl(0, 100%, 40%);">- ARGS_COREDUMPS+=("$a")</span><br><span style="color: hsl(0, 100%, 40%);">- # If any files are specified on the command line, ignore those</span><br><span style="color: hsl(0, 100%, 40%);">- # specified in the config files unless append-coredumps was specified.</span><br><span style="color: hsl(0, 100%, 40%);">- if ! $append_coredumps ; then</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS=()</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- esac</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# append coredumps/patterns specified as command line arguments to COREDUMPS.</span><br><span style="color: hsl(0, 100%, 40%);">-for i in ${!ARGS_COREDUMPS[@]} ; do</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS+=("${ARGS_COREDUMPS[$i]}")</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# At this point, all glob entries that match files should be expanded.</span><br><span style="color: hsl(0, 100%, 40%);">-# Any entries that don't exist are probably globs that didn't match anything</span><br><span style="color: hsl(0, 100%, 40%);">-# and need to be pruned. Any non coredumps are also pruned.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-for i in ${!COREDUMPS[@]} ; do</span><br><span style="color: hsl(0, 100%, 40%);">- if [ ! -f "${COREDUMPS[$i]}" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- unset COREDUMPS[$i]</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- # Some versions of 'file' don't allow only the first n bytes of the</span><br><span style="color: hsl(0, 100%, 40%);">- # file to be processed so we use dd to grab just the first 32 bytes.</span><br><span style="color: hsl(0, 100%, 40%);">- mimetype=$(dd if="${COREDUMPS[$i]}" bs=32 count=1 2>/dev/null | file -bi -)</span><br><span style="color: hsl(0, 100%, 40%);">- if [[ ! "$mimetype" =~ coredump ]] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- unset COREDUMPS[$i]</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Sort and weed out any dups</span><br><span style="color: hsl(0, 100%, 40%);">-IFS=$'\x0a'</span><br><span style="color: hsl(0, 100%, 40%);">-readarray -t COREDUMPS < <(echo -n "${COREDUMPS[*]}" | sort -u )</span><br><span style="color: hsl(0, 100%, 40%);">-unset IFS</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# If --latest, get the last modified timestamp of each file,</span><br><span style="color: hsl(0, 100%, 40%);">-# sort them, then return the latest.</span><br><span style="color: hsl(0, 100%, 40%);">-if [ ${#COREDUMPS[@]} -gt 0 ] && $latest ; then</span><br><span style="color: hsl(0, 100%, 40%);">- lf=$(find "${COREDUMPS[@]}" -printf '%T@ %p\n' | sort -n | tail -1)</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS=("${lf#* }")</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Timestamp to use for output files</span><br><span style="color: hsl(0, 100%, 40%);">-df=${tarball_uniqueid:-$(${DATEFORMAT})}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ x"$asterisk_bin" = x ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- asterisk_bin=$(which asterisk)</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if $running || $RUNNING ; then</span><br><span style="color: hsl(0, 100%, 40%);">- # We need to go through some gyrations to find the pid of the running</span><br><span style="color: hsl(0, 100%, 40%);">- # MAIN asterisk process and not someone or something running asterisk -r.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- unset pid</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # Simplest case first...</span><br><span style="color: hsl(0, 100%, 40%);">- pids=$(pgrep -f "$asterisk_bin" || : )</span><br><span style="color: hsl(0, 100%, 40%);">- pidcount=$(echo $pids | wc -w)</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # Single process, great.</span><br><span style="color: hsl(0, 100%, 40%);">- if [ $pidcount -eq 1 ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- pid=$pids</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Found a single asterisk instance running as process $pid"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # More than 1 asterisk process running</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$pid" = x ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- # More than 1 process running, let's try asking asterisk for it's</span><br><span style="color: hsl(0, 100%, 40%);">- # pidfile</span><br><span style="color: hsl(0, 100%, 40%);">- pidfile=$("$asterisk_bin" -rx "core show settings" 2>/dev/null | sed -n -r -e "s/^\s*pid file:\s+(.*)/\1/gpi")</span><br><span style="color: hsl(0, 100%, 40%);">- # We found it</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$pidfile" != x -a -f "$pidfile" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- pid=$(cat "$pidfile")</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Found pidfile $pidfile with process $pid"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- # It's possible that asterisk was started with the -C option which means the</span><br><span style="color: hsl(0, 100%, 40%);">- # control socket and pidfile might not be where we expect. We're going to</span><br><span style="color: hsl(0, 100%, 40%);">- # have to parse the process arguments to see if -C was specified.</span><br><span style="color: hsl(0, 100%, 40%);">- # The first process that has a -C argument determines which config</span><br><span style="color: hsl(0, 100%, 40%);">- # file to use to find the pidfile of the main process.</span><br><span style="color: hsl(0, 100%, 40%);">- # NOTE: The ps command doesn't quote command line arguments that it</span><br><span style="color: hsl(0, 100%, 40%);">- # displays so we need to look in /proc/<pid>/cmdline.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$pid" = x ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- # BSDs might not mount /proc by default :(</span><br><span style="color: hsl(0, 100%, 40%);">- mounted_proc=0</span><br><span style="color: hsl(0, 100%, 40%);">- if uname -o | grep -qi "bsd" ; then</span><br><span style="color: hsl(0, 100%, 40%);">- if ! mount | grep -qi "/proc" ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Temporarily mounting /proc"</span><br><span style="color: hsl(0, 100%, 40%);">- mounted_proc=1</span><br><span style="color: hsl(0, 100%, 40%);">- mount -t procfs proc /proc</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- for p in $pids ; do</span><br><span style="color: hsl(0, 100%, 40%);">- # Fields in cmdline are delimited by NULLs</span><br><span style="color: hsl(0, 100%, 40%);">- astetcconf=$(sed -n -r -e "s/.*\x00-C\x00([^\x00]+).*/\1/gp" /proc/$p/cmdline)</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$astetcconf" != x ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- pidfile=$("$asterisk_bin" -C "$astetcconf" -rx "core show settings" 2>/dev/null | sed -n -r -e "s/^\s*pid file:\s+(.*)/\1/gpi")</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$pidfile" != x -a -f "$pidfile" ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- pid=$(cat "$pidfile")</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Found pidfile $pidfile the hard way with process $pid"</span><br><span style="color: hsl(0, 100%, 40%);">- break</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- done</span><br><span style="color: hsl(0, 100%, 40%);">- if [ $mounted_proc -eq 1 ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Unmounting /proc"</span><br><span style="color: hsl(0, 100%, 40%);">- umount /proc</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if [ x"$pid" = x ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- >&2 echo "Can't determine pid of the running asterisk instance"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if $RUNNING ; then</span><br><span style="color: hsl(0, 100%, 40%);">- answer=Y</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- read -p "WARNING: Taking a core dump of the running asterisk instance will suspend call processing while the dump is saved. Do you wish to continue? (y/N) " answer</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- if [[ "$answer" =~ ^[Yy] ]] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- cf="${OUTPUTDIR:-/tmp}/core-asterisk-running-$df"</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Dumping running asterisk process to $cf"</span><br><span style="color: hsl(0, 100%, 40%);">- ${GDB} ${asterisk_bin} -p $pid -q --batch --ex "gcore $cf" >/dev/null 2>&1</span><br><span style="color: hsl(0, 100%, 40%);">- COREDUMPS+=("$cf")</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Skipping dump of running process"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "${#COREDUMPS[@]}" -eq 0 ] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "No coredumps found"</span><br><span style="color: hsl(0, 100%, 40%);">- print_help</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Extract the gdb scripts from the end of this script</span><br><span style="color: hsl(0, 100%, 40%);">-# and save them to /tmp/.gdbinit</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-gdbinit=${OUTPUTDIR:-/tmp}/.ast_coredumper.gdbinit</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-trap "rm $gdbinit" EXIT</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-ss=`egrep -n "^#@@@SCRIPTSTART@@@" $0 |cut -f1 -d:`</span><br><span style="color: hsl(0, 100%, 40%);">-tail -n +${ss} $0 >$gdbinit</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Now iterate over the coredumps and dump the debugging info</span><br><span style="color: hsl(0, 100%, 40%);">-for i in ${!COREDUMPS[@]} ; do</span><br><span style="color: hsl(0, 100%, 40%);">- cf=$(readlink -ne ${COREDUMPS[$i]})</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Processing $cf"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- cfdir=`dirname ${cf}`</span><br><span style="color: hsl(0, 100%, 40%);">- cfname=`basename ${cf}`</span><br><span style="color: hsl(0, 100%, 40%);">- outputdir=${OUTPUTDIR:-${cfdir}}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- ${GDB} -n --batch -q --ex "source $gdbinit" "$asterisk_bin" "$cf" 2>/dev/null | (</span><br><span style="color: hsl(0, 100%, 40%);">- of=/dev/null</span><br><span style="color: hsl(0, 100%, 40%);">- while IFS= read line ; do</span><br><span style="color: hsl(0, 100%, 40%);">- if [[ "$line" =~ !@!@!@!\ ([^\ ]+)\ !@!@!@! ]] ; then</span><br><span style="color: hsl(0, 100%, 40%);">- of=${outputdir}/${cfname}-${BASH_REMATCH[1]}</span><br><span style="color: hsl(0, 100%, 40%);">- of=${of//:/-}</span><br><span style="color: hsl(0, 100%, 40%);">- rm -f "$of"</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Creating $of"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- echo -e $"$line" >> "$of"</span><br><span style="color: hsl(0, 100%, 40%);">- done</span><br><span style="color: hsl(0, 100%, 40%);">- )</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if $tarball_coredumps ; then</span><br><span style="color: hsl(0, 100%, 40%);">- cfname=${cfname//:/-}</span><br><span style="color: hsl(0, 100%, 40%);">- tf=${outputdir}/${cfname}.tar.gz</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Creating ${tf}"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- dest=${outputdir}/${cfname}.output</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf ${dest} 2>/dev/null || :</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- libdir=usr/lib</span><br><span style="color: hsl(0, 100%, 40%);">- [ -d /usr/lib64 ] && libdir+=64</span><br><span style="color: hsl(0, 100%, 40%);">- mkdir -p ${dest}/tmp ${dest}/${libdir}/asterisk ${dest}/etc ${dest}/usr/sbin</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- ln -s ${cf} ${dest}/tmp/${cfname}</span><br><span style="color: hsl(0, 100%, 40%);">- cp ${outputdir}/${cfname}*.txt ${dest}/tmp/</span><br><span style="color: hsl(0, 100%, 40%);">- [ -f /etc/os-release ] && cp /etc/os-release ${dest}/etc/</span><br><span style="color: hsl(0, 100%, 40%);">- if $tarball_config ; then</span><br><span style="color: hsl(0, 100%, 40%);">- cp -a /etc/asterisk ${dest}/etc/</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- cp -a /${libdir}/libasterisk* ${dest}/${libdir}/</span><br><span style="color: hsl(0, 100%, 40%);">- cp -a /${libdir}/asterisk/* ${dest}/${libdir}/asterisk/</span><br><span style="color: hsl(0, 100%, 40%);">- cp -a /usr/sbin/asterisk ${dest}/usr/sbin</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf ${tf}</span><br><span style="color: hsl(0, 100%, 40%);">- tar -chzf ${tf} --transform="s/^[.]/${cfname}.output/" -C ${dest} .</span><br><span style="color: hsl(0, 100%, 40%);">- sleep 3</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf ${dest}</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Created $tf"</span><br><span style="color: hsl(0, 100%, 40%);">- elif $tarball_results ; then</span><br><span style="color: hsl(0, 100%, 40%);">- cfname=${cfname//:/-}</span><br><span style="color: hsl(0, 100%, 40%);">- tf=${outputdir}/${cfname}.tar.gz</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Creating ${tf}"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- dest=${outputdir}/${cfname}.output</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf ${dest} 2>/dev/null || :</span><br><span style="color: hsl(0, 100%, 40%);">- mkdir -p ${dest}</span><br><span style="color: hsl(0, 100%, 40%);">- cp ${outputdir}/${cfname}*.txt ${dest}/</span><br><span style="color: hsl(0, 100%, 40%);">- if $tarball_config ; then</span><br><span style="color: hsl(0, 100%, 40%);">- mkdir -p ${dest}/etc</span><br><span style="color: hsl(0, 100%, 40%);">- cp -a /etc/asterisk ${dest}/etc/</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- tar -chzf ${tf} --transform="s/^[.]/${cfname}/" -C ${dest} .</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf ${dest}</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Created $tf"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if $delete_coredumps_after ; then</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf "${cf}"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if $delete_results_after ; then</span><br><span style="color: hsl(0, 100%, 40%);">- rm -rf "${cf//:/-}"-{brief,full,thread1,locks,info}.txt</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-exit</span><br><span style="color: hsl(120, 100%, 40%);">+#@@@HELPEND@@@</span><br><span> </span><br><span> # Be careful editng the inline scripts.</span><br><span> # They're space-indented.</span><br><span>diff --git a/doc/CHANGES-staging/ast_coredumper.txt b/doc/CHANGES-staging/ast_coredumper.txt</span><br><span>new file mode 100644</span><br><span>index 0000000..bbff0da</span><br><span>--- /dev/null</span><br><span>+++ b/doc/CHANGES-staging/ast_coredumper.txt</span><br><span>@@ -0,0 +1,23 @@</span><br><span style="color: hsl(120, 100%, 40%);">+Subject: ast_coredumper</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+New options:</span><br><span style="color: hsl(120, 100%, 40%);">+ --pid=<asterisk_pid></span><br><span style="color: hsl(120, 100%, 40%);">+ Allows specification of an Asterisk instance when trying to</span><br><span style="color: hsl(120, 100%, 40%);">+ and the script can't determine it itself.</span><br><span style="color: hsl(120, 100%, 40%);">+ --libdir=<system library directory></span><br><span style="color: hsl(120, 100%, 40%);">+ Allows specification of a non-standard installation directory</span><br><span style="color: hsl(120, 100%, 40%);">+ containing the Asterisk modules.</span><br><span style="color: hsl(120, 100%, 40%);">+ --(no-)rename</span><br><span style="color: hsl(120, 100%, 40%);">+ Renames the coredump and the output files with readable</span><br><span style="color: hsl(120, 100%, 40%);">+ timestamps. This is the default.</span><br><span style="color: hsl(120, 100%, 40%);">+Removed unneeded or confusing options:</span><br><span style="color: hsl(120, 100%, 40%);">+ --append-coredumps</span><br><span style="color: hsl(120, 100%, 40%);">+ --conffile</span><br><span style="color: hsl(120, 100%, 40%);">+ --no-default-search</span><br><span style="color: hsl(120, 100%, 40%);">+ --tarball-uniqueid</span><br><span style="color: hsl(120, 100%, 40%);">+Changed Variables:</span><br><span style="color: hsl(120, 100%, 40%);">+ COREDUMPS is now just "/tmp/core!(*.txt)"</span><br><span style="color: hsl(120, 100%, 40%);">+ DATEFORMAT is renamed to DATEOPTS and defaults to '-u +%FT%H-%M-%SZ'</span><br><span style="color: hsl(120, 100%, 40%);">+Changed behavior:</span><br><span style="color: hsl(120, 100%, 40%);">+ If you use 'running' or 'RUNNING' you no longer need to specify</span><br><span style="color: hsl(120, 100%, 40%);">+ '--no-default-search' to ignore existing coredumps.</span><br><span></span><br></pre><div style="white-space:pre-wrap"></div><p>To view, visit <a href="https://gerrit.asterisk.org/c/asterisk/+/16656">change 16656</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/c/asterisk/+/16656"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: 19 </div>
<div style="display:none"> Gerrit-Change-Id: I674be64bdde3ef310b6a551d4911c3b600ffee59 </div>
<div style="display:none"> Gerrit-Change-Number: 16656 </div>
<div style="display:none"> Gerrit-PatchSet: 3 </div>
<div style="display:none"> Gerrit-Owner: George Joseph <gjoseph@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Friendly Automation </div>
<div style="display:none"> Gerrit-Reviewer: George Joseph <gjoseph@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Kevin Harwell <kharwell@digium.com> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>