<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/2029/1/7/_/styles/combined.css?spaceKey=AST&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace">Getting a Backtrace</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~pabelanger">Paul Belanger</a>
</h4>
<div id="versionComment">
<b>Comment:</b>
Update to latest version from 1.8 branch, new additions.<br />
</div>
<br/>
<h4>Changes (94)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">This document is intended to provide information on how to obtain the backtraces required on the asterisk bug tracker, available at https://issues.asterisk.org. The information is required by developers to help fix problem with bugs of any kind. Backtraces provide information about what was wrong when a program crashed; in our case, Asterisk. There are two kind of backtraces (aka 'bt') which are useful: bt and bt full. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">This document is intended to provide information on how to obtain the backtraces required on the asterisk bug tracker, available at https://issues.asterisk.org. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">First of all, when you start Asterisk, you MUST start it with option -g. This tells Asterisk to produce a core file if it crashes. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Overview <br>The backtrace information is required by developers to help fix problem with bugs of any kind. Backtraces provide information about what was wrong when a program crashed; in our case, Asterisk. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">If you start Asterisk with the safe_asterisk script, it automatically starts using the option -g. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Preparing Asterisk To Produce Core Files On Crash <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">First of all, when you start Asterisk, you MUST start it with option -g. This tells Asterisk to produce a core file if it crashes. <br> <br>If you start Asterisk with the safe_asterisk script, it automatically starts using the option -g. <br> <br></td></tr>
<tr><td class="diff-unchanged" >If you're not sure if Asterisk is running with the -g option, type the following command in your shell: <br> <br>{noformat} <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp#</span> <span class="diff-added-words"style="background-color: #dfd;">#</span> ps aux | grep asterisk <br></td></tr>
<tr><td class="diff-changed-lines" >root <span class="diff-added-words"style="background-color: #dfd;"> </span> 17832 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0.0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 1.2 <span class="diff-added-words"style="background-color: #dfd;"> </span> 2348 <span class="diff-added-words"style="background-color: #dfd;"> </span> 788 pts/1 <span class="diff-added-words"style="background-color: #dfd;"> </span> S <span class="diff-added-words"style="background-color: #dfd;"> </span> Aug12 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0:00 /bin/sh /usr/sbin/safe_asterisk <br></td></tr>
<tr><td class="diff-changed-lines" >root <span class="diff-added-words"style="background-color: #dfd;"> </span> 26686 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0.0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 2.8 <span class="diff-added-words"style="background-color: #dfd;"> </span> 15544 <span class="diff-added-words"style="background-color: #dfd;"> </span> 1744 pts/1 <span class="diff-added-words"style="background-color: #dfd;"> </span> S <span class="diff-added-words"style="background-color: #dfd;"> </span> Aug13 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0:02 asterisk -vvvg -c <br></td></tr>
<tr><td class="diff-unchanged" >[...] <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-unchanged" >The interesting information is located in the last column. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-unchanged" >Second, your copy of Asterisk must have been built without optimization or the backtrace will be (nearly) unusable. This can be done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags submenu in the 'make menuselect' tree before building Asterisk. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Running a production server with DONT_OPTIMIZE is generally safe. You'll notice the binary files may be a bit larger, but in terms of Asterisk performance, and impact should be negligible. <br> <br></td></tr>
<tr><td class="diff-unchanged" >After Asterisk crashes, a core file will be "dumped" in your /tmp/ directory. To make sure it's really there, you can just type the following command in your shell: <br> <br>{noformat} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp# ls l /tmp/core.* <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"># ls -l /tmp/core.* <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root 10592256 Aug 12 19:40 /tmp/core.26252 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root <span class="diff-added-words"style="background-color: #dfd;"> </span> 9924608 Aug 12 20:12 /tmp/core.26340 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root 10862592 Aug 12 20:14 /tmp/core.26374 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root <span class="diff-added-words"style="background-color: #dfd;"> </span> 9105408 Aug 12 20:19 /tmp/core.26426 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root <span class="diff-added-words"style="background-color: #dfd;"> </span> 9441280 Aug 12 20:20 /tmp/core.26462 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">-rw------<span class="diff-added-chars"style="background-color: #dfd;">- </span></span> 1 root root <span class="diff-added-words"style="background-color: #dfd;"> </span> 8331264 Aug 13 00:32 /tmp/core.26647 <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp# <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br> <br></td></tr>
<tr><td class="diff-unchanged" >In the event that there are multiple core files present (as in the above example), it is important to look at the file timestamps in order to determine which one you really intend to look at. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Now that we've verified the core file has been written to disk, the final part is to extract 'bt' from the core file. Core files are pretty big, don't be scared, it's normal. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Getting Information After A Crash <br>There are two kind of backtraces (aka 'bt') which are useful: bt and bt full. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Now that we've verified the core file has been written to disk, the final part is to extract 'bt' from the core file. Core files are pretty big, don't be scared, it's normal. <br> <br></td></tr>
<tr><td class="diff-unchanged" >{note} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Don't attach core files on the bug tracker, we only need the bt and bt full. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Don't attach core files on the bug tracker as they are only useful on the machine they were generated on. We only need the output of the 'bt' and 'bt full.' <br></td></tr>
<tr><td class="diff-unchanged" >{note} <br> <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > <br>{noformat} <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp#</span> <span class="diff-added-words"style="background-color: #dfd;">#</span> gdb -v <br></td></tr>
<tr><td class="diff-unchanged" >GNU gdb 6.3-debian <br>Copyright 2004 Free Software Foundation, Inc. <br>GDB is free software, covered by the GNU General Public License, and you are <br>welcome to change it and/or distribute copies of it under certain conditions. <br></td></tr>
<tr><td class="diff-unchanged" >Type "show copying" to see the conditions. <br></td></tr>
<tr><td class="diff-changed-lines" >There is absolutely no warranty for GDB. <span class="diff-added-words"style="background-color: #dfd;"> </span> Type "show warranty" for details. <br></td></tr>
<tr><td class="diff-unchanged" >This GDB was configured as "i386-linux". <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp# <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Which is great, we can continue. If you don't have gdb installed, go install gdb. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">If you don't have gdb installed, go install gdb. You should be able to install using something like: apt-get install gdb --or-- yum install gdb <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Now load the core file in gdb, as follows: <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">{tip} <br>Just run the following command to get the output into the backtrace.txt file, ready for uploading to the issue tracker. Be sure to change the name of the core file to your actual core dump file: <br>{tip} <br>{noformat} <br># gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c /tmp/core.26252 > /tmp/backtrace.txt <br>{noformat} <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Now load the core file in gdb with the following command. This will also save the output of gdb to the /tmp/backtract.txt file. <br> <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">debian:/tmp# gdb asterisk /tmp/core.26252 <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"># gdb -se "asterisk" -c /tmp/core.26252 | tee /tmp/backtrace.txt <br></td></tr>
<tr><td class="diff-unchanged" >[...] <br></td></tr>
<tr><td class="diff-unchanged" >(You would see a lot of output here.)
[...] <br></td></tr>
<tr><td class="diff-unchanged" >Reading symbols from /usr/lib/asterisk/modules/app_externalivr.so...done. <br></td></tr>
<tr><td class="diff-unchanged" >Loaded symbols for /usr/lib/asterisk/modules/app_externalivr.so <br></td></tr>
<tr><td class="diff-changed-lines" >#0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x29b45d7e in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >(gdb) <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">In order to make extracting the gdb output easier, you may wish to turn on logging using "set logging on". This command will save all output to the default file of gdb.txt, which in the end can be uploaded as an attachment to the bug tracker. <br> <br></td></tr>
<tr><td class="diff-unchanged" >Now at the gdb prompt, type: bt You would see output similar to: <br> <br>{noformat} <br></td></tr>
<tr><td class="diff-unchanged" >(gdb) bt <br></td></tr>
<tr><td class="diff-changed-lines" >#0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x29b45d7e in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#1 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#2 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa58 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#3 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#4 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa60 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#5 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#6 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180bf894 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#7 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x0bf80008 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#8 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180b0818 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#9 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180 <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#10 0x000000a0 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">#1<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">0</span><span class="diff-added-chars"style="background-color: #dfd;">1</span></span> 0x000000a0 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">#11 0x000000a0</span> <span class="diff-added-words"style="background-color: #dfd;">#12 0x00000000</span> in ?? () <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262 <br>#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965 <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">#12 0x00000000</span> <span class="diff-added-words"style="background-color: #dfd;">#15 0xbcdffbe0</span> in ?? () <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262 <br>#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965 <br>#15 0xbcdffbe0 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0 <br></td></tr>
<tr><td class="diff-unchanged" >#17 0x401ec92a in clone () from /lib/libc.so.6 <br>(gdb) <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > <br>{noformat} <br></td></tr>
<tr><td class="diff-unchanged" >(gdb) bt full <br></td></tr>
<tr><td class="diff-changed-lines" >#0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x29b45d7e in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#1 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#2 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa58 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#3 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#4 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa60 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#5 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#6 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180bf894 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#7 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x0bf80008 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#8 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180b0818 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" >#9 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180 <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#10 0x000000a0 in ?? () <br>No symbol table info available. <br></td></tr>
<tr><td class="diff-unchanged" >No locals. <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">#1<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">0</span><span class="diff-added-chars"style="background-color: #dfd;">1</span></span> 0x000000a0 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">#11 0x000000a0</span> <span class="diff-added-words"style="background-color: #dfd;">#12 0x00000000</span> in ?? () <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262 <br> f = (struct ast_frame *) 0x8180bf8 <br> trans = (struct ast_trans_pvt *) 0x0 <br>#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965 <br>No locals. <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available. <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">#12 0x00000000</span> <span class="diff-added-words"style="background-color: #dfd;">#15 0xbcdffbe0</span> in ?? () <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262 <br> f = (struct ast_frame *) 0x8180bf8 <br> trans = (struct ast_trans_pvt *) 0x0 <br>#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965 <br>No locals. <br>#15 0xbcdffbe0 in ?? () <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0 <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0 <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">#17 0x401ec92a in clone () from /lib/libc.so.6 <br></td></tr>
<tr><td class="diff-unchanged" >No symbol table info available.
No symbol table info available.
No symbol table info available. <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">#17 0x401ec92a in clone () from /lib/libc.so.6 <br>No symbol table info available. <br></td></tr>
<tr><td class="diff-unchanged" >(gdb) <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">We also need gdb's output. That output gives more details compared to the simple "bt". So we recommend that you use bt full instead of bt. But, if you could include both, we appreciate that. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">The final "extraction" would be to know all traces by all threads. Even if Asterisk runs on the same thread for each call, it could have created some new threads. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">The final "extraction" would be to know all traces by all threads. Even if asterisk runs on the same thread for each call, it could have created some new threads. <br> <br></td></tr>
<tr><td class="diff-changed-lines" >To make sure we have the correct information, just do: <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">(gdb) thread apply all bt</span> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">(gdb) thread apply all bt <br> <br></td></tr>
<tr><td class="diff-unchanged" >Thread 1 (process 26252): <br></td></tr>
<tr><td class="diff-changed-lines" >#0 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x29b45d7e in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#1 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#2 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa58 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#3 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#4 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0xbcdffa60 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#5 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08180bf8 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#6 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180bf894 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#7 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x0bf80008 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#8 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x180b0818 in ?? () <br></td></tr>
<tr><td class="diff-changed-lines" >#9 <span class="diff-added-words"style="background-color: #dfd;"> </span> 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180 <br></td></tr>
<tr><td class="diff-unchanged" >#10 0x000000a0 in ?? () <br></td></tr>
<tr><td class="diff-unchanged" >#11 0x000000a0 in ?? ()
#12 0x00000000 in ?? ()
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
#15 0xbcdffbe0 in ?? ()
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
#17 0x401ec92a in clone () from /lib/libc.so.6 <br></td></tr>
<tr><td class="diff-unchanged" >(gdb) <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-unchanged" >That output tells us crucial information about each thread. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Now, just create an output.txt file and dump your "bt full" (and/or "bt") ALONG WITH "thread apply all bt" into it. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Getting Information For A Deadlock <br>Whenever supplying information about a deadlock (i.e. when you run the 'core show locks' command on the Asterisk console), it is useful to also have additional information about the threads. We can generate this information by attaching to a running Asterisk process and gathering that information. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">You can easily attach to a running Asterisk process, gather the output required and then detach from the process all in a single step. Execute the following command and upload the resulting backtrace-threads.txt file to the Asterisk issue tracker: <br> <br>{noformat} <br># gdb -ex "thread apply all bt" --batch /usr/sbin/asterisk `pidof asterisk` > /tmp/backtrace-threads.txt <br>{noformat} <br> <br>Note that this gathers information from the running Asterisk process, so you want to make sure you run this command immediately before or after gathering the output of 'core show locks'. You can gather that information by running the following command: <br> <br>{noformat} <br># asterisk -rx "core show locks" > /tmp/core-show-locks.txt <br>{noformat} <br> <br>h2. Verify Your Backtraces <br> <br>Before uploading your backtraces to the issue tracker, you should double check to make sure the data you have is of use to the developers. Check your backtrace files to make sure you're not seeing several of the following: <br> <br>{code} <br><value optimized out> <br>{code} <br> <br>If you are, then you likely haven't compiled with DONT_OPTIMIZE. The impact of DONT_OPTIMIZE is negligible on most systems. Be sure you've enabled the DONT_OPTIMIZE flag within the Compiler Flags section of menuselect. After doing so, be sure to run 'make install' and restart Asterisk. <br> <br>{warning} <br>Double check your backtrace! It is important you do not see <value optimized out>, otherwise the backtrace does not provide enough information for a developer. <br>{warning} <br> <br>h2. Uploading Your Information To The Issue Tracker <br>You're now ready to upload your files to the Asterisk issue tracker (located at <br>https://issues.asterisk.org). <br> <br></td></tr>
<tr><td class="diff-unchanged" >{note} <br></td></tr>
<tr><td class="diff-changed-lines" >Please ATTACH your <span class="diff-changed-words">output<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">,</span><span class="diff-added-chars"style="background-color: #dfd;">!</span></span> DO NOT paste it as a <span class="diff-changed-words">note<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">.</span><span class="diff-added-chars"style="background-color: #dfd;">!</span></span> <br></td></tr>
<tr><td class="diff-unchanged" >{note} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">And you're ready for upload on the bug tracker. <br> <br></td></tr>
<tr><td class="diff-unchanged" >If you have questions or comments regarding this documentation, feel free to pass by the #asterisk-bugs channel on irc.freenode.net. <br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<p>This document is intended to provide information on how to obtain the backtraces required on the asterisk bug tracker, available at <a href="https://issues.asterisk.org" class="external-link" rel="nofollow">https://issues.asterisk.org</a>. </p>
<h2><a name="GettingaBacktrace-Overview"></a>Overview</h2>
<p>The backtrace information is required by developers to help fix problem with bugs of any kind. Backtraces provide information about what was wrong when a program crashed; in our case, Asterisk.</p>
<h2><a name="GettingaBacktrace-PreparingAsteriskToProduceCoreFilesOnCrash"></a>Preparing Asterisk To Produce Core Files On Crash</h2>
<p>First of all, when you start Asterisk, you MUST start it with option -g. This tells Asterisk to produce a core file if it crashes.</p>
<p>If you start Asterisk with the safe_asterisk script, it automatically starts using the option -g.</p>
<p>If you're not sure if Asterisk is running with the -g option, type the following command in your shell:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># ps aux | grep asterisk
root 17832 0.0 1.2 2348 788 pts/1 S Aug12 0:00 /bin/sh /usr/sbin/safe_asterisk
root 26686 0.0 2.8 15544 1744 pts/1 S Aug13 0:02 asterisk -vvvg -c
[...]
</pre>
</div></div>
<p>The interesting information is located in the last column.</p>
<p>Second, your copy of Asterisk must have been built without optimization or the backtrace will be (nearly) unusable. This can be done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags submenu in the 'make menuselect' tree before building Asterisk.</p>
<p>Running a production server with DONT_OPTIMIZE is generally safe. You'll notice the binary files may be a bit larger, but in terms of Asterisk performance, and impact should be negligible.</p>
<p>After Asterisk crashes, a core file will be "dumped" in your /tmp/ directory. To make sure it's really there, you can just type the following command in your shell:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># ls -l /tmp/core.*
-rw------- 1 root root 10592256 Aug 12 19:40 /tmp/core.26252
-rw------- 1 root root 9924608 Aug 12 20:12 /tmp/core.26340
-rw------- 1 root root 10862592 Aug 12 20:14 /tmp/core.26374
-rw------- 1 root root 9105408 Aug 12 20:19 /tmp/core.26426
-rw------- 1 root root 9441280 Aug 12 20:20 /tmp/core.26462
-rw------- 1 root root 8331264 Aug 13 00:32 /tmp/core.26647
</pre>
</div></div>
<p>In the event that there are multiple core files present (as in the above example), it is important to look at the file timestamps in order to determine which one you really intend to look at.</p>
<h2><a name="GettingaBacktrace-GettingInformationAfterACrash"></a>Getting Information After A Crash</h2>
<p>There are two kind of backtraces (aka 'bt') which are useful: bt and bt full.</p>
<p>Now that we've verified the core file has been written to disk, the final part is to extract 'bt' from the core file. Core files are pretty big, don't be scared, it's normal.</p>
<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Don't attach core files on the bug tracker as they are only useful on the machine they were generated on. We only need the output of the 'bt' and 'bt full.'</td></tr></table></div>
<p>For extraction, we use a really nice tool, called gdb. To verify that you have gdb installed on your system:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># gdb -v
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-linux".
</pre>
</div></div>
<p>If you don't have gdb installed, go install gdb. You should be able to install using something like: apt-get install gdb -<del>or</del>- yum install gdb</p>
<div class='panelMacro'><table class='tipMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/check.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Just run the following command to get the output into the backtrace.txt file, ready for uploading to the issue tracker. Be sure to change the name of the core file to your actual core dump file:</td></tr></table></div>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c /tmp/core.26252 > /tmp/backtrace.txt
</pre>
</div></div>
<p>Now load the core file in gdb with the following command. This will also save the output of gdb to the /tmp/backtract.txt file.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># gdb -se "asterisk" -c /tmp/core.26252 | tee /tmp/backtrace.txt
[...]
(You would see a lot of output here.)
[...]
Reading symbols from /usr/lib/asterisk/modules/app_externalivr.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_externalivr.so
#0 0x29b45d7e in ?? ()
(gdb)
</pre>
</div></div>
<p>In order to make extracting the gdb output easier, you may wish to turn on logging using "set logging on". This command will save all output to the default file of gdb.txt, which in the end can be uploaded as an attachment to the bug tracker.</p>
<p>Now at the gdb prompt, type: bt You would see output similar to:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(gdb) bt
#0 0x29b45d7e in ?? ()
#1 0x08180bf8 in ?? ()
#2 0xbcdffa58 in ?? ()
#3 0x08180bf8 in ?? ()
#4 0xbcdffa60 in ?? ()
#5 0x08180bf8 in ?? ()
#6 0x180bf894 in ?? ()
#7 0x0bf80008 in ?? ()
#8 0x180b0818 in ?? ()
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
#10 0x000000a0 in ?? ()
#11 0x000000a0 in ?? ()
#12 0x00000000 in ?? ()
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
#15 0xbcdffbe0 in ?? ()
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
#17 0x401ec92a in clone () from /lib/libc.so.6
(gdb)
</pre>
</div></div>
<p>The bt's output is the information that we need on the bug tracker.</p>
<p>Now do a bt full as follows:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(gdb) bt full
#0 0x29b45d7e in ?? ()
No symbol table info available.
#1 0x08180bf8 in ?? ()
No symbol table info available.
#2 0xbcdffa58 in ?? ()
No symbol table info available.
#3 0x08180bf8 in ?? ()
No symbol table info available.
#4 0xbcdffa60 in ?? ()
No symbol table info available.
#5 0x08180bf8 in ?? ()
No symbol table info available.
#6 0x180bf894 in ?? ()
No symbol table info available.
#7 0x0bf80008 in ?? ()
No symbol table info available.
#8 0x180b0818 in ?? ()
No symbol table info available.
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
No locals.
#10 0x000000a0 in ?? ()
No symbol table info available.
#11 0x000000a0 in ?? ()
No symbol table info available.
#12 0x00000000 in ?? ()
No symbol table info available.
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
f = (struct ast_frame *) 0x8180bf8
trans = (struct ast_trans_pvt *) 0x0
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
No locals.
#15 0xbcdffbe0 in ?? ()
No symbol table info available.
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
No symbol table info available.
#17 0x401ec92a in clone () from /lib/libc.so.6
No symbol table info available.
(gdb)
</pre>
</div></div>
<p>The final "extraction" would be to know all traces by all threads. Even if Asterisk runs on the same thread for each call, it could have created some new threads.</p>
<p>To make sure we have the correct information, just do:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(gdb) thread apply all bt
Thread 1 (process 26252):
#0 0x29b45d7e in ?? ()
#1 0x08180bf8 in ?? ()
#2 0xbcdffa58 in ?? ()
#3 0x08180bf8 in ?? ()
#4 0xbcdffa60 in ?? ()
#5 0x08180bf8 in ?? ()
#6 0x180bf894 in ?? ()
#7 0x0bf80008 in ?? ()
#8 0x180b0818 in ?? ()
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
#10 0x000000a0 in ?? ()
#11 0x000000a0 in ?? ()
#12 0x00000000 in ?? ()
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
#15 0xbcdffbe0 in ?? ()
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
#17 0x401ec92a in clone () from /lib/libc.so.6
(gdb)
</pre>
</div></div>
<p>That output tells us crucial information about each thread.</p>
<h2><a name="GettingaBacktrace-GettingInformationForADeadlock"></a>Getting Information For A Deadlock</h2>
<p>Whenever supplying information about a deadlock (i.e. when you run the 'core show locks' command on the Asterisk console), it is useful to also have additional information about the threads. We can generate this information by attaching to a running Asterisk process and gathering that information.</p>
<p>You can easily attach to a running Asterisk process, gather the output required and then detach from the process all in a single step. Execute the following command and upload the resulting backtrace-threads.txt file to the Asterisk issue tracker:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># gdb -ex "thread apply all bt" --batch /usr/sbin/asterisk `pidof asterisk` > /tmp/backtrace-threads.txt
</pre>
</div></div>
<p>Note that this gathers information from the running Asterisk process, so you want to make sure you run this command immediately before or after gathering the output of 'core show locks'. You can gather that information by running the following command:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre># asterisk -rx "core show locks" > /tmp/core-show-locks.txt
</pre>
</div></div>
<h2><a name="GettingaBacktrace-VerifyYourBacktraces"></a>Verify Your Backtraces</h2>
<p>Before uploading your backtraces to the issue tracker, you should double check to make sure the data you have is of use to the developers. Check your backtrace files to make sure you're not seeing several of the following:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<script type="syntaxhighlighter" class="toolbar: false; theme: Confluence; brush: java; gutter: false"><![CDATA[
<value optimized out>
]]></script>
</div></div>
<p>If you are, then you likely haven't compiled with DONT_OPTIMIZE. The impact of DONT_OPTIMIZE is negligible on most systems. Be sure you've enabled the DONT_OPTIMIZE flag within the Compiler Flags section of menuselect. After doing so, be sure to run 'make install' and restart Asterisk.</p>
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Double check your backtrace! It is important you do not see <value optimized out>, otherwise the backtrace does not provide enough information for a developer.</td></tr></table></div>
<h2><a name="GettingaBacktrace-UploadingYourInformationToTheIssueTracker"></a>Uploading Your Information To The Issue Tracker</h2>
<p>You're now ready to upload your files to the Asterisk issue tracker (located at
<a href="https://issues.asterisk.org" class="external-link" rel="nofollow">https://issues.asterisk.org</a>).</p>
<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Please ATTACH your output! DO NOT paste it as a note!</td></tr></table></div>
<p>If you have questions or comments regarding this documentation, feel free to pass by the #asterisk-bugs channel on irc.freenode.net.</p>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;">
<a href="https://wiki.asterisk.org/wiki/users/viewnotifications.action" class="grey">Change Notification Preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=5243139&revisedVersion=3&originalVersion=2">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>