[svn-commits] lmadsen: branch 1.6.2 r257428 - /branches/1.6.2/doc/backtrace.txt
SVN commits to the Digium repositories
svn-commits at lists.digium.com
Thu Apr 15 14:42:57 CDT 2010
Author: lmadsen
Date: Thu Apr 15 14:42:55 2010
New Revision: 257428
URL: http://svnview.digium.com/svn/asterisk?view=rev&rev=257428
Log:
Merged revisions 257427 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk
................
r257427 | lmadsen | 2010-04-15 14:41:05 -0500 (Thu, 15 Apr 2010) | 21 lines
Merged revisions 257426 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r257426 | lmadsen | 2010-04-15 14:40:33 -0500 (Thu, 15 Apr 2010) | 13 lines
Update backtrace.txt documentation.
Update the backtrace.txt documentation so it conforms to the same layout as
other documents we've been working on recently. Additionally, add a bunch of
new information about gathering backtraces for crashes and deadlocks, along
with ways of verifying your file before uploading it. Create a couple of one
line commands for people to generate the files we need.
(closes issue #17190)
Reported by: lmadsen
Patches:
backtrace.txt.patch-2 uploaded by lmadsen (license 10)
Tested by: lmadsen, pabelanger
........
................
Modified:
branches/1.6.2/doc/backtrace.txt
Modified: branches/1.6.2/doc/backtrace.txt
URL: http://svnview.digium.com/svn/asterisk/branches/1.6.2/doc/backtrace.txt?view=diff&rev=257428&r1=257427&r2=257428
==============================================================================
--- branches/1.6.2/doc/backtrace.txt (original)
+++ branches/1.6.2/doc/backtrace.txt Thu Apr 15 14:42:55 2010
@@ -1,12 +1,25 @@
+===============================================================================
+===
+=== Collecting Backtrace Information
+===
+=== Last updated: 2010-04-12
+===============================================================================
+
This document is intended to provide information on how to obtain the
backtraces required on the asterisk bug tracker, available at
-http://issues.asterisk.org.
-
-The information is required by developers to
-help fix problem with bugs of any kind. Backtraces provide information
-about what was wrong when a program crashed; in our case,
-Asterisk. There are two kind of backtraces (aka 'bt') which are
-useful: bt and bt full.
+https://issues.asterisk.org.
+
+-------------------------------------------------------------------------------
+--- Overview
+-------------------------------------------------------------------------------
+
+The backtrace information is required by developers to help fix problem with
+bugs of any kind. Backtraces provide information about what was wrong when a
+program crashed; in our case, Asterisk.
+
+-------------------------------------------------------------------------------
+--- Preparing Asterisk To Produce Core Files On Crash
+-------------------------------------------------------------------------------
First of all, when you start Asterisk, you MUST start it with option
-g. This tells Asterisk to produce a core file if it crashes.
@@ -28,6 +41,10 @@
optimization or the backtrace will be (nearly) unusable. This can be
done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags
submenu in the 'make menuselect' tree before building Asterisk.
+
+Running a production server with DONT_OPTIMIZE is generally safe.
+You'll notice the binary files may be a bit larger, but in terms of
+Asterisk performance, and impact should be negligible.
After Asterisk crashes, a core file will be "dumped" in your /tmp/
directory. To make sure it's really there, you can just type the
@@ -46,11 +63,21 @@
above example), it is important to look at the file timestamps in
order to determine which one you really intend to look at.
-Now that we've verified the core file has been written to disk, the
-final part is to extract 'bt' from the core file. Core files are
-pretty big, don't be scared, it's normal.
-
-*** NOTE: Don't attach core files on the bug tracker, we only need the bt and bt full. ***
+-------------------------------------------------------------------------------
+--- Getting Information After A Crash
+-------------------------------------------------------------------------------
+
+There are two kind of backtraces (aka 'bt') which are useful: bt and bt full.
+
+Now that we've verified the core file has been written to disk, the final part
+is to extract 'bt' from the core file. Core files are pretty big, don't be
+scared, it's normal.
+
+******************************************************************************
+*** NOTE: Don't attach core files on the bug tracker as they are only useful *
+*** on the machine they were generated on. We only need the output of *
+*** the 'bt' and 'bt full.' *
+******************************************************************************
For extraction, we use a really nice tool, called gdb. To verify that
you have gdb installed on your system:
@@ -65,11 +92,25 @@
This GDB was configured as "i386-linux".
debian:/tmp#
-Which is great, we can continue. If you don't have gdb installed, go install gdb.
-
-Now load the core file in gdb, as follows:
-
-debian:/tmp# gdb asterisk /tmp/core.26252
+If you don't have gdb installed, go install gdb. You should be able to install
+using something like: apt-get install gdb --or-- yum install gdb
+
+Now load the core file in gdb with the following command. This will also save
+the output of gdb to the /tmp/backtract.txt file.
+
+# gdb -se "asterisk" -c /tmp/core.26252 | tee /tmp/backtrace.txt
+
+******************************************************************************
+*** TIP!
+*** Just run the following command to get the output into the
+*** backtrace.txt file, ready for uploading to the issue tracker. Be sure
+*** to change the name of the core file to your actual core dump file:
+***
+*** gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c /tmp/core.26252 > /tmp/backtrace.txt
+***
+******************************************************************************
+
+
[...]
(You would see a lot of output here.)
[...]
@@ -85,6 +126,7 @@
Now at the gdb prompt, type: bt
You would see output similar to:
+
(gdb) bt
#0 0x29b45d7e in ?? ()
#1 0x08180bf8 in ?? ()
@@ -110,6 +152,7 @@
The bt's output is the information that we need on the bug tracker.
Now do a bt full as follows:
+
(gdb) bt full
#0 0x29b45d7e in ?? ()
No symbol table info available.
@@ -150,13 +193,9 @@
No symbol table info available.
(gdb)
-We also need gdb's output. That output gives more details compared to
-the simple "bt". So we recommend that you use bt full instead of bt.
-But, if you could include both, we appreciate that.
-
-The final "extraction" would be to know all traces by all
-threads. Even if asterisk runs on the same thread for each call, it
-could have created some new threads.
+The final "extraction" would be to know all traces by all threads. Even if
+Asterisk runs on the same thread for each call, it could have created some new
+threads.
To make sure we have the correct information, just do:
(gdb) thread apply all bt
@@ -185,15 +224,54 @@
That output tells us crucial information about each thread.
-Now, if you turned on logging upload your gdb.txt file. Otherwise,
-create an output.txt file and dump your "bt full" (and/or "bt")
-ALONG WITH "thread apply all bt" into it.
-
-Note: Please ATTACH your output, DO NOT paste it as a note.
-
-And you're ready for upload on the bug tracker.
-
-
-If you have questions or comments regarding this documentation, feel
-free to pass by the #asterisk-bugs channel on irc.freenode.net.
-
+-------------------------------------------------------------------------------
+--- Getting Information For A Deadlock
+-------------------------------------------------------------------------------
+
+Whenever supplying information about a deadlock (i.e. when you run the
+'core show locks' command on the Asterisk console), it is useful to also have
+additional information about the threads. We can generate this information by
+attaching to a running Asterisk process and gathering that information.
+
+You can easily attach to a running Asterisk process, gather the output required
+and then detach from the process all in a single step. Execute the following
+command and upload the resulting backtrace-threads.txt file to the Asterisk
+issue tracker:
+
+ gdb -ex "thread apply all bt" --batch /usr/sbin/asterisk `pidof asterisk` > /tmp/backtrace-threads.txt
+
+Note that this gathers information from the running Asterisk process, so you
+want to make sure you run this command immediately before or after gathering
+the output of 'core show locks'. You can gather that information by running the
+following command:
+
+ asterisk -rx "core show locks" > /tmp/core-show-locks.txt
+
+-------------------------------------------------------------------------------
+--- Verify Your Backtraces
+-------------------------------------------------------------------------------
+
+Before uploading your backtraces to the issue tracker, you should double check
+to make sure the data you have is of use to the developers. Check your
+backtrace files to make sure you're not seeing several of the following:
+
+ <value optimized out>
+
+If you are, then you likely haven't compiled with DONT_OPTIMIZE. The impact of
+DONT_OPTIMIZE is negligible on most systems. Be sure you've enabled the
+DONT_OPTIMIZE flag within the Compiler Flags section of menuselect. After
+doing so, be sure to run 'make install' and restart Asterisk.
+
+-------------------------------------------------------------------------------
+--- Uploading Your Information To The Issue Tracker
+-------------------------------------------------------------------------------
+
+You're now ready to upload your files to the Asterisk issue tracker (located at
+https://issues.asterisk.org).
+
+******************************************************************************
+*** NOTE: Please ATTACH your output! DO NOT paste it as a note! *
+******************************************************************************
+
+If you have questions or comments regarding this documentation, feel free to
+pass by the #asterisk-bugs channel on irc.freenode.net.
More information about the svn-commits
mailing list