<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2017-12-06 15:52 GMT+01:00 George Joseph <span dir="ltr"><<a href="mailto:gjoseph@digium.com" target="_blank">gjoseph@digium.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Tue, Dec 5, 2017 at 9:20 AM, Olivier <span dir="ltr"><<a href="mailto:oza.4h07@gmail.com" target="_blank">oza.4h07@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Hello,<br><br></div>I carefully read [1] which details how backtrace files can be produced.<br></div><br>Maybe this seems natural to some, but how can I go one step futher, and check that produced XXX-<code class="gmail-m_-7046299071133924745gmail-m_-5682528971970587762gmail-java gmail-m_-7046299071133924745gmail-m_-5682528971970587762gmail-plain">thread1.txt, XXX-brief.txt, ... files are OK ?<br></code><div><div><br></div><div>In other words, where can I find an example on how to use one of those files and check by myself, that if a system ever fails, I won't have to wait for another failure to provide required data to support teams ?<br></div></div></div></blockquote><div><br></div></span><div>It's a great question but I could spend a week answering it and not scratch the surface. :) </div></div></div></div></blockquote><div><br></div><div>Thanks very much for trying, anyway ;-)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> It's not a straightforward thing unless you know the code in question. The most common is a segmentation fault (segfault or SEGV).</div></div></div></div></blockquote><div><br></div><div>True ! I experienced segfaults lately and I could not configure the platform I used then (Debian Jessie) to produce core files in a directory Asterisk can write into.<br></div><div>Now, with Debian Stretch, I can produce core file at will (with a kill -s SIGSEGV <processid>).<br></div><div>I checked ast_coredumped worked OK as it produced thread.txt files and so on.<br></div><div><br>Ideally, I would like to go one step further: check now that a future .txt file would be "workable" (and not "you should have compiled with option XXX or configured with option YYY) .<br></div><div><br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> In that case, the thread1.txt file is the place to start. Since most of the objects passed around are really pointers to objects, the most obvious cause would be a 0x0 for a value. So for instance "chan=0x0". That would be a pointer to a channel object that was not set when it probably should have been. Unfortunately, it's not only 0x0 that could cause a segv. Anytime a program tries to access memory it doesn't own, that signal is raised. So let's say there a 256 byte buffer which the process owns. If there's a bug somewhere that causes the program to try and access bytes beyond the end of the buffer, you MAY get a segv if that process doesn't also own that memory. If this case, the backtrace won't show anything obvious because the pointers all look valid. There probably would be an index variable (i or ix, etc) that may be set to 257 but you'd have to know that the buffer was only 256 bytes to realize that that was the issue.</div></div></div></div></blockquote><div><br></div><div>So, with an artificial kill -s SIGSEGV <processid><span class="gmail-">, does the bellow output prove I have a workable .txt files (having .txt files that let people find the root cause of the issue is another story as we probably can only hope for the best here) ?<br><br></span><br> # head core-brief.txt<br>!@!@!@! brief.txt !@!@!@!<br><br><br>Thread 38 (Thread 0x7f2aa5dd0700 (LWP 992)):<br>#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225<br>#1 0x000055cdcb69ae84 in __ast_cond_timedwait (filename=0x55cdcb7d4910 "threadpool.c", lineno=1131, func=0x55cdcb7d4ea8 <__PRETTY_FUNCTION__.8978> "worker_idle", cond_name=0x55cdcb7d4b7f "&worker->cond", mutex_name=0x55cdcb7d4b71 "&worker->lock", cond=0x7f2abc000978, t=0x7f2abc0009a8, abstime=0x7f2aa5dcfc30) at lock.c:668<br>#2 0x000055cdcb75d153 in worker_idle (worker=0x7f2abc000970) at threadpool.c:1131<br>#3 0x000055cdcb75ce61 in worker_start (arg=0x7f2abc000970) at threadpool.c:1022<br>#4 0x000055cdcb769a8c in dummy_start (data=0x7f2abc000a80) at utils.c:1238<br>#5 0x00007f2aeddad494 in start_thread (arg=0x7f2aa5dd0700) at pthread_create.c:333<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Deadlocks are even harder to troubleshoot. For that, you need to look at full.txt to see where the threads are stuck and find the 1 thread that's holding the lock that the others are stuck on.</div><div><br></div><div>Sorry. I wish I had a better answer because it'd help a lot if folks could do more investigation themselves.</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><div dir="ltr"><div><div><br></div><div>Best regards<br></div><div><br>[1] <a href="https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace" target="_blank">https://wiki.asterisk.org/wiki<wbr>/display/AST/Getting+a+Backtra<wbr>ce</a><br></div></div></div>
<br></span><span class="gmail-HOEnZb"><font color="#888888">--<br>
______________________________<wbr>______________________________<wbr>_________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
Check out the new Asterisk community forum at: <a href="https://community.asterisk.org/" rel="noreferrer" target="_blank">https://community.asterisk.org<wbr>/</a><br>
<br>
New to Asterisk? Start here:<br>
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+Started" rel="noreferrer" target="_blank">https://wiki.asterisk.org/wiki<wbr>/display/AST/Getting+Started</a><br>
<br>
asterisk-users mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-users" rel="noreferrer" target="_blank">http://lists.digium.com/mailm<wbr>an/listinfo/asterisk-users</a><br></font></span></blockquote></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_-7046299071133924745gmail_signature"><div dir="ltr"><span style="font-size:12.8px">George Joseph</span><br style="font-size:12.8px"><span style="font-size:12.8px">Digium, Inc. | Software Developer</span><span style="font-size:12.8px"><br>445 Jan Davis Drive NW - Huntsville, AL 35806 - US<br></span><span style="font-size:12.8px">Check us out at: </span><a href="http://www.digium.com/" rel="noreferrer" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">www.digium.com</a><span style="font-size:12.8px"> & </span><a href="http://www.asterisk.org/" rel="noreferrer" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">www.<wbr>asterisk.org</a><br><div><br></div></div></div>
</font></span></div></div>
<br>--<br>
______________________________<wbr>______________________________<wbr>_________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
Check out the new Asterisk community forum at: <a href="https://community.asterisk.org/" rel="noreferrer" target="_blank">https://community.asterisk.<wbr>org/</a><br>
<br>
New to Asterisk? Start here:<br>
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+Started" rel="noreferrer" target="_blank">https://wiki.asterisk.org/<wbr>wiki/display/AST/Getting+<wbr>Started</a><br>
<br>
asterisk-users mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-users" rel="noreferrer" target="_blank">http://lists.digium.com/<wbr>mailman/listinfo/asterisk-<wbr>users</a><br></blockquote></div><br></div></div>