[asterisk-bugs] [Asterisk 0010775]: Asterisk suddenly slows down, and eats 100% cpu

[noreply at bugs.digium.com]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=10775 
====================================================================== 
Reported By:                atis
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   10775
Category:                   Core-General
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     new
Asterisk Version:            1.4.10  
SVN Branch (only for SVN checkouts, not tarball releases): N/A  
SVN Revision (number only!):  
Disclaimer on File?:        N/A 
Request Review:              
====================================================================== 
Date Submitted:             09-20-2007 16:36 CDT
Last Modified:              09-27-2007 10:31 CDT
====================================================================== 
Summary:                    Asterisk suddenly slows down, and eats 100% cpu
Description: 
I'm running 1.4.10 (with reverted res_features from
http://bugs.digium.com/view.php?id=10415). Approximately
once or twice per week (restarted daily) asterisk suddenly starts acting
slowly, and eats 100% cpu. Call volume drops as it takes really long time
for customers to get to agents, but this doesn't improve situation in any
way.

Last week i recompiled with DONT_OPTIMIZE, DEBUG_THREADS and DEBUG_LOCKS,
so now i can attach debug output.

I killed asterisk with "killall -5 asterisk", so that core is dumped.

Attached files:
backtrace.txt - backtraces taken from core

report.txt - various information ("top", "core show threads", "core show
locks", "core show channels") right before i killed asterisk.

normal.txt - top, and channel count when under normal load (30 minutes
after kill)

cli.txt - multiple consecutive executions of "core show threads" and "core
show locks", when problem raised.


====================================================================== 

---------------------------------------------------------------------- 
 atis - 09-27-07 10:31  
---------------------------------------------------------------------- 
No i'm using, i just was frightened by having asterisk console somewhere on
tty9. However, turing it off didn't helped. Actually that was my mistake,
assuming that asterisk uses separate processes, that can be monitored the
same way as httpd for example.

So, you should ignore what i said about main thread. Threads are not
visible when running regular "top". Now i'm using "top -H" instead.
Unfortuneatly crashes are getting more often now (maybe because of
increased load to manager), but this gives me more time for inspection (and
more sleepless nights). 

I monitored that whenever problem occurs, either few threads (two or
three) are having high load that sums to 100%, or there are some dozen of
them, each taking  some 5% of CPU - load is distributed quite evenly, but
it all sums up to 100%.

However i did found some similar backtrace with similar problem in google:

http://www.linuxforums.org/forum/redhat-fedora-linux-help/46781-cpu-usage-goes-100-when-pthread_cond_wait-being-used.html

I noticed that i have the same version of glibc -
pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0 related to
IAX2 functions. So, as i use IAX for connecting to Hylafax, i re-routed
IAXmodem to another machine, and i'm currently waiting to see some
improvements (loadavg already have dropped a little).

I'll let you know on further situation regarding this, as soon as i'll
have it. Meanwhile i would appreciate if you could confirm my theory as
possible. 

Issue History 
Date Modified   Username       Field                    Change               
====================================================================== 
09-27-07 10:31  atis           Note Added: 0071157                          
======================================================================