[asterisk-users] puzzle

Wed Nov 19 19:02:30 CST 2008

Hi Steve,

[root at ast ~]# ls -ltr /etc/init.d
lrwxrwxrwx  1 root root 11 Nov 29  2007 /etc/init.d -> rc.d/init.d
[root at ast ~]#

Although I agree that updating the kernel et all would be a good idea, the 
whole point is to keep the machine running for 19 more days without the 
rogue process interfering with my voice quality.  If I cannot unload the 
module or otherwise interrupt the process which is currently spinning in 
kernel space, no upgrade will be possible.  I am quite sure that rebooting 
will fix this problem, but the "puzzle" was to fix it without doing so...

Cheers,

j

On Wed, 19 Nov 2008, Steve Totaro wrote:

> Well then use whatever package manager you have.  Apt-get I assume.
> Maybe that might help.
>
> What do you get with "#ls -ltr /etc/init.d"?
> -- 
> Thanks,
> Steve Totaro
> +18887771888 (Toll Free)
> +12409381212 (Cell)
> +12024369784 (Skype)
>
> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>
>> Its not Centos - there is no 'yum'.
>> "service iptables stop" is what
>> produced the hanging process in the first place - I think my big problem
>> here is that a kernel module is broken, and there is no way to stop it,
>> and there seems to be no way to unload it (in fact it is hung trying to do
>> just that).
>>
>> Thanks for the suggestions, though!
>>
>> j
>>
>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>
>>> YUM update?  "service iptables stop" "service iptables start"?
>>>
>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>
>>>> Hmm, I am more of a BSD guy I guess.  I would expect a pipe to show a 'p'
>>>> in a long ls.  This is interesting though:
>>>>
>>>> [root at ast init.d]# cat /proc/modules | head
>>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>>> [root at ast init.d]# rmmod -f ip_conntrack
>>>> ERROR: Removing 'ip_conntrack': Device or resource busy
>>>>
>>>> (sigh)
>>>>
>>>> I am pretty sure ip_conntrack is part of the iptables stuff...
>>>>
>>>> j
>>>>
>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>
>>>>> /proc/modules is a pipe
>>>>> You can see what is in there by type cat /proc/modules|more
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>> LaCoursiere
>>>>> Sent: Wednesday, November 19, 2008 2:47 PM
>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>
>>>>>
>>>>> A good idea!  The modprobe command is actually in the ps below - it is
>>>>> part of the /etc/init.d/iptables script, and apparently was trying to
>>>>> remove the ipt_state module.  The result, however:
>>>>>
>>>>> [root at ast init.d]# rmmod ipt_state
>>>>> ERROR: Module ipt_state does not exist in /proc/modules
>>>>>
>>>>> (sigh).  In fact /proc/modules is empty.
>>>>>
>>>>> [root at ast init.d]# ls -ltr /proc/modules
>>>>> -r--r--r--  1 root root 0 Nov 19 14:46 /proc/modules
>>>>>
>>>>> j
>>>>>
>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>
>>>>>> Your could try this
>>>>>> History|grep modprobe
>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe.
>>>>>> This of course assumes that the command is in your last 1000 commands.
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>> LaCoursiere
>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM
>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>
>>>>>>
>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which means
>>>>>> its real parent died already.
>>>>>>
>>>>>> Any attempt to flush the iptables hangs :(
>>>>>>
>>>>>> j
>>>>>>
>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>
>>>>>>> Have you done a ps -elf to see if the process has a parent that is
>>>>>>> re-launching or preserving it?
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>> LaCoursiere
>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM
>>>>>>> To: asterisk-users at lists.digium.com
>>>>>>> Subject: [asterisk-users] puzzle
>>>>>>>
>>>>>>>
>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue
>>>>> does
>>>>>>> affect the voice performance I am experiencing, so I am soothing my guilt
>>>>>>> with that.
>>>>>>>
>>>>>>> Bet you don't see this every day:
>>>>>>>
>>>>>>> ast% uptime
>>>>>>>  13:48:08 up 981 days, 18:29,  1 user,  load average: 1.08, 1.02, 1.01
>>>>>>> ast%
>>>>>>>
>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing
>>>>> other
>>>>>>> than bragging rights.  Its been through mysql and asterisk upgrades, a
>>>>>>> horrible hacking nightmare that very nearly made me reboot, and several
>>>>>>> power outages where the batteries lasted JUST long enough to keep her up.
>>>>>>>
>>>>>>> After all of this, I find I may have to reboot after all.  Because there
>>>>>>> is a !$@#% process running, consuming 100% CPU (note the load average),
>>>>>>> and I cannot seem to kill it:
>>>>>>>
>>>>>>> ast% ps auxw | grep modprobe
>>>>>>> root     17744 99.9  0.0  2688  412 ?        RN   Nov03 23223:01 modprobe
>>>>>>> -r ipt_state
>>>>>>> ast% ps ealx | grep modprobe | grep -v grep
>>>>>>> 4     0 17744     1  39  19  2688  412 -      RN   ?        23223:38
>>>>>>> modprobe -r ipt_state
>>>>>>> ast% sudo kill 17744
>>>>>>> ast% sudo kill 17744
>>>>>>> ast% sudo kill -9 17744
>>>>>>> ast% sudo kill -9 17744
>>>>>>> ast% !ps
>>>>>>> ps ealx | grep modprobe | grep -v grep
>>>>>>> 4     0 17744     1  39  19  2688  412 -      RN   ?        23224:41
>>>>>>> modprobe -r ipt_state
>>>>>>> ast%
>>>>>>>
>>>>>>> You may also notice that I tried "renice" to bump it all the way to +19
>>>>>>> and still it consumes 100% of the CPU.  The result for asterisk is that I
>>>>>>> hear bits of robot noise during conversations, which is annoying as hell
>>>>>>> but not neccessarily show stopping.  But for another 19 days??  Argg!
>>>>>>>
>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug
>>>>> that
>>>>>>> is merrily spinning away and won't respond to interrupts.  I even tried
>>>>> to
>>>>>>> stop it with gdb and strace, both of which also hung and had to be killed
>>>>>>> with -9.
>>>>>>>
>>>>>>> It seems to be related to me screwing with the iptables a few weeks ago.
>>>>>>>
>>>>>>> Any ideas other than rebooting?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Steve Totaro
>>> +18887771888 (Toll Free)
>>> +12409381212 (Cell)
>>> +12024369784 (Skype)
>>>
>>> _______________________________________________
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>> _______________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
> _______________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>