[asterisk-users] puzzle
Jeff LaCoursiere
jeff at jeff.net
Wed Nov 19 19:29:59 CST 2008
Happy for all suggestions, of course! No offense intended with my reply.
Not sure what you are trying to get at with init.d, but here you go:
ast% ls /etc/init.d
/etc/init.d@
Am guessing you expected a bit more than that, so allow me to assume what
you are looking for (and sum up the state of the discussion):
ast% ls -ltr /etc/init.d/iptables
-rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables*
ast%
This is of course the script that loads and unloads the kernel modules
associated with iptables, and was run as "iptables stop" on November 3rd
which caused the problem in discussion. One of the lines in this script
does a "modprobe -r ipt_state" which hung. Apparently the actual module
which hung while unloading is ip_conntrack:
ast% cat /proc/modules | head
ip_conntrack 45573 0 - Unloading 0xf8945000
[snip]
Some bug in this module must be in an infinite loop in kernel space, as
the process eats 100% of the CPU even when reniced to +19. Because of
this the modprobe process cannot be killed and won't respond to
interrupts.
An attempt at getting a kernel stack trace failed, which is extremely
unfortunate :( Cool to learn about /proc/sysrq-trigger, though!
Did I miss anything?
Cheers,
j
On Wed, 19 Nov 2008, Steve Totaro wrote:
> I was not implying that you upgrade anything but iptables.
>
> What is the output of "ls /etc/init.d/"
>
> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>
>> Hi Steve,
>>
>> [root at ast ~]# ls -ltr /etc/init.d
>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d
>> [root at ast ~]#
>>
>> Although I agree that updating the kernel et all would be a good idea, the
>> whole point is to keep the machine running for 19 more days without the
>> rogue process interfering with my voice quality. If I cannot unload the
>> module or otherwise interrupt the process which is currently spinning in
>> kernel space, no upgrade will be possible. I am quite sure that rebooting
>> will fix this problem, but the "puzzle" was to fix it without doing so...
>>
>> Cheers,
>>
>> j
>>
>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>
>>> Well then use whatever package manager you have. Apt-get I assume.
>>> Maybe that might help.
>>>
>>> What do you get with "#ls -ltr /etc/init.d"?
>>> --
>>> Thanks,
>>> Steve Totaro
>>> +18887771888 (Toll Free)
>>> +12409381212 (Cell)
>>> +12024369784 (Skype)
>>>
>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>
>>>> Its not Centos - there is no 'yum'.
>>>> "service iptables stop" is what
>>>> produced the hanging process in the first place - I think my big problem
>>>> here is that a kernel module is broken, and there is no way to stop it,
>>>> and there seems to be no way to unload it (in fact it is hung trying to do
>>>> just that).
>>>>
>>>> Thanks for the suggestions, though!
>>>>
>>>> j
>>>>
>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>
>>>>> YUM update? "service iptables stop" "service iptables start"?
>>>>>
>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>
>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p'
>>>>>> in a long ls. This is interesting though:
>>>>>>
>>>>>> [root at ast init.d]# cat /proc/modules | head
>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>>>>> [root at ast init.d]# rmmod -f ip_conntrack
>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy
>>>>>>
>>>>>> (sigh)
>>>>>>
>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff...
>>>>>>
>>>>>> j
>>>>>>
>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>
>>>>>>> /proc/modules is a pipe
>>>>>>> You can see what is in there by type cat /proc/modules|more
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>> LaCoursiere
>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM
>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>
>>>>>>>
>>>>>>> A good idea! The modprobe command is actually in the ps below - it is
>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to
>>>>>>> remove the ipt_state module. The result, however:
>>>>>>>
>>>>>>> [root at ast init.d]# rmmod ipt_state
>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules
>>>>>>>
>>>>>>> (sigh). In fact /proc/modules is empty.
>>>>>>>
>>>>>>> [root at ast init.d]# ls -ltr /proc/modules
>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules
>>>>>>>
>>>>>>> j
>>>>>>>
>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>
>>>>>>>> Your could try this
>>>>>>>> History|grep modprobe
>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe.
>>>>>>>> This of course assumes that the command is in your last 1000 commands.
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>> LaCoursiere
>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM
>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which means
>>>>>>>> its real parent died already.
>>>>>>>>
>>>>>>>> Any attempt to flush the iptables hangs :(
>>>>>>>>
>>>>>>>> j
>>>>>>>>
>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>
>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is
>>>>>>>>> re-launching or preserving it?
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>> LaCoursiere
>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM
>>>>>>>>> To: asterisk-users at lists.digium.com
>>>>>>>>> Subject: [asterisk-users] puzzle
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue
>>>>>>> does
>>>>>>>>> affect the voice performance I am experiencing, so I am soothing my guilt
>>>>>>>>> with that.
>>>>>>>>>
>>>>>>>>> Bet you don't see this every day:
>>>>>>>>>
>>>>>>>>> ast% uptime
>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01
>>>>>>>>> ast%
>>>>>>>>>
>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing
>>>>>>> other
>>>>>>>>> than bragging rights. Its been through mysql and asterisk upgrades, a
>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and several
>>>>>>>>> power outages where the batteries lasted JUST long enough to keep her up.
>>>>>>>>>
>>>>>>>>> After all of this, I find I may have to reboot after all. Because there
>>>>>>>>> is a !$@#% process running, consuming 100% CPU (note the load average),
>>>>>>>>> and I cannot seem to kill it:
>>>>>>>>>
>>>>>>>>> ast% ps auxw | grep modprobe
>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 modprobe
>>>>>>>>> -r ipt_state
>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep
>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23223:38
>>>>>>>>> modprobe -r ipt_state
>>>>>>>>> ast% sudo kill 17744
>>>>>>>>> ast% sudo kill 17744
>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>> ast% !ps
>>>>>>>>> ps ealx | grep modprobe | grep -v grep
>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23224:41
>>>>>>>>> modprobe -r ipt_state
>>>>>>>>> ast%
>>>>>>>>>
>>>>>>>>> You may also notice that I tried "renice" to bump it all the way to +19
>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is that I
>>>>>>>>> hear bits of robot noise during conversations, which is annoying as hell
>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? Argg!
>>>>>>>>>
>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug
>>>>>>> that
>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even tried
>>>>>>> to
>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be killed
>>>>>>>>> with -9.
>>>>>>>>>
>>>>>>>>> It seems to be related to me screwing with the iptables a few weeks ago.
>>>>>>>>>
>>>>>>>>> Any ideas other than rebooting?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> j
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Steve Totaro
>>>>> +18887771888 (Toll Free)
>>>>> +12409381212 (Cell)
>>>>> +12024369784 (Skype)
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>> _______________________________________________
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>> _______________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
>
>
> --
> Thanks,
> Steve Totaro
> +18887771888 (Toll Free)
> +12409381212 (Cell)
> +12024369784 (Skype)
>
> _______________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>
More information about the asterisk-users
mailing list