[asterisk-users] puzzle
Steve Totaro
stotaro at totarotechnologies.com
Wed Nov 19 19:32:41 CST 2008
Are you using NetworkManager?
On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>
> Happy for all suggestions, of course! No offense intended with my reply.
> Not sure what you are trying to get at with init.d, but here you go:
>
> ast% ls /etc/init.d
> /etc/init.d@
>
> Am guessing you expected a bit more than that, so allow me to assume what
> you are looking for (and sum up the state of the discussion):
>
> ast% ls -ltr /etc/init.d/iptables
> -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables*
> ast%
>
> This is of course the script that loads and unloads the kernel modules
> associated with iptables, and was run as "iptables stop" on November 3rd
> which caused the problem in discussion. One of the lines in this script
> does a "modprobe -r ipt_state" which hung. Apparently the actual module
> which hung while unloading is ip_conntrack:
>
> ast% cat /proc/modules | head
> ip_conntrack 45573 0 - Unloading 0xf8945000
> [snip]
>
> Some bug in this module must be in an infinite loop in kernel space, as
> the process eats 100% of the CPU even when reniced to +19. Because of
> this the modprobe process cannot be killed and won't respond to
> interrupts.
>
> An attempt at getting a kernel stack trace failed, which is extremely
> unfortunate :( Cool to learn about /proc/sysrq-trigger, though!
>
> Did I miss anything?
>
> Cheers,
>
> j
>
> On Wed, 19 Nov 2008, Steve Totaro wrote:
>
>> I was not implying that you upgrade anything but iptables.
>>
>> What is the output of "ls /etc/init.d/"
>>
>> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>
>>> Hi Steve,
>>>
>>> [root at ast ~]# ls -ltr /etc/init.d
>>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d
>>> [root at ast ~]#
>>>
>>> Although I agree that updating the kernel et all would be a good idea, the
>>> whole point is to keep the machine running for 19 more days without the
>>> rogue process interfering with my voice quality. If I cannot unload the
>>> module or otherwise interrupt the process which is currently spinning in
>>> kernel space, no upgrade will be possible. I am quite sure that rebooting
>>> will fix this problem, but the "puzzle" was to fix it without doing so...
>>>
>>> Cheers,
>>>
>>> j
>>>
>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>
>>>> Well then use whatever package manager you have. Apt-get I assume.
>>>> Maybe that might help.
>>>>
>>>> What do you get with "#ls -ltr /etc/init.d"?
>>>> --
>>>> Thanks,
>>>> Steve Totaro
>>>> +18887771888 (Toll Free)
>>>> +12409381212 (Cell)
>>>> +12024369784 (Skype)
>>>>
>>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>
>>>>> Its not Centos - there is no 'yum'.
>>>>> "service iptables stop" is what
>>>>> produced the hanging process in the first place - I think my big problem
>>>>> here is that a kernel module is broken, and there is no way to stop it,
>>>>> and there seems to be no way to unload it (in fact it is hung trying to do
>>>>> just that).
>>>>>
>>>>> Thanks for the suggestions, though!
>>>>>
>>>>> j
>>>>>
>>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>>
>>>>>> YUM update? "service iptables stop" "service iptables start"?
>>>>>>
>>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>>
>>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p'
>>>>>>> in a long ls. This is interesting though:
>>>>>>>
>>>>>>> [root at ast init.d]# cat /proc/modules | head
>>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>>>>>> [root at ast init.d]# rmmod -f ip_conntrack
>>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy
>>>>>>>
>>>>>>> (sigh)
>>>>>>>
>>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff...
>>>>>>>
>>>>>>> j
>>>>>>>
>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>
>>>>>>>> /proc/modules is a pipe
>>>>>>>> You can see what is in there by type cat /proc/modules|more
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>> LaCoursiere
>>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM
>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>
>>>>>>>>
>>>>>>>> A good idea! The modprobe command is actually in the ps below - it is
>>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to
>>>>>>>> remove the ipt_state module. The result, however:
>>>>>>>>
>>>>>>>> [root at ast init.d]# rmmod ipt_state
>>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules
>>>>>>>>
>>>>>>>> (sigh). In fact /proc/modules is empty.
>>>>>>>>
>>>>>>>> [root at ast init.d]# ls -ltr /proc/modules
>>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules
>>>>>>>>
>>>>>>>> j
>>>>>>>>
>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>
>>>>>>>>> Your could try this
>>>>>>>>> History|grep modprobe
>>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe.
>>>>>>>>> This of course assumes that the command is in your last 1000 commands.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>> LaCoursiere
>>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM
>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which means
>>>>>>>>> its real parent died already.
>>>>>>>>>
>>>>>>>>> Any attempt to flush the iptables hangs :(
>>>>>>>>>
>>>>>>>>> j
>>>>>>>>>
>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>
>>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is
>>>>>>>>>> re-launching or preserving it?
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>> LaCoursiere
>>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM
>>>>>>>>>> To: asterisk-users at lists.digium.com
>>>>>>>>>> Subject: [asterisk-users] puzzle
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue
>>>>>>>> does
>>>>>>>>>> affect the voice performance I am experiencing, so I am soothing my guilt
>>>>>>>>>> with that.
>>>>>>>>>>
>>>>>>>>>> Bet you don't see this every day:
>>>>>>>>>>
>>>>>>>>>> ast% uptime
>>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01
>>>>>>>>>> ast%
>>>>>>>>>>
>>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing
>>>>>>>> other
>>>>>>>>>> than bragging rights. Its been through mysql and asterisk upgrades, a
>>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and several
>>>>>>>>>> power outages where the batteries lasted JUST long enough to keep her up.
>>>>>>>>>>
>>>>>>>>>> After all of this, I find I may have to reboot after all. Because there
>>>>>>>>>> is a !$@#% process running, consuming 100% CPU (note the load average),
>>>>>>>>>> and I cannot seem to kill it:
>>>>>>>>>>
>>>>>>>>>> ast% ps auxw | grep modprobe
>>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 modprobe
>>>>>>>>>> -r ipt_state
>>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep
>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23223:38
>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>> ast% !ps
>>>>>>>>>> ps ealx | grep modprobe | grep -v grep
>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23224:41
>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>> ast%
>>>>>>>>>>
>>>>>>>>>> You may also notice that I tried "renice" to bump it all the way to +19
>>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is that I
>>>>>>>>>> hear bits of robot noise during conversations, which is annoying as hell
>>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? Argg!
>>>>>>>>>>
>>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug
>>>>>>>> that
>>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even tried
>>>>>>>> to
>>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be killed
>>>>>>>>>> with -9.
>>>>>>>>>>
>>>>>>>>>> It seems to be related to me screwing with the iptables a few weeks ago.
>>>>>>>>>>
>>>>>>>>>> Any ideas other than rebooting?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> j
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Steve Totaro
>>>>>> +18887771888 (Toll Free)
>>>>>> +12409381212 (Cell)
>>>>>> +12024369784 (Skype)
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>> _______________________________________________
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>>
>>
>> --
>> Thanks,
>> Steve Totaro
>> +18887771888 (Toll Free)
>> +12409381212 (Cell)
>> +12024369784 (Skype)
>>
>> _______________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
> _______________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>
--
Thanks,
Steve Totaro
+18887771888 (Toll Free)
+12409381212 (Cell)
+12024369784 (Skype)
More information about the asterisk-users
mailing list