[asterisk-users] puzzle
Steve Totaro
stotaro at totarotechnologies.com
Wed Nov 19 20:50:15 CST 2008
Well that is a puzzle, sort of like the crypto outside of the CIA.
Just reboot and upgrade if you feel a need.
BUT, if I were you, I would figure it out and hit 1,000, but if it is
a production box, then just reboot, time is money.....
I see quite a few people with the same issue while googling, old
kernels and various distros.
One fix was to stop NetworkManager. I don't know about your
particular server, but did you set it up? If not, maybe "service
NetworkManager stop" may do the trick.
Thanks,
Steve Totaro
On Wed, Nov 19, 2008 at 8:58 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>
> No... isn't that a GUI? This is a colo'ed server running a prepaid
> calling card app.
>
> Cheers,
>
> j
>
> On Wed, 19 Nov 2008, Steve Totaro wrote:
>
>> Are you using NetworkManager?
>>
>> On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>
>>> Happy for all suggestions, of course! No offense intended with my reply.
>>> Not sure what you are trying to get at with init.d, but here you go:
>>>
>>> ast% ls /etc/init.d
>>> /etc/init.d@
>>>
>>> Am guessing you expected a bit more than that, so allow me to assume what
>>> you are looking for (and sum up the state of the discussion):
>>>
>>> ast% ls -ltr /etc/init.d/iptables
>>> -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables*
>>> ast%
>>>
>>> This is of course the script that loads and unloads the kernel modules
>>> associated with iptables, and was run as "iptables stop" on November 3rd
>>> which caused the problem in discussion. One of the lines in this script
>>> does a "modprobe -r ipt_state" which hung. Apparently the actual module
>>> which hung while unloading is ip_conntrack:
>>>
>>> ast% cat /proc/modules | head
>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>> [snip]
>>>
>>> Some bug in this module must be in an infinite loop in kernel space, as
>>> the process eats 100% of the CPU even when reniced to +19. Because of
>>> this the modprobe process cannot be killed and won't respond to
>>> interrupts.
>>>
>>> An attempt at getting a kernel stack trace failed, which is extremely
>>> unfortunate :( Cool to learn about /proc/sysrq-trigger, though!
>>>
>>> Did I miss anything?
>>>
>>> Cheers,
>>>
>>> j
>>>
>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>
>>>> I was not implying that you upgrade anything but iptables.
>>>>
>>>> What is the output of "ls /etc/init.d/"
>>>>
>>>> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>
>>>>> Hi Steve,
>>>>>
>>>>> [root at ast ~]# ls -ltr /etc/init.d
>>>>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d
>>>>> [root at ast ~]#
>>>>>
>>>>> Although I agree that updating the kernel et all would be a good idea, the
>>>>> whole point is to keep the machine running for 19 more days without the
>>>>> rogue process interfering with my voice quality. If I cannot unload the
>>>>> module or otherwise interrupt the process which is currently spinning in
>>>>> kernel space, no upgrade will be possible. I am quite sure that rebooting
>>>>> will fix this problem, but the "puzzle" was to fix it without doing so...
>>>>>
>>>>> Cheers,
>>>>>
>>>>> j
>>>>>
>>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>>
>>>>>> Well then use whatever package manager you have. Apt-get I assume.
>>>>>> Maybe that might help.
>>>>>>
>>>>>> What do you get with "#ls -ltr /etc/init.d"?
>>>>>> --
>>>>>> Thanks,
>>>>>> Steve Totaro
>>>>>> +18887771888 (Toll Free)
>>>>>> +12409381212 (Cell)
>>>>>> +12024369784 (Skype)
>>>>>>
>>>>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>>
>>>>>>> Its not Centos - there is no 'yum'.
>>>>>>> "service iptables stop" is what
>>>>>>> produced the hanging process in the first place - I think my big problem
>>>>>>> here is that a kernel module is broken, and there is no way to stop it,
>>>>>>> and there seems to be no way to unload it (in fact it is hung trying to do
>>>>>>> just that).
>>>>>>>
>>>>>>> Thanks for the suggestions, though!
>>>>>>>
>>>>>>> j
>>>>>>>
>>>>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>>>>
>>>>>>>> YUM update? "service iptables stop" "service iptables start"?
>>>>>>>>
>>>>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>>>>
>>>>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p'
>>>>>>>>> in a long ls. This is interesting though:
>>>>>>>>>
>>>>>>>>> [root at ast init.d]# cat /proc/modules | head
>>>>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>>>>>>>> [root at ast init.d]# rmmod -f ip_conntrack
>>>>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy
>>>>>>>>>
>>>>>>>>> (sigh)
>>>>>>>>>
>>>>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff...
>>>>>>>>>
>>>>>>>>> j
>>>>>>>>>
>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>
>>>>>>>>>> /proc/modules is a pipe
>>>>>>>>>> You can see what is in there by type cat /proc/modules|more
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>> LaCoursiere
>>>>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM
>>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> A good idea! The modprobe command is actually in the ps below - it is
>>>>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to
>>>>>>>>>> remove the ipt_state module. The result, however:
>>>>>>>>>>
>>>>>>>>>> [root at ast init.d]# rmmod ipt_state
>>>>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules
>>>>>>>>>>
>>>>>>>>>> (sigh). In fact /proc/modules is empty.
>>>>>>>>>>
>>>>>>>>>> [root at ast init.d]# ls -ltr /proc/modules
>>>>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules
>>>>>>>>>>
>>>>>>>>>> j
>>>>>>>>>>
>>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>>
>>>>>>>>>>> Your could try this
>>>>>>>>>>> History|grep modprobe
>>>>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe.
>>>>>>>>>>> This of course assumes that the command is in your last 1000 commands.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>>> LaCoursiere
>>>>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM
>>>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which means
>>>>>>>>>>> its real parent died already.
>>>>>>>>>>>
>>>>>>>>>>> Any attempt to flush the iptables hangs :(
>>>>>>>>>>>
>>>>>>>>>>> j
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is
>>>>>>>>>>>> re-launching or preserving it?
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>>>> LaCoursiere
>>>>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM
>>>>>>>>>>>> To: asterisk-users at lists.digium.com
>>>>>>>>>>>> Subject: [asterisk-users] puzzle
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue
>>>>>>>>>> does
>>>>>>>>>>>> affect the voice performance I am experiencing, so I am soothing my guilt
>>>>>>>>>>>> with that.
>>>>>>>>>>>>
>>>>>>>>>>>> Bet you don't see this every day:
>>>>>>>>>>>>
>>>>>>>>>>>> ast% uptime
>>>>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01
>>>>>>>>>>>> ast%
>>>>>>>>>>>>
>>>>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing
>>>>>>>>>> other
>>>>>>>>>>>> than bragging rights. Its been through mysql and asterisk upgrades, a
>>>>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and several
>>>>>>>>>>>> power outages where the batteries lasted JUST long enough to keep her up.
>>>>>>>>>>>>
>>>>>>>>>>>> After all of this, I find I may have to reboot after all. Because there
>>>>>>>>>>>> is a !$@#% process running, consuming 100% CPU (note the load average),
>>>>>>>>>>>> and I cannot seem to kill it:
>>>>>>>>>>>>
>>>>>>>>>>>> ast% ps auxw | grep modprobe
>>>>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 modprobe
>>>>>>>>>>>> -r ipt_state
>>>>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep
>>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23223:38
>>>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>>>> ast% !ps
>>>>>>>>>>>> ps ealx | grep modprobe | grep -v grep
>>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23224:41
>>>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>>>> ast%
>>>>>>>>>>>>
>>>>>>>>>>>> You may also notice that I tried "renice" to bump it all the way to +19
>>>>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is that I
>>>>>>>>>>>> hear bits of robot noise during conversations, which is annoying as hell
>>>>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? Argg!
>>>>>>>>>>>>
>>>>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug
>>>>>>>>>> that
>>>>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even tried
>>>>>>>>>> to
>>>>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be killed
>>>>>>>>>>>> with -9.
>>>>>>>>>>>>
>>>>>>>>>>>> It seems to be related to me screwing with the iptables a few weeks ago.
>>>>>>>>>>>>
>>>>>>>>>>>> Any ideas other than rebooting?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> j
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>>
>>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>>
>>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>
>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>
>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Steve Totaro
>>>>>>>> +18887771888 (Toll Free)
>>>>>>>> +12409381212 (Cell)
>>>>>>>> +12024369784 (Skype)
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Steve Totaro
>>>> +18887771888 (Toll Free)
>>>> +12409381212 (Cell)
>>>> +12024369784 (Skype)
>>>>
>>>> _______________________________________________
>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>> _______________________________________________
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>>
>>
>> --
>> Thanks,
>> Steve Totaro
>> +18887771888 (Toll Free)
>> +12409381212 (Cell)
>> +12024369784 (Skype)
>>
>> _______________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
> _______________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>
--
Thanks,
Steve Totaro
+18887771888 (Toll Free)
+12409381212 (Cell)
+12024369784 (Skype)
More information about the asterisk-users
mailing list