[asterisk-users] puzzle
Jeff LaCoursiere
jeff at jeff.net
Wed Nov 19 19:58:17 CST 2008
No... isn't that a GUI? This is a colo'ed server running a prepaid
calling card app.
Cheers,
j
On Wed, 19 Nov 2008, Steve Totaro wrote:
> Are you using NetworkManager?
>
> On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>
>> Happy for all suggestions, of course! No offense intended with my reply.
>> Not sure what you are trying to get at with init.d, but here you go:
>>
>> ast% ls /etc/init.d
>> /etc/init.d@
>>
>> Am guessing you expected a bit more than that, so allow me to assume what
>> you are looking for (and sum up the state of the discussion):
>>
>> ast% ls -ltr /etc/init.d/iptables
>> -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables*
>> ast%
>>
>> This is of course the script that loads and unloads the kernel modules
>> associated with iptables, and was run as "iptables stop" on November 3rd
>> which caused the problem in discussion. One of the lines in this script
>> does a "modprobe -r ipt_state" which hung. Apparently the actual module
>> which hung while unloading is ip_conntrack:
>>
>> ast% cat /proc/modules | head
>> ip_conntrack 45573 0 - Unloading 0xf8945000
>> [snip]
>>
>> Some bug in this module must be in an infinite loop in kernel space, as
>> the process eats 100% of the CPU even when reniced to +19. Because of
>> this the modprobe process cannot be killed and won't respond to
>> interrupts.
>>
>> An attempt at getting a kernel stack trace failed, which is extremely
>> unfortunate :( Cool to learn about /proc/sysrq-trigger, though!
>>
>> Did I miss anything?
>>
>> Cheers,
>>
>> j
>>
>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>
>>> I was not implying that you upgrade anything but iptables.
>>>
>>> What is the output of "ls /etc/init.d/"
>>>
>>> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>
>>>> Hi Steve,
>>>>
>>>> [root at ast ~]# ls -ltr /etc/init.d
>>>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d
>>>> [root at ast ~]#
>>>>
>>>> Although I agree that updating the kernel et all would be a good idea, the
>>>> whole point is to keep the machine running for 19 more days without the
>>>> rogue process interfering with my voice quality. If I cannot unload the
>>>> module or otherwise interrupt the process which is currently spinning in
>>>> kernel space, no upgrade will be possible. I am quite sure that rebooting
>>>> will fix this problem, but the "puzzle" was to fix it without doing so...
>>>>
>>>> Cheers,
>>>>
>>>> j
>>>>
>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>
>>>>> Well then use whatever package manager you have. Apt-get I assume.
>>>>> Maybe that might help.
>>>>>
>>>>> What do you get with "#ls -ltr /etc/init.d"?
>>>>> --
>>>>> Thanks,
>>>>> Steve Totaro
>>>>> +18887771888 (Toll Free)
>>>>> +12409381212 (Cell)
>>>>> +12024369784 (Skype)
>>>>>
>>>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>
>>>>>> Its not Centos - there is no 'yum'.
>>>>>> "service iptables stop" is what
>>>>>> produced the hanging process in the first place - I think my big problem
>>>>>> here is that a kernel module is broken, and there is no way to stop it,
>>>>>> and there seems to be no way to unload it (in fact it is hung trying to do
>>>>>> just that).
>>>>>>
>>>>>> Thanks for the suggestions, though!
>>>>>>
>>>>>> j
>>>>>>
>>>>>> On Wed, 19 Nov 2008, Steve Totaro wrote:
>>>>>>
>>>>>>> YUM update? "service iptables stop" "service iptables start"?
>>>>>>>
>>>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <jeff at jeff.net> wrote:
>>>>>>>>
>>>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p'
>>>>>>>> in a long ls. This is interesting though:
>>>>>>>>
>>>>>>>> [root at ast init.d]# cat /proc/modules | head
>>>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000
>>>>>>>> [root at ast init.d]# rmmod -f ip_conntrack
>>>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy
>>>>>>>>
>>>>>>>> (sigh)
>>>>>>>>
>>>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff...
>>>>>>>>
>>>>>>>> j
>>>>>>>>
>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>
>>>>>>>>> /proc/modules is a pipe
>>>>>>>>> You can see what is in there by type cat /proc/modules|more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>> LaCoursiere
>>>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM
>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A good idea! The modprobe command is actually in the ps below - it is
>>>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to
>>>>>>>>> remove the ipt_state module. The result, however:
>>>>>>>>>
>>>>>>>>> [root at ast init.d]# rmmod ipt_state
>>>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules
>>>>>>>>>
>>>>>>>>> (sigh). In fact /proc/modules is empty.
>>>>>>>>>
>>>>>>>>> [root at ast init.d]# ls -ltr /proc/modules
>>>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules
>>>>>>>>>
>>>>>>>>> j
>>>>>>>>>
>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>
>>>>>>>>>> Your could try this
>>>>>>>>>> History|grep modprobe
>>>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe.
>>>>>>>>>> This of course assumes that the command is in your last 1000 commands.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>> LaCoursiere
>>>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM
>>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion
>>>>>>>>>> Subject: Re: [asterisk-users] puzzle
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which means
>>>>>>>>>> its real parent died already.
>>>>>>>>>>
>>>>>>>>>> Any attempt to flush the iptables hangs :(
>>>>>>>>>>
>>>>>>>>>> j
>>>>>>>>>>
>>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote:
>>>>>>>>>>
>>>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is
>>>>>>>>>>> re-launching or preserving it?
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: asterisk-users-bounces at lists.digium.com
>>>>>>>>>>> [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff
>>>>>>>>>>> LaCoursiere
>>>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM
>>>>>>>>>>> To: asterisk-users at lists.digium.com
>>>>>>>>>>> Subject: [asterisk-users] puzzle
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue
>>>>>>>>> does
>>>>>>>>>>> affect the voice performance I am experiencing, so I am soothing my guilt
>>>>>>>>>>> with that.
>>>>>>>>>>>
>>>>>>>>>>> Bet you don't see this every day:
>>>>>>>>>>>
>>>>>>>>>>> ast% uptime
>>>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01
>>>>>>>>>>> ast%
>>>>>>>>>>>
>>>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing
>>>>>>>>> other
>>>>>>>>>>> than bragging rights. Its been through mysql and asterisk upgrades, a
>>>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and several
>>>>>>>>>>> power outages where the batteries lasted JUST long enough to keep her up.
>>>>>>>>>>>
>>>>>>>>>>> After all of this, I find I may have to reboot after all. Because there
>>>>>>>>>>> is a !$@#% process running, consuming 100% CPU (note the load average),
>>>>>>>>>>> and I cannot seem to kill it:
>>>>>>>>>>>
>>>>>>>>>>> ast% ps auxw | grep modprobe
>>>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 modprobe
>>>>>>>>>>> -r ipt_state
>>>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep
>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23223:38
>>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>>> ast% sudo kill 17744
>>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>>> ast% sudo kill -9 17744
>>>>>>>>>>> ast% !ps
>>>>>>>>>>> ps ealx | grep modprobe | grep -v grep
>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23224:41
>>>>>>>>>>> modprobe -r ipt_state
>>>>>>>>>>> ast%
>>>>>>>>>>>
>>>>>>>>>>> You may also notice that I tried "renice" to bump it all the way to +19
>>>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is that I
>>>>>>>>>>> hear bits of robot noise during conversations, which is annoying as hell
>>>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? Argg!
>>>>>>>>>>>
>>>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug
>>>>>>>>> that
>>>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even tried
>>>>>>>>> to
>>>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be killed
>>>>>>>>>>> with -9.
>>>>>>>>>>>
>>>>>>>>>>> It seems to be related to me screwing with the iptables a few weeks ago.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas other than rebooting?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> j
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>
>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>>
>>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>>
>>>>>>>>>> asterisk-users mailing list
>>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>>
>>>>>>>>> asterisk-users mailing list
>>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>>
>>>>>>>> asterisk-users mailing list
>>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Steve Totaro
>>>>>>> +18887771888 (Toll Free)
>>>>>>> +12409381212 (Cell)
>>>>>>> +12024369784 (Skype)
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>>
>>>>>>> asterisk-users mailing list
>>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>>
>>>>>> asterisk-users mailing list
>>>>>> To UNSUBSCRIBE or update options visit:
>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>>
>>>>> asterisk-users mailing list
>>>>> To UNSUBSCRIBE or update options visit:
>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Steve Totaro
>>> +18887771888 (Toll Free)
>>> +12409381212 (Cell)
>>> +12024369784 (Skype)
>>>
>>> _______________________________________________
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>> _______________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>> http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
>
>
> --
> Thanks,
> Steve Totaro
> +18887771888 (Toll Free)
> +12409381212 (Cell)
> +12024369784 (Skype)
>
> _______________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>
More information about the asterisk-users
mailing list