[asterisk-users] How are your PRI interrupts balanced? (+ Soft lockup BUG)

[James Lamanna]

On Mon, Mar 29, 2010 at 8:38 PM, Matt Watson <matt at mattgwatson.ca> wrote:
> Dell server by any chance?
> I have a similar problem with a TE220B in a Dell 1950 III server - i've seen
> several other people having issues with digium cards in dell servers as
> well.
> I've actually done something similar to what you have done - isolated the
> TE220B onto its own IRQ and set processor affinity for all the IRQs to
> particular cores... so far I haven't had kernel pancs since doing this, but
> its still a little too early to say if it has fixed the issue 100% or not.

Interesting. It is actually a Dell SC1425 - Dual, dual-core Xeon Processors.
I'm hopefully going to be able to stress test this machine to see if I
can make it panic again with the PRI card IRQ isolated to CPU0. If so,
I'll see if it does the same thing on the other cores...

-- James

> --
> Matt
>
> On Mon, Mar 29, 2010 at 8:30 PM, James Lamanna <jlamanna at gmail.com> wrote:
>>
>> Hi,
>> I'm trying to figure out the cause of a soft lockup I experienced:
>>
>> Mar 29 09:38:24 pstn1 kernel: BUG: soft lockup - CPU#0 stuck for 10s!
>> [asterisk:32029]
>> Mar 29 09:38:24 pstn1 kernel: Pid: 32029, comm:             asterisk
>> Mar 29 09:38:24 pstn1 kernel: EIP: 0060:[<c046e7fe>] CPU: 0
>> Mar 29 09:38:24 pstn1 kernel: EIP is at kfree+0x68/0x6c
>> Mar 29 09:38:24 pstn1 kernel:  EFLAGS: 00000286    Tainted: GF
>> (2.6.18-128.1.10.el5 #1)
>> Mar 29 09:38:24 pstn1 kernel: EAX: 00000029 EBX: f7ff9380 ECX:
>> f7fff880 EDX: c11ff9a0
>> Mar 29 09:38:24 pstn1 kernel: ESI: 00000286 EDI: cffcda00 EBP:
>> e5e10c80 DS: 007b ES: 007b
>> Mar 29 09:38:24 pstn1 kernel: CR0: 80050033 CR2: b7ce39e0 CR3:
>> 0f911000 CR4: 000006d0
>> Mar 29 09:38:24 pstn1 kernel:  [<c05b067c>] kfree_skbmem+0x8/0x61
>> Mar 29 09:38:24 pstn1 kernel:  [<c05e9aaf>] __udp_queue_rcv_skb+0x4a/0x51
>> Mar 29 09:38:24 pstn1 kernel:  [<c05ad993>] release_sock+0x44/0x91
>> Mar 29 09:38:24 pstn1 kernel:  [<c05ea939>] udp_sendmsg+0x44e/0x514
>> Mar 29 09:38:24 pstn1 kernel:  [<c05efdec>] inet_sendmsg+0x35/0x3f
>> Mar 29 09:38:24 pstn1 kernel:  [<c05ab30c>] sock_sendmsg+0xce/0xe8
>> Mar 29 09:38:24 pstn1 kernel:  [<c043464f>]
>> autoremove_wake_function+0x0/0x2d
>> Mar 29 09:38:24 pstn1 kernel:  [<c04ea17b>] copy_from_user+0x17/0x5d
>> Mar 29 09:38:24 pstn1 kernel:  [<c04ea3a1>] copy_to_user+0x31/0x48
>> Mar 29 09:38:24 pstn1 kernel:  [<f89ab141>] zt_chan_read+0x1e0/0x20b
>> [zaptel]
>> Mar 29 09:38:24 pstn1 kernel:  [<c04ea195>] copy_from_user+0x31/0x5d
>> Mar 29 09:38:24 pstn1 kernel:  [<c05ac4c4>] sys_sendto+0x116/0x140
>> Mar 29 09:38:24 pstn1 kernel:  [<c0415d4f>] flush_tlb_page+0x74/0x77
>> Mar 29 09:38:24 pstn1 kernel:  [<c0461331>] do_wp_page+0x3bf/0x40a
>> Mar 29 09:38:24 pstn1 kernel:  [<c04284f1>] current_fs_time+0x4a/0x55
>> Mar 29 09:38:24 pstn1 kernel:  [<c0488f9b>] touch_atime+0x60/0x91
>> Mar 29 09:38:24 pstn1 kernel:  [<c047d9d0>] pipe_readv+0x315/0x321
>> Mar 29 09:38:24 pstn1 kernel:  [<c05acde4>] sys_socketcall+0x106/0x19e
>> Mar 29 09:38:24 pstn1 kernel:  [<c0404f17>] syscall_call+0x7/0xb
>> Mar 29 09:38:24 pstn1 kernel:  =======================
>>
>>
>> This occurred during a "high load" period (52 calls across 3 PRI spans).
>>
>> A couple days ago I moved the interrupts for my PRI card to CPU0 from
>> CPU3, because CPU3 was handling everything else:
>>           CPU0       CPU1       CPU2       CPU3
>>  0:        306          0          0 3684057379    IO-APIC-edge  timer
>>  1:          0          0          0      13468    IO-APIC-edge  i8042
>>  8:          0          0          0          3    IO-APIC-edge  rtc
>>  9:          0          0          0          0   IO-APIC-level  acpi
>>  12:          0          0          0          4    IO-APIC-edge  i8042
>> 169:          0          0          0          0   IO-APIC-level
>>  uhci_hcd:usb2
>> 177:          0          0          0   18392593   IO-APIC-level  ata_piix
>> 185:          0          0          0          1   IO-APIC-level
>>  ehci_hcd:usb1
>> 193:          0          0          0          0   IO-APIC-level
>>  uhci_hcd:usb3
>> 201:          0          0          0 2090021759   IO-APIC-level  eth0
>> 209:  149621223          0          0 3534419461   IO-APIC-level  wct4xxp
>>
>>
>> (The CPU3 number for wct4xxp is not increasing any more).
>>
>> What is the interrupt distribution of other people's systems?
>> Before I made this change I was having a problem with D-channels
>> dropping occasionally, so I thought it might be an interrupt/load
>> issue.
>>
>> Thank you.
>>
>> -- James
>>
>> --
>> _____________________________________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>> New to Asterisk? Join us for a live introductory webinar every Thurs:
>>               http://www.asterisk.org/hello
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>               http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>