[asterisk-users] How are your PRI interrupts balanced? (+ Soft lockup BUG)

[James Lamanna]

Hi,
I'm trying to figure out the cause of a soft lockup I experienced:

Mar 29 09:38:24 pstn1 kernel: BUG: soft lockup - CPU#0 stuck for 10s!
[asterisk:32029]
Mar 29 09:38:24 pstn1 kernel: Pid: 32029, comm:             asterisk
Mar 29 09:38:24 pstn1 kernel: EIP: 0060:[<c046e7fe>] CPU: 0
Mar 29 09:38:24 pstn1 kernel: EIP is at kfree+0x68/0x6c
Mar 29 09:38:24 pstn1 kernel:  EFLAGS: 00000286    Tainted: GF
(2.6.18-128.1.10.el5 #1)
Mar 29 09:38:24 pstn1 kernel: EAX: 00000029 EBX: f7ff9380 ECX:
f7fff880 EDX: c11ff9a0
Mar 29 09:38:24 pstn1 kernel: ESI: 00000286 EDI: cffcda00 EBP:
e5e10c80 DS: 007b ES: 007b
Mar 29 09:38:24 pstn1 kernel: CR0: 80050033 CR2: b7ce39e0 CR3:
0f911000 CR4: 000006d0
Mar 29 09:38:24 pstn1 kernel:  [<c05b067c>] kfree_skbmem+0x8/0x61
Mar 29 09:38:24 pstn1 kernel:  [<c05e9aaf>] __udp_queue_rcv_skb+0x4a/0x51
Mar 29 09:38:24 pstn1 kernel:  [<c05ad993>] release_sock+0x44/0x91
Mar 29 09:38:24 pstn1 kernel:  [<c05ea939>] udp_sendmsg+0x44e/0x514
Mar 29 09:38:24 pstn1 kernel:  [<c05efdec>] inet_sendmsg+0x35/0x3f
Mar 29 09:38:24 pstn1 kernel:  [<c05ab30c>] sock_sendmsg+0xce/0xe8
Mar 29 09:38:24 pstn1 kernel:  [<c043464f>] autoremove_wake_function+0x0/0x2d
Mar 29 09:38:24 pstn1 kernel:  [<c04ea17b>] copy_from_user+0x17/0x5d
Mar 29 09:38:24 pstn1 kernel:  [<c04ea3a1>] copy_to_user+0x31/0x48
Mar 29 09:38:24 pstn1 kernel:  [<f89ab141>] zt_chan_read+0x1e0/0x20b [zaptel]
Mar 29 09:38:24 pstn1 kernel:  [<c04ea195>] copy_from_user+0x31/0x5d
Mar 29 09:38:24 pstn1 kernel:  [<c05ac4c4>] sys_sendto+0x116/0x140
Mar 29 09:38:24 pstn1 kernel:  [<c0415d4f>] flush_tlb_page+0x74/0x77
Mar 29 09:38:24 pstn1 kernel:  [<c0461331>] do_wp_page+0x3bf/0x40a
Mar 29 09:38:24 pstn1 kernel:  [<c04284f1>] current_fs_time+0x4a/0x55
Mar 29 09:38:24 pstn1 kernel:  [<c0488f9b>] touch_atime+0x60/0x91
Mar 29 09:38:24 pstn1 kernel:  [<c047d9d0>] pipe_readv+0x315/0x321
Mar 29 09:38:24 pstn1 kernel:  [<c05acde4>] sys_socketcall+0x106/0x19e
Mar 29 09:38:24 pstn1 kernel:  [<c0404f17>] syscall_call+0x7/0xb
Mar 29 09:38:24 pstn1 kernel:  =======================


This occurred during a "high load" period (52 calls across 3 PRI spans).

A couple days ago I moved the interrupts for my PRI card to CPU0 from
CPU3, because CPU3 was handling everything else:
           CPU0       CPU1       CPU2       CPU3
  0:        306          0          0 3684057379    IO-APIC-edge  timer
  1:          0          0          0      13468    IO-APIC-edge  i8042
  8:          0          0          0          3    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:          0          0          0          4    IO-APIC-edge  i8042
169:          0          0          0          0   IO-APIC-level  uhci_hcd:usb2
177:          0          0          0   18392593   IO-APIC-level  ata_piix
185:          0          0          0          1   IO-APIC-level  ehci_hcd:usb1
193:          0          0          0          0   IO-APIC-level  uhci_hcd:usb3
201:          0          0          0 2090021759   IO-APIC-level  eth0
209:  149621223          0          0 3534419461   IO-APIC-level  wct4xxp


(The CPU3 number for wct4xxp is not increasing any more).

What is the interrupt distribution of other people's systems?
Before I made this change I was having a problem with D-channels
dropping occasionally, so I thought it might be an interrupt/load
issue.

Thank you.

-- James