[Asterisk-Users] Dual T400P, SMP, performance issues

Alex Zarubin ZAlex at Webley.COM
Wed Jun 25 14:07:46 MST 2003 

Mark, here is the info you requested. As far as multiple T400P boards
question, I believe this is the most probable reason for this behavior
(we haven't seen it on a single board machines). But in order to
prove it we need 4-5 days of load testing. Hopefully we'll be able
to do it next week.

ksymoops 2.4.4 on i686 2.4.21.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.21 (specified)
     -m /boot/System.map-2.4.21 (default)
     -i

Jun 24 18:23:25 mspgate03 kernel: wait_on_irq, CPU 1:
Jun 24 18:23:25 mspgate03 kernel: irq:  1 [ 0 0 1 0 ]
Jun 24 18:23:25 mspgate03 kernel: bh:   0 [ 0 0 0 0 ]
Jun 24 18:23:25 mspgate03 kernel: Stack dumps:
Jun 24 18:23:25 mspgate03 kernel: CPU 0:02000000 0000036f 00e14603 18020000
03000010 00006647 008e0200 48030000
Jun 24 18:23:25 mspgate03 kernel:        00000078 001ffa02 5b490300 06000000
000001c7 074e0308 00001afe 01c74d03
Jun 24 18:23:25 mspgate03 kernel:        23020000 d7080000 e1000001 09000000
000001d7 f5030001 04000023 09300207
Jun 24 18:23:25 mspgate03 kernel: Call Trace:    [<f89bd281>] [<f89bb132>]
[<f89bbb47>] [<f89bd281>] [<f89bd281>]
Jun 24 18:23:25 mspgate03 kernel:   [<f89bb132>] [<f89bd281>] [<f89bd281>]
[<f89bb132>] [<f89bbb47>] [<f89e7737>]
Jun 24 18:23:25 mspgate03 kernel:   [<f89aa80a>] [<f89aa80a>] [<c01feee4>]
[<f89e7737>] [<c01f4eae>] [<c010a98e>]
Jun 24 18:23:25 mspgate03 kernel:   [<c020d122>] [<c010abe3>] [<c020d122>]
[<c020d550>] [<c010a98e>] [<c020d550>]
Jun 24 18:23:25 mspgate03 kernel:   [<c010abfe>] [<c01f0919>] [<c01f0919>]
[<c022a1ef>] [<c022a1ef>] [<c022a5f5>]
Jun 24 18:23:25 mspgate03 kernel:   [<f89bd281>] [<f89bd281>] [<f89bd281>]
[<f89bb132>] [<f89bd510>] [<f89e7737>]
Jun 24 18:23:25 mspgate03 kernel:   [<c022a5f5>] [<c01f0ffd>] [<c01f112e>]
[<c01f53c2>] [<c012005b>] [<c010abfe>]
Jun 24 18:23:25 mspgate03 kernel:   [<c015147a>] [<c01509dc>] [<c0147460>]
[<c0147fb8>] [<f89e7737>] [<f89e7737>]
Jun 24 18:23:25 mspgate03 kernel:   [<c01f0998>] [<c01f0fac>] [<c01f112e>]
[<c01f53c2>] [<c0117fce>] [<c0117ef0>]
Jun 24 18:23:25 mspgate03 kernel:   [<c0144a64>] [<c01246db>] [<c0109023>]
Jun 24 18:23:25 mspgate03 kernel: CPU 2:00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
Jun 24 18:23:25 mspgate03 kernel:        00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
Jun 24 18:23:25 mspgate03 kernel:        00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
Jun 24 18:23:25 mspgate03 kernel: CPU 3:00000070 cce30002 0cd80000 08fa0000
69530000 656c706d 6c616e41 73697379
Jun 24 18:23:25 mspgate03 kernel:        0009a700 46534c00 65746e69 6c6f7072
32657461 6e655f61 0a810063 69530000
Jun 24 18:23:25 mspgate03 kernel:        656c706d 65746e49 6c6f7072 4c657461
39004653 5300000b 6c706d69 66736c65
Jun 24 18:23:25 mspgate03 kernel: CPU 1:e14d5eac c025c896 00000001 00000001
ffffffff 00000001 c010a7c2 c025c8ab
Jun 24 18:23:25 mspgate03 kernel:        00000000 f2d92124 e14d5f00 c0191104
00000500 00001805 000000bf 00008a01
Jun 24 18:23:25 mspgate03 kernel:        7f1c0300 01000415 1a131100 170f1200
00000000 e14d4000 00000000 00000000
Jun 24 18:23:25 mspgate03 kernel: Call Trace:    [<c010a7c2>] [<c0191104>]
[<c01913d4>] [<c018e1e2>] [<c014c2c7>]
Jun 24 18:23:25 mspgate03 kernel:   [<c0109023>]
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910>
Trace; f89bbb47 <[zaptel]zt_getbuf_chunk+1f7/4b0>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910>
Trace; f89bbb47 <[zaptel]zt_getbuf_chunk+1f7/4b0>
Trace; f89e7737 <[tor2]tor2_intr+847/cb0>
Trace; f89aa80a <[eepro100]speedo_start_xmit+17a/210>
Trace; f89aa80a <[eepro100]speedo_start_xmit+17a/210>
Trace; c01feee4 <qdisc_restart+14/170>
Trace; f89e7737 <[tor2]tor2_intr+847/cb0>
Trace; c01f4eae <dev_queue_xmit+14e/320>
Trace; c010a98e <handle_IRQ_event+5e/90>
Trace; c020d122 <ip_output+102/170>
Trace; c010abe3 <do_IRQ+e3/110>
Trace; c020d122 <ip_output+102/170>
Trace; c020d550 <ip_queue_xmit+3c0/520>
Trace; c010a98e <handle_IRQ_event+5e/90>
Trace; c020d550 <ip_queue_xmit+3c0/520>
Trace; c010abfe <do_IRQ+fe/110>
Trace; c01f0919 <sock_def_readable+39/70>
Trace; c01f0919 <sock_def_readable+39/70>
Trace; c022a1ef <udp_queue_rcv_skb+18f/200>
Trace; c022a1ef <udp_queue_rcv_skb+18f/200>
Trace; c022a5f5 <udp_rcv+165/340>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70>
Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910>
Trace; f89bd510 <[zaptel]zt_putbuf_chunk+c0/730>
Trace; f89e7737 <[tor2]tor2_intr+847/cb0>
Trace; c022a5f5 <udp_rcv+165/340>
Trace; c01f0ffd <kfree_skbmem+5d/70>
Trace; c01f112e <__kfree_skb+11e/130>
Trace; c01f53c2 <net_tx_action+62/140>
Trace; c012005b <do_softirq+6b/d0>
Trace; c010abfe <do_IRQ+fe/110>
Trace; c015147a <d_lookup+ba/120>
Trace; c01509dc <dput+1c/160>
Trace; c0147460 <cached_lookup+10/50>
Trace; c0147fb8 <link_path_walk+8f8/a10>
Trace; f89e7737 <[tor2]tor2_intr+847/cb0>
Trace; f89e7737 <[tor2]tor2_intr+847/cb0>
Trace; c01f0998 <sock_def_write_space+48/a0>
Trace; c01f0fac <kfree_skbmem+c/70>
Trace; c01f112e <__kfree_skb+11e/130>
Trace; c01f53c2 <net_tx_action+62/140>
Trace; c0117fce <schedule_timeout+7e/a0>
Trace; c0117ef0 <process_timeout+0/60>
Trace; c0144a64 <sys_stat64+64/70>
Trace; c01246db <sys_nanosleep+11b/18c>
Trace; c0109023 <system_call+33/38>
Trace; c010a7c2 <__global_cli+e2/170>
Trace; c0191104 <change_termios+24/190>
Trace; c01913d4 <set_termios+164/170>
Trace; c018e1e2 <tty_ioctl+372/390>
Trace; c014c2c7 <sys_ioctl+1c7/1fe>
Trace; c0109023 <system_call+33/38>


1 warning issued.  Results may not be reliable.

-----Original Message-----
From: Mark Spencer [mailto:markster at digium.com]
Sent: Wednesday, June 25, 2003 11:11 AM
To: 'asterisk-users at lists.digium.com'
Subject: RE: [Asterisk-Users] Dual T400P, SMP, performance issues


Oooh, how neat!  I wonder if there is some sort of race and that the
kernel is detecting and defeating it somehow.  Will ksymoops on your
machine handle that output?  Maybe we can track it down!

Again, does the problem occur with only one board?  i.e. is the problem
tied to having multiple boards in the machine?

Mark

On Tue, 24 Jun 2003, Alex Zarubin wrote:

> Mark & Oliver,
>
> It is too early to say, but the picture is different now. Our dual CPU,
> dual T400P box is up for 4 days, under the load of 10 - 100 simultaneous
> PRI -> SIP calls. We installed 2.4.21 #2 SMP (it was still freezing after
> that) and, what I think made the difference, recompiled
> zaptel-libpri-asterisk
> with gcc 3.3.
>
> The problem, on the way, was that asterisk wouldn't start after that. It
was
> crashing while loading mp3 and lpc10 codecs. We put 'noload' for these two
> into modules.conf - temporary solution, of course.
>
> There are problems, still, with multiple connections at the same time.
> Windows
> to the box get frozen for a sec, D-channel error messages. The following
> messages are dumped into /var/log/messages. What do you think?
>
> ...
>
> Thank you.
> Alex Zarubin
>
> -----Original Message-----
> From: The Traveller [mailto:traveler at xs4all.nl]
> Sent: Tuesday, June 17, 2003 3:10 PM
> To: asterisk-users at lists.digium.com
> Subject: Re: [Asterisk-Users] Dual T400P, SMP, performance issues
>
>
> On Tue, Jun 17, 2003 at 20:54:39 +0200, The Traveller wrote:
> >
> > BTW: As I reported in my previous mail to the list, I've now installed
> kernel
> > 2.4.21-rc2 with ACPI-patch on the box with the E100P.  I've been trying
> > very hard to reproduce a freeze with this kernel, but haven't succeeded
> yet.
> [...]
>
> Ok, it crashed again, so that wasn't it either.  What I did to trigger
> it was using the auto-dialer to loop as many calls to app_datetime out
> and then back over the same E-1 as it would take, queueing the calls
> to "/var/spool/asterisk/outgoing/" 14 at a time.  It froze at the first
> attempt.  The "good" news is that it produced a visible kernel-panic.
> this time.  My guess is that you only don't see it if the console
> screensaver has already come on while it happens.
>
> It read something like "Unable to handle kernel paging request" and
> happened in the swapper-task.  As usual, it dumped a lot of numbers on the
> screen, which I didn't want to write down.
>
> Mark: If you want my help in debugging this, I'll hook it up to a
> serial console, trigger the crash and provide you with the exact
> panic, together with the ksyms and modules-info to trace it.
>
>
>
>     Grtz,
>
>        Oliver
> _______________________________________________
> Asterisk-Users mailing list
> Asterisk-Users at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
>

_______________________________________________
Asterisk-Users mailing list
Asterisk-Users at lists.digium.com
http://lists.digium.com/mailman/listinfo/asterisk-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.digium.com/pipermail/asterisk-users/attachments/20030625/aaa55eaf/attachment.htm

More information about the asterisk-users mailing list