<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [Asterisk-Users] Dual T400P, SMP, performance issues</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Mark, here is the info you requested. As far as multiple T400P boards</FONT>
<BR><FONT SIZE=2>question, I believe this is the most probable reason for this behavior</FONT>
<BR><FONT SIZE=2>(we haven't seen it on a single board machines). But in order to</FONT>
<BR><FONT SIZE=2>prove it we need 4-5 days of load testing. Hopefully we'll be able</FONT>
<BR><FONT SIZE=2>to do it next week.</FONT>
</P>
<P><FONT SIZE=2>ksymoops 2.4.4 on i686 2.4.21. Options used</FONT>
<BR><FONT SIZE=2> -V (default)</FONT>
<BR><FONT SIZE=2> -k /proc/ksyms (default)</FONT>
<BR><FONT SIZE=2> -l /proc/modules (default)</FONT>
<BR><FONT SIZE=2> -o /lib/modules/2.4.21 (specified)</FONT>
<BR><FONT SIZE=2> -m /boot/System.map-2.4.21 (default)</FONT>
<BR><FONT SIZE=2> -i</FONT>
</P>
<P><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: wait_on_irq, CPU 1:</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: irq: 1 [ 0 0 1 0 ]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: bh: 0 [ 0 0 0 0 ]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: Stack dumps:</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: CPU 0:02000000 0000036f 00e14603 18020000 03000010 00006647 008e0200 48030000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 00000078 001ffa02 5b490300 06000000 000001c7 074e0308 00001afe 01c74d03</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 23020000 d7080000 e1000001 09000000 000001d7 f5030001 04000023 09300207</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: Call Trace: [<f89bd281>] [<f89bb132>] [<f89bbb47>] [<f89bd281>] [<f89bd281>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<f89bb132>] [<f89bd281>] [<f89bd281>] [<f89bb132>] [<f89bbb47>] [<f89e7737>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<f89aa80a>] [<f89aa80a>] [<c01feee4>] [<f89e7737>] [<c01f4eae>] [<c010a98e>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c020d122>] [<c010abe3>] [<c020d122>] [<c020d550>] [<c010a98e>] [<c020d550>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c010abfe>] [<c01f0919>] [<c01f0919>] [<c022a1ef>] [<c022a1ef>] [<c022a5f5>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<f89bd281>] [<f89bd281>] [<f89bd281>] [<f89bb132>] [<f89bd510>] [<f89e7737>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c022a5f5>] [<c01f0ffd>] [<c01f112e>] [<c01f53c2>] [<c012005b>] [<c010abfe>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c015147a>] [<c01509dc>] [<c0147460>] [<c0147fb8>] [<f89e7737>] [<f89e7737>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c01f0998>] [<c01f0fac>] [<c01f112e>] [<c01f53c2>] [<c0117fce>] [<c0117ef0>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c0144a64>] [<c01246db>] [<c0109023>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: CPU 2:00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: CPU 3:00000070 cce30002 0cd80000 08fa0000 69530000 656c706d 6c616e41 73697379</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 0009a700 46534c00 65746e69 6c6f7072 32657461 6e655f61 0a810063 69530000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 656c706d 65746e49 6c6f7072 4c657461 39004653 5300000b 6c706d69 66736c65</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: CPU 1:e14d5eac c025c896 00000001 00000001 ffffffff 00000001 c010a7c2 c025c8ab</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 00000000 f2d92124 e14d5f00 c0191104 00000500 00001805 000000bf 00008a01</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: 7f1c0300 01000415 1a131100 170f1200 00000000 e14d4000 00000000 00000000</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: Call Trace: [<c010a7c2>] [<c0191104>] [<c01913d4>] [<c018e1e2>] [<c014c2c7>]</FONT>
<BR><FONT SIZE=2>Jun 24 18:23:25 mspgate03 kernel: [<c0109023>]</FONT>
<BR><FONT SIZE=2>Warning (Oops_read): Code line not seen, dumping what data is available</FONT>
</P>
<P><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910></FONT>
<BR><FONT SIZE=2>Trace; f89bbb47 <[zaptel]zt_getbuf_chunk+1f7/4b0></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910></FONT>
<BR><FONT SIZE=2>Trace; f89bbb47 <[zaptel]zt_getbuf_chunk+1f7/4b0></FONT>
<BR><FONT SIZE=2>Trace; f89e7737 <[tor2]tor2_intr+847/cb0></FONT>
<BR><FONT SIZE=2>Trace; f89aa80a <[eepro100]speedo_start_xmit+17a/210></FONT>
<BR><FONT SIZE=2>Trace; f89aa80a <[eepro100]speedo_start_xmit+17a/210></FONT>
<BR><FONT SIZE=2>Trace; c01feee4 <qdisc_restart+14/170></FONT>
<BR><FONT SIZE=2>Trace; f89e7737 <[tor2]tor2_intr+847/cb0></FONT>
<BR><FONT SIZE=2>Trace; c01f4eae <dev_queue_xmit+14e/320></FONT>
<BR><FONT SIZE=2>Trace; c010a98e <handle_IRQ_event+5e/90></FONT>
<BR><FONT SIZE=2>Trace; c020d122 <ip_output+102/170></FONT>
<BR><FONT SIZE=2>Trace; c010abe3 <do_IRQ+e3/110></FONT>
<BR><FONT SIZE=2>Trace; c020d122 <ip_output+102/170></FONT>
<BR><FONT SIZE=2>Trace; c020d550 <ip_queue_xmit+3c0/520></FONT>
<BR><FONT SIZE=2>Trace; c010a98e <handle_IRQ_event+5e/90></FONT>
<BR><FONT SIZE=2>Trace; c020d550 <ip_queue_xmit+3c0/520></FONT>
<BR><FONT SIZE=2>Trace; c010abfe <do_IRQ+fe/110></FONT>
<BR><FONT SIZE=2>Trace; c01f0919 <sock_def_readable+39/70></FONT>
<BR><FONT SIZE=2>Trace; c01f0919 <sock_def_readable+39/70></FONT>
<BR><FONT SIZE=2>Trace; c022a1ef <udp_queue_rcv_skb+18f/200></FONT>
<BR><FONT SIZE=2>Trace; c022a1ef <udp_queue_rcv_skb+18f/200></FONT>
<BR><FONT SIZE=2>Trace; c022a5f5 <udp_rcv+165/340></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bd281 <[zaptel]zt_process_putaudio_chunk+9a1/b70></FONT>
<BR><FONT SIZE=2>Trace; f89bb132 <[zaptel]zt_process_getaudio_chunk+f2/910></FONT>
<BR><FONT SIZE=2>Trace; f89bd510 <[zaptel]zt_putbuf_chunk+c0/730></FONT>
<BR><FONT SIZE=2>Trace; f89e7737 <[tor2]tor2_intr+847/cb0></FONT>
<BR><FONT SIZE=2>Trace; c022a5f5 <udp_rcv+165/340></FONT>
<BR><FONT SIZE=2>Trace; c01f0ffd <kfree_skbmem+5d/70></FONT>
<BR><FONT SIZE=2>Trace; c01f112e <__kfree_skb+11e/130></FONT>
<BR><FONT SIZE=2>Trace; c01f53c2 <net_tx_action+62/140></FONT>
<BR><FONT SIZE=2>Trace; c012005b <do_softirq+6b/d0></FONT>
<BR><FONT SIZE=2>Trace; c010abfe <do_IRQ+fe/110></FONT>
<BR><FONT SIZE=2>Trace; c015147a <d_lookup+ba/120></FONT>
<BR><FONT SIZE=2>Trace; c01509dc <dput+1c/160></FONT>
<BR><FONT SIZE=2>Trace; c0147460 <cached_lookup+10/50></FONT>
<BR><FONT SIZE=2>Trace; c0147fb8 <link_path_walk+8f8/a10></FONT>
<BR><FONT SIZE=2>Trace; f89e7737 <[tor2]tor2_intr+847/cb0></FONT>
<BR><FONT SIZE=2>Trace; f89e7737 <[tor2]tor2_intr+847/cb0></FONT>
<BR><FONT SIZE=2>Trace; c01f0998 <sock_def_write_space+48/a0></FONT>
<BR><FONT SIZE=2>Trace; c01f0fac <kfree_skbmem+c/70></FONT>
<BR><FONT SIZE=2>Trace; c01f112e <__kfree_skb+11e/130></FONT>
<BR><FONT SIZE=2>Trace; c01f53c2 <net_tx_action+62/140></FONT>
<BR><FONT SIZE=2>Trace; c0117fce <schedule_timeout+7e/a0></FONT>
<BR><FONT SIZE=2>Trace; c0117ef0 <process_timeout+0/60></FONT>
<BR><FONT SIZE=2>Trace; c0144a64 <sys_stat64+64/70></FONT>
<BR><FONT SIZE=2>Trace; c01246db <sys_nanosleep+11b/18c></FONT>
<BR><FONT SIZE=2>Trace; c0109023 <system_call+33/38></FONT>
<BR><FONT SIZE=2>Trace; c010a7c2 <__global_cli+e2/170></FONT>
<BR><FONT SIZE=2>Trace; c0191104 <change_termios+24/190></FONT>
<BR><FONT SIZE=2>Trace; c01913d4 <set_termios+164/170></FONT>
<BR><FONT SIZE=2>Trace; c018e1e2 <tty_ioctl+372/390></FONT>
<BR><FONT SIZE=2>Trace; c014c2c7 <sys_ioctl+1c7/1fe></FONT>
<BR><FONT SIZE=2>Trace; c0109023 <system_call+33/38></FONT>
</P>
<BR>
<P><FONT SIZE=2>1 warning issued. Results may not be reliable.</FONT>
</P>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Mark Spencer [<A HREF="mailto:markster@digium.com">mailto:markster@digium.com</A>]</FONT>
<BR><FONT SIZE=2>Sent: Wednesday, June 25, 2003 11:11 AM</FONT>
<BR><FONT SIZE=2>To: 'asterisk-users@lists.digium.com'</FONT>
<BR><FONT SIZE=2>Subject: RE: [Asterisk-Users] Dual T400P, SMP, performance issues</FONT>
</P>
<BR>
<P><FONT SIZE=2>Oooh, how neat! I wonder if there is some sort of race and that the</FONT>
<BR><FONT SIZE=2>kernel is detecting and defeating it somehow. Will ksymoops on your</FONT>
<BR><FONT SIZE=2>machine handle that output? Maybe we can track it down!</FONT>
</P>
<P><FONT SIZE=2>Again, does the problem occur with only one board? i.e. is the problem</FONT>
<BR><FONT SIZE=2>tied to having multiple boards in the machine?</FONT>
</P>
<P><FONT SIZE=2>Mark</FONT>
</P>
<P><FONT SIZE=2>On Tue, 24 Jun 2003, Alex Zarubin wrote:</FONT>
</P>
<P><FONT SIZE=2>> Mark & Oliver,</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> It is too early to say, but the picture is different now. Our dual CPU,</FONT>
<BR><FONT SIZE=2>> dual T400P box is up for 4 days, under the load of 10 - 100 simultaneous</FONT>
<BR><FONT SIZE=2>> PRI -> SIP calls. We installed 2.4.21 #2 SMP (it was still freezing after</FONT>
<BR><FONT SIZE=2>> that) and, what I think made the difference, recompiled</FONT>
<BR><FONT SIZE=2>> zaptel-libpri-asterisk</FONT>
<BR><FONT SIZE=2>> with gcc 3.3.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> The problem, on the way, was that asterisk wouldn't start after that. It was</FONT>
<BR><FONT SIZE=2>> crashing while loading mp3 and lpc10 codecs. We put 'noload' for these two</FONT>
<BR><FONT SIZE=2>> into modules.conf - temporary solution, of course.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> There are problems, still, with multiple connections at the same time.</FONT>
<BR><FONT SIZE=2>> Windows</FONT>
<BR><FONT SIZE=2>> to the box get frozen for a sec, D-channel error messages. The following</FONT>
<BR><FONT SIZE=2>> messages are dumped into /var/log/messages. What do you think?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> ...</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Thank you.</FONT>
<BR><FONT SIZE=2>> Alex Zarubin</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: The Traveller [<A HREF="mailto:traveler@xs4all.nl">mailto:traveler@xs4all.nl</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Tuesday, June 17, 2003 3:10 PM</FONT>
<BR><FONT SIZE=2>> To: asterisk-users@lists.digium.com</FONT>
<BR><FONT SIZE=2>> Subject: Re: [Asterisk-Users] Dual T400P, SMP, performance issues</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> On Tue, Jun 17, 2003 at 20:54:39 +0200, The Traveller wrote:</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > BTW: As I reported in my previous mail to the list, I've now installed</FONT>
<BR><FONT SIZE=2>> kernel</FONT>
<BR><FONT SIZE=2>> > 2.4.21-rc2 with ACPI-patch on the box with the E100P. I've been trying</FONT>
<BR><FONT SIZE=2>> > very hard to reproduce a freeze with this kernel, but haven't succeeded</FONT>
<BR><FONT SIZE=2>> yet.</FONT>
<BR><FONT SIZE=2>> [...]</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Ok, it crashed again, so that wasn't it either. What I did to trigger</FONT>
<BR><FONT SIZE=2>> it was using the auto-dialer to loop as many calls to app_datetime out</FONT>
<BR><FONT SIZE=2>> and then back over the same E-1 as it would take, queueing the calls</FONT>
<BR><FONT SIZE=2>> to "/var/spool/asterisk/outgoing/" 14 at a time. It froze at the first</FONT>
<BR><FONT SIZE=2>> attempt. The "good" news is that it produced a visible kernel-panic.</FONT>
<BR><FONT SIZE=2>> this time. My guess is that you only don't see it if the console</FONT>
<BR><FONT SIZE=2>> screensaver has already come on while it happens.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> It read something like "Unable to handle kernel paging request" and</FONT>
<BR><FONT SIZE=2>> happened in the swapper-task. As usual, it dumped a lot of numbers on the</FONT>
<BR><FONT SIZE=2>> screen, which I didn't want to write down.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Mark: If you want my help in debugging this, I'll hook it up to a</FONT>
<BR><FONT SIZE=2>> serial console, trigger the crash and provide you with the exact</FONT>
<BR><FONT SIZE=2>> panic, together with the ksyms and modules-info to trace it.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Grtz,</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Oliver</FONT>
<BR><FONT SIZE=2>> _______________________________________________</FONT>
<BR><FONT SIZE=2>> Asterisk-Users mailing list</FONT>
<BR><FONT SIZE=2>> Asterisk-Users@lists.digium.com</FONT>
<BR><FONT SIZE=2>> <A HREF="http://lists.digium.com/mailman/listinfo/asterisk-users" TARGET="_blank">http://lists.digium.com/mailman/listinfo/asterisk-users</A></FONT>
<BR><FONT SIZE=2>></FONT>
</P>
<P><FONT SIZE=2>_______________________________________________</FONT>
<BR><FONT SIZE=2>Asterisk-Users mailing list</FONT>
<BR><FONT SIZE=2>Asterisk-Users@lists.digium.com</FONT>
<BR><FONT SIZE=2><A HREF="http://lists.digium.com/mailman/listinfo/asterisk-users" TARGET="_blank">http://lists.digium.com/mailman/listinfo/asterisk-users</A></FONT>
</P>
</BODY>
</HTML>