[asterisk-dev] rtp scalability improvement...

Mon Mar 20 04:40:04 MST 2006

> On Mon, Mar 20, 2006 at 04:03:26PM +0600, Paul Cadach wrote:
>> Hello,
>>
>> Roy Sigurd Karlsbakk wrote:
>> [skipped]
>>> These IRQ storms are only happening on crappy network hardware. My
>>> testing was one with intel gigabit NICs with large buffers,
>>> effectively producing < 100 interrupts per second. kernel profiling
>>> showed time was indeed spent in system calls
>>
>> The one solution I see is to move RTP _switching_ into the kernel  
>> level (like
>> zaptel conferencing doing), and optimize Asterisk core to do  
>> better codec
>> selection logic (i.e. to always minimize/eliminate transcoding  
>> when it is
>> possible).
>
> Transcoding is expensive but has no relation to system calls load.
> I went to look what syscall rates are sustainable by a modern box and
> a random result from google says this:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=103942289025149&w=2
>
> so it's over 200k syscalls per second. I doubt the 40k pps may
> account for more than 10-20% of the total system time spent,
> so the suggested idea of mmapping packets won't help.
>
> There are many other expensive things e.g. i would look into
> lock/unlock calls (depending on which thread package you have,
> those can be very expensive).
> Hard to tell without an indication on how many syscalls per second
> the original poster (Roy ?) was seeing, how many pps, and which
> type of syscalls were involved.

oprofiling kernel with some ~40 concurrent calls now shows, first, to  
summarize, these spend most time

   1829721 59.1120 vmlinux-2.6.15.4
    401787 12.9804 asterisk
    275705  8.9071 libc-2.3.2.so
    251861  8.1368 chan_sip.so
    110671  3.5754 libpthread-0.60.so
    101430  3.2769 libperl.so.5.8.4
     33307  1.0760 libstdc++.so.5.0.7
     26450  0.8545 oprofiled
     24355  0.7868 opreport
     11923  0.3852 ld-2.3.2.so
      7814  0.2524 libcrypto.so.0.9.7
      4342  0.1403 syslogd
      3439  0.1111 sshd

then these parts of kernel is called. this doesn't tell me as much as  
it should, as i lack some kernel knowledge. also note that the amount  
of interrupts are not tuned on this box, standard broadcom gigE...

roy

118436    3.8263  vmlinux-2.6.15.4         __sched_text_start
83899     2.7105  vmlinux-2.6.15.4         fget
58723     1.8972  vmlinux-2.6.15.4         try_to_wake_up
55498     1.7930  vmlinux-2.6.15.4         __switch_to
54774     1.7696  vmlinux-2.6.15.4         pipe_poll
51320     1.6580  vmlinux-2.6.15.4         mwait_idle
49087     1.5859  vmlinux-2.6.15.4         datagram_poll
42895     1.3858  vmlinux-2.6.15.4         sys_poll
42263     1.3654  vmlinux-2.6.15.4         sock_poll
41713     1.3476  vmlinux-2.6.15.4         tg3_poll
36751     1.1873  vmlinux-2.6.15.4         fput
34627     1.1187  vmlinux-2.6.15.4         uhci_irq
33243     1.0740  vmlinux-2.6.15.4         copy_user_generic
30885     0.9978  vmlinux-2.6.15.4         copy_page_range
25460     0.8225  vmlinux-2.6.15.4         IRQ0xa9_interrupt
23639     0.7637  vmlinux-2.6.15.4         thread_return
22922     0.7405  vmlinux-2.6.15.4         system_call
22332     0.7215  vmlinux-2.6.15.4         tg3_interrupt_tagged
20380     0.6584  vmlinux-2.6.15.4         find_busiest_group
19232     0.6213  vmlinux-2.6.15.4         kfree
18331     0.5922  vmlinux-2.6.15.4         memcpy
18039     0.5828  vmlinux-2.6.15.4         copy_page
15999     0.5169  vmlinux-2.6.15.4         unmap_vmas
15384     0.4970  vmlinux-2.6.15.4         udp_v4_lookup_longway
15334     0.4954  vmlinux-2.6.15.4         __kmalloc
14597     0.4716  vmlinux-2.6.15.4         __ip_route_output_key
14228     0.4597  vmlinux-2.6.15.4         udp_rcv
13733     0.4437  vmlinux-2.6.15.4         poll_freewait
13475     0.4353  vmlinux-2.6.15.4         math_state_restore
13362     0.4317  vmlinux-2.6.15.4         udp_sendmsg
13164     0.4253  vmlinux-2.6.15.4         ip_route_input
12915     0.4172  vmlinux-2.6.15.4         sock_def_readable
12779     0.4129  vmlinux-2.6.15.4         __pollwait
12481     0.4032  vmlinux-2.6.15.4         __kfree_skb
12248     0.3957  vmlinux-2.6.15.4         add_wait_queue
12089     0.3906  vmlinux-2.6.15.4         get_page_from_freelist
11985     0.3872  vmlinux-2.6.15.4         device_not_available
11789     0.3809  vmlinux-2.6.15.4         __do_IRQ
11637     0.3760  vmlinux-2.6.15.4         resched_task
11623     0.3755  vmlinux-2.6.15.4         remove_wait_queue
11223     0.3626  vmlinux-2.6.15.4         cpu_idle
11089     0.3583  vmlinux-2.6.15.4         __wake_up_common
10942     0.3535  vmlinux-2.6.15.4         sync_buffer
10704     0.3458  vmlinux-2.6.15.4         dev_queue_xmit
10533     0.3403  vmlinux-2.6.15.4         find_vma
9919      0.3205  vmlinux-2.6.15.4         ip_append_data
9688      0.3130  vmlinux-2.6.15.4         sysret_check
9473      0.3060  vmlinux-2.6.15.4         sock_def_write_space
9205      0.2974  vmlinux-2.6.15.4         kmem_cache_alloc
9188      0.2968  vmlinux-2.6.15.4         ip_output
9037      0.2920  vmlinux-2.6.15.4         ip_push_pending_frames
8783      0.2838  vmlinux-2.6.15.4         __kprobes_text_start
8753      0.2828  vmlinux-2.6.15.4         flush_tlb_others
8715      0.2816  vmlinux-2.6.15.4         sched_clock
8608      0.2781  vmlinux-2.6.15.4         qdisc_restart
8471      0.2737  vmlinux-2.6.15.4         page_remove_rmap
8421      0.2721  vmlinux-2.6.15.4         pipe_readv
8374      0.2705  vmlinux-2.6.15.4         recalc_sigpending_tsk
8359      0.2701  vmlinux-2.6.15.4         tg3_start_xmit
8352      0.2698  vmlinux-2.6.15.4         timer_interrupt
...

--
Roy Sigurd Karlsbakk
roy at karlsbakk.net
---
In space, loud sounds, like explosions, are even louder because there  
is no air to get in the way.