[asterisk-dev] [Code Review] Avoid wasteful frame queue traversals in chan_iax2 network_thread()

[Russell Bryant]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://reviewboard.digium.com/r/212/
-----------------------------------------------------------

(Updated 2009-03-28 23:57:53.607625)


Review request for Asterisk Developers.


Summary
-------

While doing some profiling for <http://reviewboard.digium.com/r/205/>, one function that caught my eye was network_thread() in chan_iax2.c.  After the things that I was working on there, it was the next target for analysis and optimization.  I used oprofile's source annotation functionality and found that the loop traversing the frame queue in network_thread() was to blame for the excessive CPU cycle consumption.

The frame_queue in chan_iax2 previously held all frames that either were pending transmission or had been transmitted and are still pending acknowledgment.

In network_thread(), the previous code would go back through the main for loop after reading a single incoming frame or after being signaled because a frame had been queued up for initial transmission.  In each iteration of the loop, it traverses the entire frame queue looking for frames that need to be transmitted.  On a busy server, this could easily be quite a few entries.

This patch is actually quite simple.  The frame_queue has become only a list of frames pending acknowledgment.  Frames that need to be transmitted are queued up to a dedicated transmit thread via the taskprocessor API.

As a result, the code in network_thread() becomes much simpler, as its only job is to read incoming frames.


Diffs (updated)
-----

  /trunk/channels/chan_iax2.c 184835 

Diff: http://reviewboard.digium.com/r/212/diff


Testing
-------

I used a similar test that I used in R205 for this.  I sent 15000 IAX2 registrations evenly spaced out over 1 minute.  The server was configured using realtime with a MySQL backend via res_config_odbc.  IAX2 realtime caching was enabled.  I ran the test, used oprofile, and watched CPU usage.

The box has an Intel Core 2 Duo @ 2.33 GHz and 2 GB of RAM.  The noted CPU consumption is out of 200%, since it's a dual core machine.

----------------------------

Asterisk SVN-trunk-r184801 built by russell @ milo on a x86_64 running Linux on 2009-02-12 19:26:12 UTC

Average CPU usage by Asterisk: 10 to 15%

(results below 0.5% omitted)

samples  %        image name               app name                 symbol name
33690    39.8500  chan_iax2.so             chan_iax2.so             network_thread
13263    15.6881  chan_iax2.so             chan_iax2.so             socket_process
6972      8.2468  chan_iax2.so             chan_iax2.so             __attempt_transmit
6618      7.8281  chan_iax2.so             chan_iax2.so             pvt_destructor
6542      7.7382  chan_iax2.so             chan_iax2.so             __find_callno
2037      2.4095  asterisk                 asterisk                 internal_ao2_callback
620       0.7334  asterisk                 asterisk                 sched_time_cmp
602       0.7121  asterisk                 asterisk                 handle_statechange
563       0.6659  [vdso] (tgid:7462 range:0x7fffe31fe000-0x7fffe3200000) asterisk                 (no symbols)
555       0.6565  chan_iax2.so             chan_iax2.so             peer_cmp_cb
518       0.6127  asterisk                 asterisk                 internal_ao2_ref
...

----------------------------

Asterisk SVN-russell-iax2_transmit_q-r184836-/trunk built by root @ milo on a x86_64 running Linux on 2009-03-29 03:43:13 UTC

Average CPU usage by Asterisk: 5 to 10%

(most results below 0.5% omitted)

samples  %        image name               app name                 symbol name
9290     24.5969  chan_iax2.so             chan_iax2.so             socket_process
6553     17.3502  chan_iax2.so             chan_iax2.so             __find_callno
3630      9.6111  chan_iax2.so             chan_iax2.so             pvt_destructor
2392      6.3332  chan_iax2.so             chan_iax2.so             __attempt_transmit
1946      5.1524  asterisk                 asterisk                 internal_ao2_callback
582       1.5409  asterisk                 asterisk                 handle_statechange
557       1.4748  chan_iax2.so             chan_iax2.so             peer_cmp_cb
528       1.3980  asterisk                 asterisk                 internal_ao2_ref
523       1.3847  [vdso] (tgid:3088 range:0x7fff3c3fe000-0x7fff3c400000) asterisk                 (no symbols)
478       1.2656  asterisk                 asterisk                 sched_time_cmp
416       1.1014  res_config_odbc.so       res_config_odbc.so       realtime_odbc
348       0.9214  chan_iax2.so             chan_iax2.so             register_verify
339       0.8976  chan_iax2.so             chan_iax2.so             build_peer
320       0.8473  chan_iax2.so             chan_iax2.so             .plt
291       0.7705  asterisk                 asterisk                 __bt_defcmp
236       0.6249  asterisk                 asterisk                 _ast_heap_remove
233       0.6169  asterisk                 asterisk                 .plt
221       0.5851  asterisk                 asterisk                 __ao2_iterator_next
216       0.5719  asterisk                 asterisk                 tps_processing_function
204       0.5401  asterisk                 asterisk                 ast_get_channel_tech
193       0.5110  chan_iax2.so             chan_iax2.so             iax2_process_thread
190       0.5031  asterisk                 asterisk                 ast_hashtab_remove_this_object
...
56        0.1483  chan_iax2.so             chan_iax2.so             transmit_frame
...
36        0.0953  chan_iax2.so             chan_iax2.so             network_thread
...

----------------------------


Thanks,

Russell