[asterisk-bugs] [JIRA] (ASTERISK-23719) Asterisk locks, UDP buffer overflow, 1000+ spawns of 'chan_iax2.c find_idle_thread()'

[Matt Jordan (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-23719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=217949#comment-217949 ] 

Matt Jordan commented on ASTERISK-23719:
----------------------------------------

The output of {{core show locks}} shows that everything is piled up waiting for this channel lock to be released:

{noformat}
=== Thread ID: 0x7fb6261d9700 (pbx_thread           started at [ 6865] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 3853 __ast_read chan 0x7fb60401f0d0 (1)
	main/logger.c:1599 ast_bt_get_addresses() (0x50a283+1D)
	main/lock.c:218 __ast_pthread_mutex_lock() (0x502454+C9)
	main/astobj2.c:192 __ao2_lock() (0x44c6fa+96)
	main/channel.c:3856 __ast_read()
	main/channel.c:4365 ast_read() (0x47e09a+1D)
	main/channel.c:7576 ast_generic_bridge()
	main/channel.c:8046 ast_channel_bridge() (0x4898a4+19D3)
	main/features.c:4483 ast_bridge_call() (0x4d3ea0+F80)
	apps/app_dial.c:3045 dial_exec_full()
	apps/app_dial.c:3129 dial_exec()
	main/pbx.c:1622 pbx_exec() (0x52bc19+214)
	main/pbx.c:4918 pbx_extension_helper()
	main/pbx.c:6035 ast_spawn_extension() (0x539a1f+65)
	main/pbx.c:6509 __ast_pbx_run()
	main/pbx.c:6840 pbx_thread()
	main/utils.c:1162 dummy_start()
	pthread_create.c:0 start_thread()
	:0 __clone() (0x7fb6982cab00+6D)
{noformat}

There isn't a condition in {{ast_read}} that is causing the lock to be unreleased. This appears to be an issue with {{DEBUG_THREDS}} causing locks to not be released.

Does this issue persist if you disable {{DEBUG_THREADS}}?

> Asterisk locks, UDP buffer overflow, 1000+ spawns of 'chan_iax2.c find_idle_thread()'
> -------------------------------------------------------------------------------------
>
>                 Key: ASTERISK-23719
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-23719
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Channels/chan_iax2
>    Affects Versions: 11.6.1
>         Environment: CentOS 6.4min
>            Reporter: SteelPivot
>            Severity: Critical
>         Attachments: 1399319401-core-show-locks.txt, 1399324201-backtrace-threads.txt, 1399324201-core-show-threads.txt, 1399324201-netstat.txt
>
>
> We've been experience an issue for a few months concerning IAX2 peers which has recently gotten more severe after upgrading from 11.2 to 11.6cert2.
> The initial symptom was all (100+) IAX2 peers going UNREACHABLE. However, after inspecting further it seems that what will happen is the UDP queues will sharply increase (seen by netstat -antup), the number of asterisk threads increases (to over 1000 threads in some cases), and Asterisk, of course, stops responding to inbound/outbound calls from any channel (SIP or IAX2).
> After recompiling with DEBUG_THREADS and BETTER_BACKTRACES, I discovered that issuing a "gdb -ex "thread apply all bt"...(etc) " to grab a backtrace will free up the UDP queues, and Asterisk will then become responsive again. Currently I have a script running each 5 minutes that pulls the UDP queues for asterisk processes, and upon seeing a queue above 300,000packets, I issue a "netstat -antup", "core show locks", "core show threads", and "gdb -ex "thread apply all bt" --batch asterisk `pidof asterisk` > $debugdir/$date-backtrace-threads.txt".
> I have previously increased the kernel UDP maximums in sysctl.conf, and added options for iaxthreadcount/iaxmaxthreadcount in iax.conf.
> I cannot repeat this issue at will, but it happens every hour or so (sometimes every few minutes). I have debug logs and backtraces for each occurrence.



--
This message was sent by Atlassian JIRA
(v6.2#6252)