[asterisk-bugs] [JIRA] (ASTERISK-28437) Taskprocessor doesn't process the tasks
sungtae kim (JIRA)
noreply at issues.asterisk.org
Wed Jun 5 09:20:47 CDT 2019
[ https://issues.asterisk.org/jira/browse/ASTERISK-28437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sungtae kim updated ASTERISK-28437:
-----------------------------------
Description:
Hi Asterisk team,
One of Asterisk's taskprocessor doesn't process the tasks. Seems like stuck some deadlock, but couldn't figure out why.
{noformat}
CLI> core show taskprocessors
Processor Processed In Queue Max Depth Low water High water
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
dns_system_resolver_tp 0 0 0 450 500
hep_queue_tp 0 0 0 450 500
iax2_transmit 0 0 0 450 500
pjsip/default-0000000a 1207948 0 2 450 500
pjsip/default-0000000b 1207946 0 1 450 500
pjsip/default-0000000c 1207949 0 2 450 500
pjsip/default-0000000d 1207948 0 2 450 500
pjsip/default-0000000e 1207949 0 2 450 500
pjsip/default-0000000f 1207950 0 2 450 500
pjsip/default-00000010 1207947 0 2 450 500
pjsip/default-00000011 1207945 0 1 450 500
pjsip/distributor-0000001e 267542 0 4 450 500
pjsip/distributor-0000001f 272156 0 6 450 500
pjsip/distributor-00000020 264634 0 5 450 500
pjsip/distributor-00000021 268282 0 5 450 500
pjsip/distributor-00000022 270238 0 7 450 500
pjsip/distributor-00000023 270059 0 4 450 500
pjsip/distributor-00000024 270525 0 6 450 500
pjsip/distributor-00000025 270730 0 4 450 500
pjsip/distributor-00000026 272161 0 5 450 500
pjsip/distributor-00000027 273404 0 6 450 500
pjsip/distributor-00000028 274615 0 9 450 500
pjsip/distributor-00000029 273141 0 4 450 500
pjsip/distributor-0000002a 269424 0 7 450 500
pjsip/distributor-0000002b 269376 0 5 450 500
pjsip/distributor-0000002c 269265 0 6 450 500
pjsip/distributor-0000002d 270574 0 4 450 500
pjsip/distributor-0000002e 269473 0 4 450 500
pjsip/distributor-0000002f 275079 0 5 450 500
pjsip/distributor-00000030 270892 0 5 450 500
pjsip/distributor-00000031 274360 0 4 450 500
pjsip/distributor-00000032 274523 0 4 450 500
pjsip/distributor-00000033 268106 0 7 450 500
pjsip/distributor-00000034 269143 0 6 450 500
pjsip/distributor-00000035 273046 0 4 450 500
pjsip/distributor-00000036 270502 1354 5 450 500
pjsip/distributor-00000037 274427 0 6 450 500
pjsip/distributor-00000038 268568 0 5 450 500
pjsip/distributor-00000039 271983 0 4 450 500
pjsip/distributor-0000003a 271705 0 4 450 500
pjsip/distributor-0000003b 269543 0 5 450 500
pjsip/distributor-0000003c 271750 0 5 450 500
...
289 taskprocessors
{noformat}
I'm doubting about this part, but not clear how to digging in deeper.
{noformat}
Thread 76 (LWP 7723):
#0 0x00007fe4216d6f5c in ?? () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:107 from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fe40cea1460 in ?? () from /usr/lib/asterisk/modules/res_stasis.so
#2 0x0000558571255db2 in __ast_string_field_empty_buffer ()
#3 0x00007fe4216d487c in pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:117
#4 0x0000558570e8deb8 in __ast_cond_signal (filename=<optimized out>, lineno=<optimized out>, func=<optimized out>, cond_name=<optimized out>, cond=<optimized out>) at lock.c:508
#5 0x00007fe40ce9c5da in exec_command_on_condition (can_exec_fn=0x0, data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:273
#6 exec_command (data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:283
#7 stasis_app_send_command_async (control=0x7fe207c56520, command_fn=<optimized out>, data=<optimized out>, data_destructor=<optimized out>) at stasis/control.c:843
#8 0x00007fe40ce9d43b in dial_bridge_after_cb (data=0x7fe207c56520, chan=<optimized out>) at stasis/control.c:941
#9 dial_bridge_after_cb_failed (reason=<optimized out>, data=0x7fe207c56520) at stasis/control.c:951
#10 0x0000558570dbbdf8 in after_bridge_cb_failed (node=0x7fe2b6c95630) at bridge_after.c:70
#11 after_bridge_cb_run_discard (reason=AST_BRIDGE_AFTER_CB_REASON_DESTROY, after_bridge=0x7fe206a3bab0) at bridge_after.c:99
#12 after_bridge_cb_destroy (data=0x7fe206a3bab0) at bridge_after.c:117
#13 0x0000558570e3e60c in ast_datastore_free (datastore=datastore at entry=0x7fe207638580) at datastore.c:68
#14 0x0000558570dffa56 in ast_channel_destructor (obj=0x7fe185b2ffd0) at channel.c:2255
#15 0x0000558570da1310 in __ao2_ref (user_data=0x7fe185b2ffd0, delta=delta at entry=-1, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2312, func=func at entry=0x7fe386409580 <__PRETTY_FUNCTION__.29394> "hangup_data_destroy") at astobj2.c:584
#16 0x00007fe3863fdcad in hangup_data_destroy (obj=0x7fe206717bc0) at chan_pjsip.c:2312
#17 0x0000558570da1310 in __ao2_ref (user_data=user_data at entry=0x7fe206717bc0, delta=delta at entry=-1, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2354, func=func at entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:584
#18 0x0000558570da16bb in __ao2_cleanup_debug (obj=obj at entry=0x7fe206717bc0, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2354, function=function at entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:657
#19 0x00007fe386402590 in hangup (data=0x7fe206717bc0) at chan_pjsip.c:2354
#20 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=tps at entry=0x5585723c4a30) at taskprocessor.c:970
#21 0x0000558570f51dd0 in execute_tasks (data=0x5585723c4a30) at threadpool.c:1322
#22 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=0x55857212d280) at taskprocessor.c:970
#23 0x0000558570f52304 in threadpool_execute (pool=0x55857212d760) at threadpool.c:351
#24 worker_active (worker=0x7fe3a8282340) at threadpool.c:1105
#25 worker_start (arg=arg at entry=0x7fe3a8282340) at threadpool.c:1024
#26 0x0000558570f5d949 in dummy_start (data=<optimized out>) at utils.c:1258
#27 0x00007fe4216ce494 in start_thread (arg=0x7fe3724f8700) at pthread_create.c:456
#28 0x00007fe420949acf in __libc_ifunc_impl_list (name=<optimized out>, array=0x7fe3724f8700, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:387
#29 0x0000000000000000 in ?? ()
{noformat}
After restarting the Asterisk, everything works fine. But we want to prevent this again in the future.
However, no idea what causes this problem and how to fix this if this happens again.
was:
One of Asterisk's taskprocessor doesn't process the tasks.
{noformat}
CLI> core show taskprocessors
Processor Processed In Queue Max Depth Low water High water
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
dns_system_resolver_tp 0 0 0 450 500
hep_queue_tp 0 0 0 450 500
iax2_transmit 0 0 0 450 500
pjsip/default-0000000a 1207948 0 2 450 500
pjsip/default-0000000b 1207946 0 1 450 500
pjsip/default-0000000c 1207949 0 2 450 500
pjsip/default-0000000d 1207948 0 2 450 500
pjsip/default-0000000e 1207949 0 2 450 500
pjsip/default-0000000f 1207950 0 2 450 500
pjsip/default-00000010 1207947 0 2 450 500
pjsip/default-00000011 1207945 0 1 450 500
pjsip/distributor-0000001e 267542 0 4 450 500
pjsip/distributor-0000001f 272156 0 6 450 500
pjsip/distributor-00000020 264634 0 5 450 500
pjsip/distributor-00000021 268282 0 5 450 500
pjsip/distributor-00000022 270238 0 7 450 500
pjsip/distributor-00000023 270059 0 4 450 500
pjsip/distributor-00000024 270525 0 6 450 500
pjsip/distributor-00000025 270730 0 4 450 500
pjsip/distributor-00000026 272161 0 5 450 500
pjsip/distributor-00000027 273404 0 6 450 500
pjsip/distributor-00000028 274615 0 9 450 500
pjsip/distributor-00000029 273141 0 4 450 500
pjsip/distributor-0000002a 269424 0 7 450 500
pjsip/distributor-0000002b 269376 0 5 450 500
pjsip/distributor-0000002c 269265 0 6 450 500
pjsip/distributor-0000002d 270574 0 4 450 500
pjsip/distributor-0000002e 269473 0 4 450 500
pjsip/distributor-0000002f 275079 0 5 450 500
pjsip/distributor-00000030 270892 0 5 450 500
pjsip/distributor-00000031 274360 0 4 450 500
pjsip/distributor-00000032 274523 0 4 450 500
pjsip/distributor-00000033 268106 0 7 450 500
pjsip/distributor-00000034 269143 0 6 450 500
pjsip/distributor-00000035 273046 0 4 450 500
pjsip/distributor-00000036 270502 1354 5 450 500
pjsip/distributor-00000037 274427 0 6 450 500
pjsip/distributor-00000038 268568 0 5 450 500
pjsip/distributor-00000039 271983 0 4 450 500
pjsip/distributor-0000003a 271705 0 4 450 500
pjsip/distributor-0000003b 269543 0 5 450 500
pjsip/distributor-0000003c 271750 0 5 450 500
...
289 taskprocessors
{noformat}
> Taskprocessor doesn't process the tasks
> ---------------------------------------
>
> Key: ASTERISK-28437
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-28437
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Core/Stasis
> Affects Versions: 15.7.2
> Reporter: sungtae kim
> Severity: Minor
> Labels: pjsip
> Attachments: core-asterisk-running-2019-06-05T13-01-31+0000-brief.txt, core-asterisk-running-2019-06-05T13-01-31+0000-full.txt, core-asterisk-running-2019-06-05T13-01-31+0000-locks.txt, core-asterisk-running-2019-06-05T13-01-31+0000-thread1.txt
>
>
> Hi Asterisk team,
> One of Asterisk's taskprocessor doesn't process the tasks. Seems like stuck some deadlock, but couldn't figure out why.
> {noformat}
> CLI> core show taskprocessors
> Processor Processed In Queue Max Depth Low water High water
> ast_msg_queue 0 0 0 450 500
> CCSS_core 0 0 0 450 500
> dns_system_resolver_tp 0 0 0 450 500
> hep_queue_tp 0 0 0 450 500
> iax2_transmit 0 0 0 450 500
> pjsip/default-0000000a 1207948 0 2 450 500
> pjsip/default-0000000b 1207946 0 1 450 500
> pjsip/default-0000000c 1207949 0 2 450 500
> pjsip/default-0000000d 1207948 0 2 450 500
> pjsip/default-0000000e 1207949 0 2 450 500
> pjsip/default-0000000f 1207950 0 2 450 500
> pjsip/default-00000010 1207947 0 2 450 500
> pjsip/default-00000011 1207945 0 1 450 500
> pjsip/distributor-0000001e 267542 0 4 450 500
> pjsip/distributor-0000001f 272156 0 6 450 500
> pjsip/distributor-00000020 264634 0 5 450 500
> pjsip/distributor-00000021 268282 0 5 450 500
> pjsip/distributor-00000022 270238 0 7 450 500
> pjsip/distributor-00000023 270059 0 4 450 500
> pjsip/distributor-00000024 270525 0 6 450 500
> pjsip/distributor-00000025 270730 0 4 450 500
> pjsip/distributor-00000026 272161 0 5 450 500
> pjsip/distributor-00000027 273404 0 6 450 500
> pjsip/distributor-00000028 274615 0 9 450 500
> pjsip/distributor-00000029 273141 0 4 450 500
> pjsip/distributor-0000002a 269424 0 7 450 500
> pjsip/distributor-0000002b 269376 0 5 450 500
> pjsip/distributor-0000002c 269265 0 6 450 500
> pjsip/distributor-0000002d 270574 0 4 450 500
> pjsip/distributor-0000002e 269473 0 4 450 500
> pjsip/distributor-0000002f 275079 0 5 450 500
> pjsip/distributor-00000030 270892 0 5 450 500
> pjsip/distributor-00000031 274360 0 4 450 500
> pjsip/distributor-00000032 274523 0 4 450 500
> pjsip/distributor-00000033 268106 0 7 450 500
> pjsip/distributor-00000034 269143 0 6 450 500
> pjsip/distributor-00000035 273046 0 4 450 500
> pjsip/distributor-00000036 270502 1354 5 450 500
> pjsip/distributor-00000037 274427 0 6 450 500
> pjsip/distributor-00000038 268568 0 5 450 500
> pjsip/distributor-00000039 271983 0 4 450 500
> pjsip/distributor-0000003a 271705 0 4 450 500
> pjsip/distributor-0000003b 269543 0 5 450 500
> pjsip/distributor-0000003c 271750 0 5 450 500
> ...
> 289 taskprocessors
> {noformat}
> I'm doubting about this part, but not clear how to digging in deeper.
> {noformat}
> Thread 76 (LWP 7723):
> #0 0x00007fe4216d6f5c in ?? () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:107 from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fe40cea1460 in ?? () from /usr/lib/asterisk/modules/res_stasis.so
> #2 0x0000558571255db2 in __ast_string_field_empty_buffer ()
> #3 0x00007fe4216d487c in pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:117
> #4 0x0000558570e8deb8 in __ast_cond_signal (filename=<optimized out>, lineno=<optimized out>, func=<optimized out>, cond_name=<optimized out>, cond=<optimized out>) at lock.c:508
> #5 0x00007fe40ce9c5da in exec_command_on_condition (can_exec_fn=0x0, data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:273
> #6 exec_command (data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:283
> #7 stasis_app_send_command_async (control=0x7fe207c56520, command_fn=<optimized out>, data=<optimized out>, data_destructor=<optimized out>) at stasis/control.c:843
> #8 0x00007fe40ce9d43b in dial_bridge_after_cb (data=0x7fe207c56520, chan=<optimized out>) at stasis/control.c:941
> #9 dial_bridge_after_cb_failed (reason=<optimized out>, data=0x7fe207c56520) at stasis/control.c:951
> #10 0x0000558570dbbdf8 in after_bridge_cb_failed (node=0x7fe2b6c95630) at bridge_after.c:70
> #11 after_bridge_cb_run_discard (reason=AST_BRIDGE_AFTER_CB_REASON_DESTROY, after_bridge=0x7fe206a3bab0) at bridge_after.c:99
> #12 after_bridge_cb_destroy (data=0x7fe206a3bab0) at bridge_after.c:117
> #13 0x0000558570e3e60c in ast_datastore_free (datastore=datastore at entry=0x7fe207638580) at datastore.c:68
> #14 0x0000558570dffa56 in ast_channel_destructor (obj=0x7fe185b2ffd0) at channel.c:2255
> #15 0x0000558570da1310 in __ao2_ref (user_data=0x7fe185b2ffd0, delta=delta at entry=-1, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2312, func=func at entry=0x7fe386409580 <__PRETTY_FUNCTION__.29394> "hangup_data_destroy") at astobj2.c:584
> #16 0x00007fe3863fdcad in hangup_data_destroy (obj=0x7fe206717bc0) at chan_pjsip.c:2312
> #17 0x0000558570da1310 in __ao2_ref (user_data=user_data at entry=0x7fe206717bc0, delta=delta at entry=-1, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2354, func=func at entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:584
> #18 0x0000558570da16bb in __ao2_cleanup_debug (obj=obj at entry=0x7fe206717bc0, tag=tag at entry=0x7fe386409b29 "", file=file at entry=0x7fe3864099bf "chan_pjsip.c", line=line at entry=2354, function=function at entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:657
> #19 0x00007fe386402590 in hangup (data=0x7fe206717bc0) at chan_pjsip.c:2354
> #20 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=tps at entry=0x5585723c4a30) at taskprocessor.c:970
> #21 0x0000558570f51dd0 in execute_tasks (data=0x5585723c4a30) at threadpool.c:1322
> #22 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=0x55857212d280) at taskprocessor.c:970
> #23 0x0000558570f52304 in threadpool_execute (pool=0x55857212d760) at threadpool.c:351
> #24 worker_active (worker=0x7fe3a8282340) at threadpool.c:1105
> #25 worker_start (arg=arg at entry=0x7fe3a8282340) at threadpool.c:1024
> #26 0x0000558570f5d949 in dummy_start (data=<optimized out>) at utils.c:1258
> #27 0x00007fe4216ce494 in start_thread (arg=0x7fe3724f8700) at pthread_create.c:456
> #28 0x00007fe420949acf in __libc_ifunc_impl_list (name=<optimized out>, array=0x7fe3724f8700, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:387
> #29 0x0000000000000000 in ?? ()
> {noformat}
> After restarting the Asterisk, everything works fine. But we want to prevent this again in the future.
> However, no idea what causes this problem and how to fix this if this happens again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list