[asterisk-bugs] [JIRA] (ASTERISK-29415) Crash in PJSIP TLS transport
Friendly Automation (JIRA)
noreply at issues.asterisk.org
Thu Jul 22 16:21:38 CDT 2021
[ https://issues.asterisk.org/jira/browse/ASTERISK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=255694#comment-255694 ]
Friendly Automation commented on ASTERISK-29415:
------------------------------------------------
Change 16209 merged by Friendly Automation:
AST-2021-009 - pjproject-bundled: Avoid crash during handshake for TLS
[https://gerrit.asterisk.org/c/asterisk/+/16209|https://gerrit.asterisk.org/c/asterisk/+/16209]
> Crash in PJSIP TLS transport
> -----------------------------
>
> Key: ASTERISK-29415
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-29415
> Project: Asterisk
> Issue Type: Security
> Security Level: None
> Components: Channels/chan_pjsip
> Affects Versions: 18.3.0
> Environment: Ubuntu 20.04
> OpenSSL 1.1.1f
> Reporter: Andrew Yager
> Assignee: Unassigned
> Severity: Major
> Labels: patch, security
> Attachments: ASTERISK-29415.diff, CoreDump-brief.txt, CoreDump-full.txt, CoreDump-info.txt.zip, CoreDump-locks.txt, CoreDump-thread1.txt
>
>
> We've been able to reproduce a repeatable crash on a system with a large number of endpoints where there are mixed TLS versions in use by the clients, and Asterisk will reliably crash within 2 - 3 minutes of startup.
> We have not been able to verify if the crash is simply related to unsupported TLS version traffic being recieved, or bad TLS traffic, or other memory management issues, but the stack trace seems to be pretty reliably:
> {code}
> #0 0x00007f34451d0e74 in on_accept_complete2 (ssock=0x555f4fabe688, new_ssock=0x7f3211d0dc38, src_addr=0x7f3211d0e31c, src_addr_len=16, accept_status=0) at ../src/pjsip/sip_transport_tls.c:1352
> #1 0x00007f344527953e in on_handshake_complete (ssock=0x7f3211d0dc38, status=0) at ../src/pj/ssl_sock_imp_common.c:290
> #2 0x00007f3445279f89 in asock_on_data_read (asock=0x7f320a7d90a0, data=0x7f320a7c9078, size=395, status=0, remainder=0x7f33bfffeb10) at ../src/pj/ssl_sock_imp_common.c:681
> #3 0x00007f344526cc1c in ioqueue_on_read_complete (key=0x555f4f83b7e0, op_key=0x7f327bfffbc8, bytes_read=395) at ../src/pj/activesock.c:504
> #4 0x00007f3445265180 in ioqueue_dispatch_read_event (ioqueue=0x7f33f85940a0, h=0x555f4f83b7e0) at ../src/pj/ioqueue_common_abs.c:605
> #5 0x00007f34452672d5 in pj_ioqueue_poll (ioqueue=0x7f33f85940a0, timeout=0x7f33bfffedf0) at ../src/pj/ioqueue_epoll.c:720
> #6 0x00007f34451bb759 in pjsip_endpt_handle_events2 (endpt=0x555f4f7fcf28, max_timeout=0x7f33bfffee50, p_count=0x0) at ../src/pjsip/sip_endpoint.c:745
> #7 0x00007f34451bb832 in pjsip_endpt_handle_events (endpt=0x555f4f7fcf28, max_timeout=0x7f33bfffee50) at ../src/pjsip/sip_endpoint.c:777
> #8 0x00007f33fb6cc8d6 in monitor_thread_exec (endpt=0x0) at res_pjsip.c:5166
> #9 0x00007f3445268298 in thread_main (param=0x555f4f7b0cc8) at ../src/pj/os_core_unix.c:541
> #10 0x00007f3444a68609 in start_thread (arg=<optimized out>) at pthread_create.c:477
> #11 0x00007f34446a3293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> {code}
> with the full thread1 trace as:
> {code}
> !@!@!@! thread1.txt !@!@!@!
> $1 = {si_signo = 11, si_errno = 0, si_code = 1, _sifields = {_pad = {176, 0 <repeats 27 times>}, _kill = {si_pid = 176, si_uid = 0}, _timer = {si_tid = 176, si_overrun = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {si_pid = 176, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _sigchld = {si_pid = 176, si_uid = 0, si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0xb0, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 176, si_fd = 0}}}
> Signal Stop Print Pass to program Description
> SIGSEGV Yes Yes Yes Segmentation fault
> Thread 1 (Thread 0x7f33bffff700 (LWP 59868)):
> #0 0x00007f34451d0e74 in on_accept_complete2 (ssock=0x555f4fabe688, new_ssock=0x7f3211d0dc38, src_addr=0x7f3211d0e31c, src_addr_len=16, accept_status=0) at ../src/pjsip/sip_transport_tls.c:1352
> listener = 0x0
> tls = 0x0
> ssl_info = {established = -1073747744, proto = 32563, cipher = 1674508032, local_addr = {addr = {sa_family = 64638}, ipv4 = {sin_family = 64638, sin_port = 60033, sin_addr = {s_addr = 2816287024}, sin_zero = "3177000000000367316c"}, ipv6 = {sin6_family = 64638, sin6_port = 60033, sin6_flowinfo = 2816287024, sin6_addr = {s6_addr = "3177000000000367316c~374201352260315324021", u6_addr32 = {32563, 1674508032, 3934387326, 299158960}}, sin6_scope_id = 32562}}, remote_addr = {addr = {sa_family = 0}, ipv4 = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "360367062246063177000"}, ipv6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {s6_addr = "360367062246063177000000276026sR376177000", u6_addr32 = {2788358128, 32563, 1383274174, 32766}}, sin6_scope_id = 1383274175}}, local_cert_info = 0x7f344527cd07 <sk_X509_num+24>, remote_cert_info = 0x7f33bfffef80, verify_status = 1010030128, last_native_err = 139860241279344, grp_lock = 0x7f3445280424 <ssl_update_remote_cert_chain_info+412>}
> addr = "330022270061001000000000060326063<2177000000350337320021062177000000260310324342062177000000p351377277063177000000000E'E002000000000060!ݧ3177000"
> state_cb = 0x7f344461b00b <_int_free+1611>
> tmp_src_addr = {addr = {sa_family = 5}, ipv4 = {sin_family = 5, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "x356N023062177000"}, ipv6 = {sin6_family = 5, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {s6_addr = "x356N023062177000000356065*E4177000", u6_addr32 = {323939960, 32562, 1160394222, 32564}}, sin6_scope_id = 1674508032}}
> is_shutdown = 32564
> status = 1153712788
> addr_buf = "360367062246063177000000000367316c~374201352260351377277063177000000n005(E4177000000240351377277063177000000070334320021062177000000070334320021000000000"
> #1 0x00007f344527953e in on_handshake_complete (ssock=0x7f3211d0dc38, status=0) at ../src/pj/ssl_sock_imp_common.c:290
> ret = 1383274174
> #2 0x00007f3445279f89 in asock_on_data_read (asock=0x7f320a7d90a0, data=0x7f320a7c9078, size=395, status=0, remainder=0x7f33bfffeb10) at ../src/pj/ssl_sock_imp_common.c:681
> ret = 1
> ssock = 0x7f3211d0dc38
> #3 0x00007f344526cc1c in ioqueue_on_read_complete (key=0x555f4f83b7e0, op_key=0x7f327bfffbc8, bytes_read=395) at ../src/pj/activesock.c:504
> remainder = 0
> ret = 1
> flags = 32563
> asock = 0x7f320a7d90a0
> r = 0x7f327bfffbc8
> loop = 0
> status = -1073747152
> #4 0x00007f3445265180 in ioqueue_dispatch_read_event (ioqueue=0x7f33f85940a0, h=0x555f4f83b7e0) at ../src/pj/ioqueue_common_abs.c:605
> read_op = 0x7f327bfffbc8
> bytes_read = 395
> has_lock = 1
> rc = 0
> #5 0x00007f34452672d5 in pj_ioqueue_poll (ioqueue=0x7f33f85940a0, timeout=0x7f33bfffedf0) at ../src/pj/ioqueue_epoll.c:720
> i = 0
> count = 1
> event_cnt = 1
> processed_cnt = 0
> msec = 10
> events = {{events = 1, data = {ptr = 0x555f4f83b7e0, fd = 1334032352, u32 = 1334032352, u64 = 93867844286432}}, {events = 32563, data = {ptr = 0x7f33bfffec10, fd = -1073746928, u32 = 3221220368, u64 = 139860241280016}}, {events = 1160201472, data = {ptr = 0xbfffec1000000000, fd = 0, u32 = 0, u64 = 13835036133769084928}}, {events = 32563, data = {ptr = 0x7f34452716c5 <pj_lock_release+50>, fd = 1160189637, u32 = 1160189637, u64 = 139862475216581}}, {events = 3805720496, data = {ptr = 0xfb72400800007f32, fd = 32562, u32 = 32562, u64 = 18118614653968875314}}, {events = 32563, data = {ptr = 0x7f33bfffec60, fd = -1073746848, u32 = 3221220448, u64 = 139860241280096}}, {events = 1160204127, data = {ptr = 0xe2d6afb000007f34, fd = 32564, u32 = 32564, u64 = 16345445068036931380}}, {events = 32562, data = {ptr = 0x7f33fb723de0 <caching_pool>, fd = -76399136, u32 = 4218568160, u64 = 139861238627808}}, {events = 3221220448, data = {ptr = 0xe2d6b0f800007f33, fd = 32563, u32 = 32563, u64 = 16345446476786204467}}, {events = 1, data = {ptr = 0x7f33bfffed08, fd = -1073746680, u32 = 3221220616, u64 = 139860241280264}}, {events = 3221220624, data = {ptr = 0xbfffec9000007f33, fd = 32563, u32 = 32563, u64 = 13835036683524931379}}, {events = 32563, data = {ptr = 0x7f344526abbd <elapsed_msec+125>, fd = 1160162237, u32 = 1160162237, u64 = 139862475189181}}, {events = 3221220616, data = {ptr = 0xbfffed1000007f33, fd = 32563, u32 = 32563, u64 = 13835037233280745267}}, {events = 32563, data = {ptr = 0x3b9aca00, fd = 1000000000, u32 = 1000000000, u64 = 1000000000}}, {events = 0, data = {ptr = 0x963e5b8541cdcd65, fd = 1104006501, u32 = 1104006501, u64 = 10826191182138035557}}, {events = 1093863564, data = {ptr = 0xea81fc7e63cef700, fd = 1674508032, u32 = 1674508032, u64 = 16898064896641398528}}}
> queue = {{key = 0x555f4f83b7e0, event_type = READABLE_EVENT}, {key = 0x7f33bfffed08, event_type = 3221220624}, {key = 0x7f33bfffece0, event_type = 1160154766}, {key = 0x40937d7d876adcb1, event_type = 4166598744}, {key = 0x4df, event_type = 372}, {key = 0x7f33bfffed00, event_type = 1160189637}, {key = 0x0, event_type = 4166598704}, {key = 0x7f33bfffed20, event_type = 1160262370}, {key = 0x0, event_type = 1333776912}, {key = 0x7f33bfffeda0, event_type = 1160267970}, {key = 0x7f33bfffedf0, event_type = 1333776912}, {key = 0x555f4f9c83c8, event_type = READABLE_EVENT}, {key = 0x1, event_type = 4167558488}, {key = 0x7f341dc3f4b8, event_type = 3805720664}, {key = 0x4df, event_type = 372}, {key = 0x4df, event_type = 398}}
> t1 = {u32 = {lo = 1832071522, hi = 290}, u64 = 1247372587362}
> t2 = {u32 = {lo = 1835242228, hi = 290}, u64 = 1247375758068}
> #6 0x00007f34451bb759 in pjsip_endpt_handle_events2 (endpt=0x555f4f7fcf28, max_timeout=0x7f33bfffee50, p_count=0x0) at ../src/pjsip/sip_endpoint.c:745
> timeout = {sec = 0, msec = 10}
> count = 0
> net_event_count = 0
> c = 0
> #7 0x00007f34451bb832 in pjsip_endpt_handle_events (endpt=0x555f4f7fcf28, max_timeout=0x7f33bfffee50) at ../src/pjsip/sip_endpoint.c:777
> No locals.
> #8 0x00007f33fb6cc8d6 in monitor_thread_exec (endpt=0x0) at res_pjsip.c:5166
> delay = {sec = 0, msec = 10}
> #9 0x00007f3445268298 in thread_main (param=0x555f4f7b0cc8) at ../src/pj/os_core_unix.c:541
> rec = 0x555f4f7b0cc8
> result = 0x0
> rc = 0
> #10 0x00007f3444a68609 in start_thread (arg=<optimized out>) at pthread_create.c:477
> ret = <optimized out>
> pd = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139860241282816, -4867490839655063574, 140730281694910, 140730281694911, 140730281695120, 139860241280896, 4761515510424952810, 4763607274558480362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = 0
> #11 0x00007f34446a3293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> No locals.
> {code}
> While we can "reliably reproduce" this, we need to migrate a large portion of endpoints to our testing environment, which is not particularly easy to schedule to allow us to reproduce, so I'm hoping the above or the core dumps we have may be useful to help identify the likely candidate.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list