[asterisk-bugs] [JIRA] (ASTERISK-30445) stasis: Failure to allocate memory causes crashes

[Quality Unit (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-30445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=261792#comment-261792 ] 

Quality Unit edited comment on ASTERISK-30445 at 4/18/23 5:20 AM:
------------------------------------------------------------------

Was obtained with:
{code:none}
$ grep -A 1000 -B 1000 'Failed assertion' full > asterisk_incident_1000_lines_before_and_after.txt
{code}

Full log size is 1.6GB, let me know if you need more, I can try to compress/truncate it and share full.

Asterisk is running with `-vvvvvdddddg` verbosity levels, so hopefully logs are as verbose as possible.


was (Author: adminqu):
Was obtained with:
{code:none}
$ grep -A 1000 -B 1000 'Failed assertion' full > asterisk_incident_1000_lines_before_and_after.txt
{code}

Full log size if 1.6GB, let me know if you need more, I can try to compress/truncate it and share full.

> stasis: Failure to allocate memory causes crashes
> -------------------------------------------------
>
>                 Key: ASTERISK-30445
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-30445
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Resources/res_stasis
>    Affects Versions: 18.16.0, 20.1.0
>         Environment: CentOS 7, kernel 3.10.0-1160.83.1.el7.x86_64.
>            Reporter: Quality Unit
>            Assignee: Unassigned
>            Severity: Major
>         Attachments: asterisk_dump_logs_2023-04-17-10_core.1913.zip, asterisk_dump_logs_2023-04-17_core.18037.zip, asterisk_dump_logs_2023-04-17_core.26817.zip, asterisk_incident_1000_lines_before_and_after.txt, core_asterisk.zip, module_show_output&logs_before_incident.txt.zip
>
>
> This started to happen after upgrade to `18.16.0` from `16.29.0`. Asterisk starting to spam logs with similar messages like:
> {code:none}
> [2023-02-24 09:30:57.001] ERROR[11361] stasis.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x7f7ec4031d10 (0)
> [2023-02-24 09:30:57.006] ERROR[11361] : Got 14 backtrace records
> # 0: /usr/sbin/asterisk() [0x459ae6]
> # 1: /usr/sbin/asterisk(__ao2_ref+0x79) [0x459fc9]
> # 2: /usr/sbin/asterisk(stasis_forward_all+0x83) [0x555283]
> # 3: /usr/lib64/asterisk/modules/res_stasis.so(+0x107e1) [0x7f7f62a7e7e1]
> # 4: /usr/lib64/asterisk/modules/res_stasis.so(+0x14e2a) [0x7f7f62a82e2a]
> # 5: /usr/lib64/asterisk/modules/res_stasis.so(+0x12966) [0x7f7f62a80966]
> # 6: /usr/lib64/asterisk/modules/res_stasis.so(+0x1554e) [0x7f7f62a8354e]
> # 7: /usr/lib64/asterisk/modules/res_stasis.so(stasis_app_exec+0x569) [0x7f7f62a79789]
> # 8: /usr/lib64/asterisk/modules/app_stasis.so(+0xbd7) [0x7f7f4d6f4bd7]
> # 9: /usr/sbin/asterisk(pbx_exec+0xb9) [0x50db89]
> #10: /usr/lib64/asterisk/modules/res_ari_channels.so(+0xa528) [0x7f7f48594528]
> #11: /usr/sbin/asterisk() [0x57e80d]
> #12: /lib64/libpthread.so.0(+0x7ea5) [0x7f7fb932fea5]
> #13: /lib64/libc.so.6(clone+0x6d) [0x7f7fb88ff98d]
> {code}
> After around ~3 minutes of these (Asterisk during that time is already unresponsive), we have final errors before service crash:
> {code:none}
> [2023-02-24 09:32:48.153] ERROR[11361] astmm.c: Memory Allocation Failure in function topic_add_subscription at line 1214 of stasis.c
> {code}
> And, from `/var/log/messages`:
> {code:none}
> Feb 24 09:32:48 localhost kernel: asterisk[11361]: segfault at 1 ip 0000000000552827 sp 00007f7ed3e0a6b0 error 4 in asterisk[400000+2a2000]
> {code}
> It happens randomly, once, twice a day, sometime doesn't happen at all.
> Though, it's always happening when Asterisk is processing calls, so it's not happening during night (when asterisk is almost idle)



--
This message was sent by Atlassian JIRA
(v6.2#6252)