[asterisk-bugs] [JIRA] (ASTERISK-28997) res_pjsip: Asterisk locks up and stops processing any SIP requests
Yury Kirsanov (JIRA)
noreply at issues.asterisk.org
Tue Jul 21 18:26:27 CDT 2020
[ https://issues.asterisk.org/jira/browse/ASTERISK-28997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=251488#comment-251488 ]
Yury Kirsanov commented on ASTERISK-28997:
------------------------------------------
Hi Joshua,
This server has a complex call-flow setup with a lot of dialplans and endpoints, we have 2195 endpoints created currently though not all of them are in use. Asterisk was compiled with all modules except for all channels rather than pjsip. It uses both UDP, TCP and TLS. Realtime is not used and not loaded. It uses FAX in T.38 and V30 modes via res_fax_spandsp. Server is using Asterisk DB to store a lot of parameters.
I'm not sure why sorcery memory cache is in use, probably because it's compiled in by default?
Here's a list of loaded modules:
asterisk -rx "module show"
Module Description Use Count Status Support Level
acl Named ACL system 2 Running core
app_authenticate.so Authentication Application 0 Running core
app_bridgewait.so Place the channel into a holding bridge 0 Running core
app_cdr.so Tell Asterisk to not maintain a CDR for 0 Running core
app_confbridge.so Conference Bridge Application 0 Running core
app_db.so Database Access Functions 0 Running core
app_dial.so Dialing Application 70 Running core
app_directed_pickup.so Directed Call Pickup Application 0 Running core
app_disa.so DISA (Direct Inward System Access) Appli 0 Running core
app_echo.so Simple Echo Application 0 Running core
app_exec.so Executes dialplan applications 0 Running core
app_macro.so Extension Macros 0 Running deprecated
app_mixmonitor.so Mixed Audio Monitoring Application 26 Running core
app_playback.so Sound File Playback Application 2 Running core
app_queue.so True Call Queueing 7 Running core
app_read.so Read Variable Application 1 Running core
app_readexten.so Read and evaluate extension validity 0 Running core
app_record.so Trivial Record Application 0 Running core
app_softhangup.so Hangs up the requested channel 0 Running core
app_stack.so Dialplan subroutines (Gosub, Return, etc 109 Running core
app_system.so Generic System() application 0 Running core
app_userevent.so Custom User Event Application 0 Running core
app_verbose.so Send verbose output 0 Running core
app_voicemail.so Comedian Mail (Voicemail System) 0 Running core
app_while.so While Loops and Conditional Execution 0 Running core
bridge_builtin_features.so Built in bridging features 1 Running core
bridge_builtin_interval_features.so Built in bridging interval features 0 Running core
bridge_holding.so Holding bridge module 5 Running core
bridge_native_rtp.so Native RTP bridging module 16 Running core
bridge_simple.so Simple two channel bridging module 45 Running core
bridge_softmix.so Multi-party software based channel mixin 0 Running core
ccss Call Completion Supplementary Services 2 Running core
cdr CDR Engine 2 Running core
cel CEL Engine 1 Running core
chan_pjsip.so PJSIP Channel Driver 910 Running core
codec_a_mu.so A-law and Mulaw direct Coder/Decoder 41 Running core
codec_adpcm.so Adaptive Differential PCM Coder/Decoder 0 Running core
codec_alaw.so A-law Coder/Decoder 71 Running core
codec_g722.so ITU G.722-64kbps G722 Transcoder 39 Running core
codec_g723.so g723 Coder/Decoder, based on Intel IPP 0 Running unknown
codec_g726.so ITU G.726-32kbps G726 Transcoder 0 Running core
codec_g729.so g729 Coder/Decoder, based on Intel IPP 7 Running unknown
codec_gsm.so GSM Coder/Decoder 0 Running core
codec_ilbc.so iLBC Coder/Decoder 0 Running core
codec_lpc10.so LPC10 2.4kbps Coder/Decoder 0 Running core
codec_resample.so SLIN Resampling Codec 52 Running core
codec_ulaw.so mu-Law Coder/Decoder 2 Running core
dnsmgr DNS Manager 2 Running core
dsp DSP 1 Running core
enum ENUM Support 1 Running core
extconfig Configuration 14 Running core
features Call Features 123 Running core
format_g719.so ITU G.719 0 Running core
format_g723.so G.723.1 Simple Timestamp File Format 0 Running core
format_g726.so Raw G.726 (16/24/32/40kbps) data 0 Running core
format_g729.so Raw G.729 data 0 Running core
format_gsm.so Raw GSM data 0 Running core
format_h263.so Raw H.263 data 0 Running core
format_h264.so Raw H.264 data 0 Running core
format_ilbc.so Raw iLBC data 0 Running core
format_mp3.so MP3 format [Any rate but 8000hz mono is 0 Running extended
format_pcm.so Raw/Sun uLaw/ALaw 8KHz (PCM,PCMA,AU), G. 2 Running core
format_siren14.so ITU G.722.1 Annex C (Siren14, licensed f 0 Running core
format_siren7.so ITU G.722.1 (Siren7, licensed from Polyc 0 Running core
format_sln.so Raw Signed Linear Audio support (SLN) 8k 0 Running core
format_vox.so Dialogic VOX (ADPCM) File Format 0 Running extended
format_wav.so Microsoft WAV/WAV16 format (8kHz/16kHz S 17 Running core
format_wav_gsm.so Microsoft WAV format (Proprietary GSM) 0 Running core
func_callerid.so Party ID related dialplan functions (Cal 0 Running core
func_cdr.so Call Detail Record (CDR) dialplan functi 0 Running core
func_channel.so Channel information dialplan functions 0 Running core
func_config.so Asterisk configuration file variable acc 0 Running core
func_curl.so Load external URL 1 Running core
func_cut.so Cut out information from a string 0 Running core
func_db.so Database (astdb) related dialplan functi 0 Running core
func_devstate.so Gets or sets a device state in the dialp 0 Running core
func_env.so Environment/filesystem dialplan function 0 Running core
func_global.so Variable dialplan functions 0 Running core
func_groupcount.so Channel group dialplan functions 0 Running core
func_hangupcause.so HANGUPCAUSE related functions and applic 0 Running core
func_logic.so Logical dialplan functions 0 Running core
func_math.so Mathematical dialplan function 0 Running core
func_periodic_hook.so Periodic dialplan hooks. 2 Running core
func_strings.so String handling dialplan functions 0 Running core
func_timeout.so Channel timeout dialplan functions 0 Running core
http Built-in HTTP Server 2 Running core
indications Indication Tone Handling 1 Running core
logger Logger 1 Running core
manager Asterisk Manager Interface 1 Running core
pbx_config.so Text Extension Configuration 0 Running core
pbx_loopback.so Loopback Switch 0 Running core
pbx_spool.so Outgoing Spool Support 0 Running core
plc PLC 1 Running core
res_adsi.so ADSI Resource 1 Running deprecated
res_agi.so Asterisk Gateway Interface (AGI) 1 Running core
res_clialiases.so CLI Aliases 0 Running core
res_clioriginate.so Call origination and redirection from th 0 Running core
res_config_curl.so Realtime Curl configuration 0 Running core
res_convert.so File format conversion CLI command 0 Running core
res_crypto.so Cryptographic Digital Signatures 0 Running core
res_curl.so cURL Resource Module 2 Running core
res_fax.so Generic FAX Applications 26 Running core
res_fax_spandsp.so Spandsp G.711 and T.38 FAX Technologies 25 Running extended
res_format_attr_celt.so CELT Format Attribute Module 1 Running core
res_format_attr_g729.so G.729 Format Attribute Module 1 Running core
res_format_attr_h263.so H.263 Format Attribute Module 1 Running core
res_format_attr_h264.so H.264 Format Attribute Module 1 Running core
res_format_attr_ilbc.so iLBC Format Attribute Module 1 Running core
res_format_attr_opus.so Opus Format Attribute Module 1 Running core
res_format_attr_silk.so SILK Format Attribute Module 1 Running core
res_format_attr_siren14.so Siren14 Format Attribute Module 1 Running core
res_format_attr_siren7.so Siren7 Format Attribute Module 1 Running core
res_format_attr_vp8.so VP8 Format Attribute Module 1 Running core
res_limit.so Resource limits 0 Running core
res_monitor.so Call Monitoring Resource 2 Running deprecated
res_musiconhold.so Music On Hold Resource 13 Running core
res_parking.so Call Parking Resource 3 Running core
res_pjproject.so PJPROJECT Log and Utility Support 4 Running core
res_pjsip.so Basic SIP resource 211 Running core
res_pjsip_authenticator_digest.so PJSIP authentication resource 0 Running core
res_pjsip_caller_id.so PJSIP Caller ID Support 183 Running core
res_pjsip_dialog_info_body_generator.so PJSIP Extension State Dialog Info+XML Pr 0 Running core
res_pjsip_diversion.so PJSIP Add Diversion Header Support 183 Running core
res_pjsip_dlg_options.so SIP OPTIONS in dialog handler 182 Running core
res_pjsip_dtmf_info.so PJSIP DTMF INFO Support 182 Running core
res_pjsip_empty_info.so PJSIP Empty INFO Support 182 Running core
res_pjsip_endpoint_identifier_anonymous.so PJSIP Anonymous endpoint identifier 0 Running core
res_pjsip_endpoint_identifier_ip.so PJSIP IP endpoint identifier 0 Running core
res_pjsip_endpoint_identifier_user.so PJSIP username endpoint identifier 0 Running core
res_pjsip_exten_state.so PJSIP Extension State Notifications 0 Running core
res_pjsip_header_funcs.so PJSIP Header Functions 182 Running core
res_pjsip_logger.so PJSIP Packet Logger 0 Running core
res_pjsip_messaging.so PJSIP Messaging Support 182 Running core
res_pjsip_nat.so PJSIP NAT Support 182 Running core
res_pjsip_outbound_publish.so PJSIP Outbound Publish Support 1 Running core
res_pjsip_pidf_body_generator.so PJSIP Extension State PIDF Provider 0 Running core
res_pjsip_pidf_digium_body_supplement.so PJSIP PIDF Digium presence supplement 0 Running core
res_pjsip_pidf_eyebeam_body_supplement.so PJSIP PIDF Eyebeam supplement 0 Running core
res_pjsip_pubsub.so PJSIP event resource 4116 Running core
res_pjsip_refer.so PJSIP Blind and Attended Transfer Suppor 183 Running core
res_pjsip_registrar.so PJSIP Registrar Support 0 Running core
res_pjsip_rfc3326.so PJSIP RFC3326 Support 182 Running core
res_pjsip_sdp_rtp.so PJSIP SDP RTP/AVP stream handler 182 Running core
res_pjsip_session.so PJSIP Session resource 14 Running core
res_pjsip_sips_contact.so UAC SIPS Contact support 0 Running core
res_pjsip_t38.so PJSIP T.38 UDPTL Support 311 Running core
res_pjsip_xpidf_body_generator.so PJSIP Extension State PIDF Provider 0 Running core
res_rtp_asterisk.so Asterisk RTP Stack 128 Running core
res_security_log.so Security Event Logging 0 Running core
res_smdi.so Simplified Message Desk Interface (SMDI) 1 Running extended
res_sorcery_astdb.so Sorcery Astdb Object Wizard 2 Running core
res_sorcery_config.so Sorcery Configuration File Object Wizard 12 Running core
res_sorcery_memory.so Sorcery In-Memory Object Wizard 1 Running core
res_sorcery_memory_cache.so Sorcery Memory Cache Object Wizard 6 Running core
res_speech.so Generic Speech Recognition API 1 Running core
res_srtp.so Secure RTP (SRTP) 6 Running core
res_statsd.so StatsD client support 1 Running extended
res_stun_monitor.so STUN Network Monitor 0 Running core
res_timing_pthread.so pthread Timing Interface 164 Running extended
sounds Sounds Index 1 Running core
udptl UDPTL 2 Running core
161 modules loaded
> res_pjsip: Asterisk locks up and stops processing any SIP requests
> ------------------------------------------------------------------
>
> Key: ASTERISK-28997
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-28997
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Resources/res_pjsip, Resources/res_sorcery_memory_cache
> Affects Versions: 16.11.1
> Environment: Ubuntu Linux 18.04.4 LTS, kernel 4.15.0-111-generic
> Reporter: Yury Kirsanov
> Assignee: Yury Kirsanov
> Severity: Minor
> Labels: fax
> Attachments: backtrace-threads.txt, taskprocessors.txt, threads.txt
>
>
> Hi,
> We have a problem with one of many of our Asterisk-based servers. From time to time it just locks up and stops processing SIP messages, complaining about pjsip/distributor queue has reached 500 scheduled tasks. We have several servers based on same Asterisk version with exactly the same environment, but different Asterisk configuration files, some have more SIP devices and more complex dial plans, some has less.
> This issue happens at any time of the day and I don't see any way to reproduce it. There is no segfault generated as Asterisk keeps running, the only thing is that all distributor queues are overflown for some reason.
> After Asterisk restart it starts to work like nothing happened. I've checked all network interfaces and there is no DDoS attack or anything like that.
> Our Asterisk is patched with ASTERISK-28923 patch and it's the same server that was segfaulting before. Other similar servers are running unpatched version of Asterisk 16.11.1.
> I'm attaching debug logs that I was able to grab during this lockup. Asterisk was compiled with 'NO_DEBUG' option.
> Please let me know if I can provide any additional information or recompile Asterisk with some additional debug versions. Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list